Pod Priority and Preemption

FEATURE STATE: Kubernetes 1.14stableThis feature is stable, meaning:

• The version name is vX where X is an integer.
• Stable versions of features will appear in released software for many subsequent versions.

Pods can have priority. Priority indicates theimportance of a Pod relative to other Pods. If a Pod cannot be scheduled, thescheduler tries to preempt (evict) lower priority Pods to make scheduling of thepending Pod possible.

In Kubernetes 1.9 and later, Priority also affects scheduling order of Pods andout-of-resource eviction ordering on the Node.

Pod priority and preemption graduated to beta in Kubernetes 1.11 and to GA inKubernetes 1.14. They have been enabled by default since 1.11.

In Kubernetes versions where Pod priority and preemption is still an alpha-levelfeature, you need to explicitly enable it. To use these features in the olderversions of Kubernetes, follow the instructions in the documentation for yourKubernetes version, by going to the documentation archive version for yourKubernetes version.

Warning: In a cluster where not all users are trusted, amalicious user could create pods at the highest possible priorities, causingother pods to be evicted/not get scheduled. To resolve this issue,ResourceQuota isaugmented to support Pod priority. An admin can create ResourceQuota for usersat specific priority levels, preventing them from creating pods at highpriorities. This feature is in beta since Kubernetes 1.12.

How to use priority and preemption

To use priority and preemption in Kubernetes 1.11 and later, follow these steps:

• Add one or more PriorityClasses.

• Create Pods withpriorityClassName set to one of the addedPriorityClasses. Of course you do not need to create the Pods directly;normally you would add priorityClassName to the Pod template of acollection object like a Deployment.

Keep reading for more information about these steps.

If you try the feature and then decide to disable it, you must remove thePodPriority command-line flag or set it to false, and then restart the APIserver and scheduler. After the feature is disabled, the existing Pods keeptheir priority fields, but preemption is disabled, and priority fields areignored. If the feature is disabled, you cannot set priorityClassName in newPods.

How to disable preemption

Note: In Kubernetes 1.12+, critical pods rely on scheduler preemption to be scheduledwhen a cluster is under resource pressure. For this reason, it is notrecommended to disable preemption.

Note: In Kubernetes 1.15 and later,if the feature NonPreemptingPriority is enabled,PriorityClasses have the option to set preemptionPolicy: Never.This will prevent pods of that PriorityClass from preempting other pods.

In Kubernetes 1.11 and later, preemption is controlled by a kube-scheduler flagdisablePreemption, which is set to false by default.If you want to disable preemption despite the above note, you can setdisablePreemption to true.

This option is available in component configs only and is not available inold-style command line options. Below is a sample component config to disablepreemption:

apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
algorithmSource:
  provider: DefaultProvider
...
disablePreemption: true

PriorityClass

A PriorityClass is a non-namespaced object that defines a mapping from apriority class name to the integer value of the priority. The name is specifiedin the name field of the PriorityClass object’s metadata. The value isspecified in the required value field. The higher the value, the higher thepriority.

A PriorityClass object can have any 32-bit integer value smaller than or equalto 1 billion. Larger numbers are reserved for critical system Pods that shouldnot normally be preempted or evicted. A cluster admin should create onePriorityClass object for each such mapping that they want.

PriorityClass also has two optional fields: globalDefault and description.The globalDefault field indicates that the value of this PriorityClass shouldbe used for Pods without a priorityClassName. Only one PriorityClass withglobalDefault set to true can exist in the system. If there is noPriorityClass with globalDefault set, the priority of Pods with nopriorityClassName is zero.

The description field is an arbitrary string. It is meant to tell users of thecluster when they should use this PriorityClass.

Notes about PodPriority and existing clusters

• If you upgrade your existing cluster and enable this feature, the priorityof your existing Pods is effectively zero.

• Addition of a PriorityClass with globalDefault set to true does notchange the priorities of existing Pods. The value of such a PriorityClass isused only for Pods created after the PriorityClass is added.

• If you delete a PriorityClass, existing Pods that use the name of thedeleted PriorityClass remain unchanged, but you cannot create more Pods thatuse the name of the deleted PriorityClass.

Example PriorityClass

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for XYZ service pods only."

Non-preempting PriorityClasses (alpha)

1.15 adds the PreemptionPolicy field as an alpha feature.It is disabled by default in 1.15,and requires the NonPreemptingPriorityfeature gate to be enabled.

Pods with PreemptionPolicy: Never will be placed in the scheduling queueahead of lower-priority pods,but they cannot preempt other pods.A non-preempting pod waiting to be scheduled will stay in the scheduling queue,until sufficient resources are free,and it can be scheduled.Non-preempting pods,like other pods,are subject to scheduler back-off.This means that if the scheduler tries these pods and they cannot be scheduled,they will be retried with lower frequency,allowing other pods with lower priority to be scheduled before them.

Non-preempting pods may still be preempted by other,high-priority pods.

PreemptionPolicy defaults to PreemptLowerPriority,which will allow pods of that PriorityClass to preempt lower-priority pods(as is existing default behavior).If PreemptionPolicy is set to Never,pods in that PriorityClass will be non-preempting.

An example use case is for data science workloads.A user may submit a job that they want to be prioritized above other workloads,but do not wish to discard existing work by preempting running pods.The high priority job with PreemptionPolicy: Never will be scheduledahead of other queued pods,as soon as sufficient cluster resources “naturally” become free.

Example Non-preempting PriorityClass

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority-nonpreempting
value: 1000000
preemptionPolicy: Never
globalDefault: false
description: "This priority class will not cause other pods to be preempted."

Pod priority

After you have one or more PriorityClasses, you can create Pods that specify oneof those PriorityClass names in their specifications. The priority admissioncontroller uses the priorityClassName field and populates the integer value ofthe priority. If the priority class is not found, the Pod is rejected.

The following YAML is an example of a Pod configuration that uses thePriorityClass created in the preceding example. The priority admissioncontroller checks the specification and resolves the priority of the Pod to1000000.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: high-priority

Effect of Pod priority on scheduling order

In Kubernetes 1.9 and later, when Pod priority is enabled, scheduler orderspending Pods by their priority and a pending Pod is placed ahead of otherpending Pods with lower priority in the scheduling queue. As a result, thehigher priority Pod may be scheduled sooner than Pods with lower priority if itsscheduling requirements are met. If such Pod cannot be scheduled, scheduler willcontinue and tries to schedule other lower priority Pods.

Preemption

When Pods are created, they go to a queue and wait to be scheduled. Thescheduler picks a Pod from the queue and tries to schedule it on a Node. If noNode is found that satisfies all the specified requirements of the Pod,preemption logic is triggered for the pending Pod. Let’s call the pending Pod P.Preemption logic tries to find a Node where removal of one or more Pods withlower priority than P would enable P to be scheduled on that Node. If such aNode is found, one or more lower priority Pods get evicted from the Node. Afterthe Pods are gone, P can be scheduled on the Node.

User exposed information

When Pod P preempts one or more Pods on Node N, nominatedNodeName field of PodP’s status is set to the name of Node N. This field helps scheduler trackresources reserved for Pod P and also gives users information about preemptionsin their clusters.

Please note that Pod P is not necessarily scheduled to the “nominated Node”.After victim Pods are preempted, they get their graceful termination period. Ifanother node becomes available while scheduler is waiting for the victim Pods toterminate, scheduler will use the other node to schedule Pod P. As a resultnominatedNodeName and nodeName of Pod spec are not always the same. Also, ifscheduler preempts Pods on Node N, but then a higher priority Pod than Pod Parrives, scheduler may give Node N to the new higher priority Pod. In such acase, scheduler clears nominatedNodeName of Pod P. By doing this, schedulermakes Pod P eligible to preempt Pods on another Node.

Limitations of preemption

Graceful termination of preemption victims

When Pods are preempted, the victims get theirgraceful termination period.They have that much time to finish their work and exit. If they don’t, they arekilled. This graceful termination period creates a time gap between the pointthat the scheduler preempts Pods and the time when the pending Pod (P) can bescheduled on the Node (N). In the meantime, the scheduler keeps scheduling otherpending Pods. As victims exit or get terminated, the scheduler tries to schedulePods in the pending queue. Therefore, there is usually a time gap between thepoint that scheduler preempts victims and the time that Pod P is scheduled. Inorder to minimize this gap, one can set graceful termination period of lowerpriority Pods to zero or a small number.

PodDisruptionBudget is supported, but not guaranteed!

A Pod Disruption Budget (PDB)allows application owners to limit the number of Pods of a replicated applicationthat are down simultaneously from voluntary disruptions. Kubernetes 1.9 supportsPDB when preempting Pods, but respecting PDB is best effort. The Scheduler triesto find victims whose PDB are not violated by preemption, but if no such victimsare found, preemption will still happen, and lower priority Pods will be removeddespite their PDBs being violated.

Inter-Pod affinity on lower-priority Pods

A Node is considered for preemption only when the answer to this question isyes: “If all the Pods with lower priority than the pending Pod are removed fromthe Node, can the pending Pod be scheduled on the Node?”

Note: Preemption does not necessarily remove all lower-priorityPods. If the pending Pod can be scheduled by removing fewer than alllower-priority Pods, then only a portion of the lower-priority Pods are removed.Even so, the answer to the preceding question must be yes. If the answer is no,the Node is not considered for preemption.

If a pending Pod has inter-pod affinity to one or more of the lower-priorityPods on the Node, the inter-Pod affinity rule cannot be satisfied in the absenceof those lower-priority Pods. In this case, the scheduler does not preempt anyPods on the Node. Instead, it looks for another Node. The scheduler might find asuitable Node or it might not. There is no guarantee that the pending Pod can bescheduled.

Our recommended solution for this problem is to create inter-Pod affinity onlytowards equal or higher priority Pods.

Cross node preemption

Suppose a Node N is being considered for preemption so that a pending Pod P canbe scheduled on N. P might become feasible on N only if a Pod on another Node ispreempted. Here’s an example:

• Pod P is being considered for Node N.
• Pod Q is running on another Node in the same Zone as Node N.
• Pod P has Zone-wide anti-affinity with Pod Q (topologyKey:failure-domain.beta.kubernetes.io/zone).
• There are no other cases of anti-affinity between Pod P and other Pods inthe Zone.
• In order to schedule Pod P on Node N, Pod Q can be preempted, but schedulerdoes not perform cross-node preemption. So, Pod P will be deemedunschedulable on Node N.

If Pod Q were removed from its Node, the Pod anti-affinity violation would begone, and Pod P could possibly be scheduled on Node N.

We may consider adding cross Node preemption in future versions if there isenough demand and if we find an algorithm with reasonable performance.

Debugging Pod Priority and Preemption

Pod Priority and Preemption is a major feature that could potentially disruptPod scheduling if it has bugs.

Potential problems caused by Priority and Preemption

The followings are some of the potential problems that could be caused by bugsin the implementation of the feature. This list is not exhaustive.

Pods are preempted unnecessarily

Preemption removes existing Pods from a cluster under resource pressure to makeroom for higher priority pending Pods. If a user gives high priorities tocertain Pods by mistake, these unintentional high priority Pods may causepreemption in the cluster. As mentioned above, Pod priority is specified bysetting the priorityClassName field of podSpec. The integer value ofpriority is then resolved and populated to the priority field of podSpec.

To resolve the problem, priorityClassName of the Pods must be changed to uselower priority classes or should be left empty. Empty priorityClassName isresolved to zero by default.

When a Pod is preempted, there will be events recorded for the preempted Pod.Preemption should happen only when a cluster does not have enough resources fora Pod. In such cases, preemption happens only when the priority of the pendingPod (preemptor) is higher than the victim Pods. Preemption must not happen whenthere is no pending Pod, or when the pending Pods have equal or lower prioritythan the victims. If preemption happens in such scenarios, please file an issue.

Pods are preempted, but the preemptor is not scheduled

When pods are preempted, they receive their requested graceful terminationperiod, which is by default 30 seconds, but it can be any different value asspecified in the PodSpec. If the victim Pods do not terminate within this period,they are force-terminated. Once all the victims go away, the preemptor Pod canbe scheduled.

While the preemptor Pod is waiting for the victims to go away, a higher priorityPod may be created that fits on the same node. In this case, the scheduler willschedule the higher priority Pod instead of the preemptor.

In the absence of such a higher priority Pod, we expect the preemptor Pod to bescheduled after the graceful termination period of the victims is over.

Higher priority Pods are preempted before lower priority pods

The scheduler tries to find nodes that can run a pending Pod and if no node isfound, it tries to remove Pods with lower priority from one node to make roomfor the pending pod. If a node with low priority Pods is not feasible to run thepending Pod, the scheduler may choose another node with higher priority Pods(compared to the Pods on the other node) for preemption. The victims must stillhave lower priority than the preemptor Pod.

When there are multiple nodes available for preemption, the scheduler tries tochoose the node with a set of Pods with lowest priority. However, if such Podshave PodDisruptionBudget that would be violated if they are preempted then thescheduler may choose another node with higher priority Pods.

When multiple nodes exist for preemption and none of the above scenarios apply,we expect the scheduler to choose a node with the lowest priority. If that isnot the case, it may indicate a bug in the scheduler.

Interactions of Pod priority and QoS

Pod priority andQoSare two orthogonal features with few interactions and no default restrictions onsetting the priority of a Pod based on its QoS classes. The scheduler’spreemption logic does not consider QoS when choosing preemption targets.Preemption considers Pod priority and attempts to choose a set of targets withthe lowest priority. Higher-priority Pods are considered for preemption only ifthe removal of the lowest priority Pods is not sufficient to allow the schedulerto schedule the preemptor Pod, or if the lowest priority Pods are protected byPodDisruptionBudget.

The only component that considers both QoS and Pod priority isKubelet out-of-resource eviction.The kubelet ranks Pods for eviction first by whether or not their usage of thestarved resource exceeds requests, then by Priority, and then by the consumptionof the starved compute resource relative to the Pods’ scheduling requests.SeeEvicting end-user podsfor more details. Kubelet out-of-resource eviction does not evict Pods whoseusage does not exceed their requests. If a Pod with lower priority is notexceeding its requests, it won’t be evicted. Another Pod with higher prioritythat exceeds its requests may be evicted.

Feedback

Was this page helpful?

Thanks for the feedback. If you have a specific, answerable question about how to use Kubernetes, ask it onStack Overflow.Open an issue in the GitHub repo if you want toreport a problemorsuggest an improvement.