Question: What is Note:?

Answer: S o m e   w o r k l o a d s   m a y   r e q u i r e   a   l o w e r   a m o u n t   o f   s y s c a l l   r e s t r i c t i o n s   t h a n   o t h e r s .   T h i s   m e a n s   t h a t   t h e y   c a n   f a i l   d u r i n g   r u n t i m e   e v e n   w i t h   t h e   R u n t i m e D e f a u l t   p r o f i l e .   T o   m i t i g a t e   s u c h   a   f a i l u r e ,   y o u   c a n :   R u n   t h e   w o r k l o a d   e x p l i c i t l y   a s   U n c o n f i n e d . D i s a b l e   t h e   S e c c o m p D e f a u l t   f e a t u r e   f o r   t h e   n o d e s .   A l s o   m a k i n g   s u r e   t h a t   w o r k l o a d s   g e t   s c h e d u l e d   o n   n o d e s   w h e r e   t h e   f e a t u r e   i s   d i s a b l e d . C r e a t e   a   c u s t o m   s e c c o m p   p r o f i l e   f o r   t h e   w o r k l o a d .   I f   y o u   w e r e   i n t r o d u c i n g   t h i s   f e a t u r e   i n t o   p r o d u c t i o n - l i k e   c l u s t e r ,   t h e   K u b e r n e t e s   p r o j e c t   r e c o m m e n d s   t h a t   y o u   e n a b l e   t h i s   f e a t u r e   g a t e   o n   a   s u b s e t   o f   y o u r   n o d e s   a n d   t h e n   t e s t   w o r k l o a d   e x e c u t i o n   b e f o r e   r o l l i n g   t h e   c h a n g e   o u t   c l u s t e r - w i d e .   Y o u   c a n   f i n d   m o r e   d e t a i l e d   i n f o r m a t i o n   a b o u t   a   p o s s i b l e   u p g r a d e   a n d   d o w n g r a d e   s t r a t e g y   i n   t h e   r e l a t e d   K u b e r n e t e s   E n h a n c e m e n t   P r o p o s a l   ( K E P ) :   E n a b l e   s e c c o m p   b y   d e f a u l t .   K u b e r n e t e s   1 . 3 4   l e t s   y o u   c o n f i g u r e   t h e   s e c c o m p   p r o f i l e   t h a t   a p p l i e s   w h e n   t h e   s p e c   f o r   a   P o d   d o e s n ' t   d e f i n e   a   s p e c i f i c   s e c c o m p   p r o f i l e .   H o w e v e r ,   y o u   s t i l l   n e e d   t o   e n a b l e


Question: What is What's next?

Answer: Y o u   c a n   l e a r n   m o r e   a b o u t   L i n u x   s e c c o m p :   A   s e c c o m p   O v e r v i e w S e c c o m p   S e c u r i t y   P r o f i l e s   f o r   D o c k e r


Question: What is Decrypt Confidential Data that is Already Encrypted at Rest?

Answer: Decrypt Confidential Data that is Already Encrypted at RestAll of the APIs in Kubernetes that let you write persistent API resource data support at-rest encryption. For example, you can enable at-rest encryption for Secrets. This at-rest encryption is additional to any system-level encryption for the etcd cluster or for the filesystem(s) on hosts where you are running the kube-apiserver.This page shows how to switch from encryption of API data at rest, so that API data are stored unencrypted. You might want to do this to improve performance; usually, though, if it was a good idea to encrypt some data, it's also a good idea to leave them encrypted.Note:This task covers encryption for resource data stored using the Kubernetes API. For example, you can encrypt Secret objects, including the key-value data they contain.If you wanted to manage encryption for data in filesystems that are mounted into containers, you instead need to either:use a storage integration that provides encrypted volumesencrypt the data within your own applicationBefore you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesThis task assumes that you are running the Kubernetes API server as a static pod on each control plane node.Your cluster's control plane must use etcd v3.x (major version 3, any minor version).To encrypt a custom resource, your cluster must be running Kubernetes v1.26 or newer.You should have some API data that are already encrypted.To check the version, enter kubectl version.Determine whether encryption at rest is already enabledBy default, the API server uses an identity provider that stores plain-text representations of resources. The default identity provider does not provide any confidentiality protection.The kube-apiserver process accepts an argument --encryption-provider-config that specifies a path to a configuration file. The contents of that file, if you specify one, control how Kubernetes API data is encrypted in etcd. If it is not specified, you do not have encryption at rest enabled.The format of that configuration file is YAML, representing a configuration API kind named EncryptionConfiguration. You can see an example configuration in Encryption at rest configuration.If --encryption-provider-config is set, check which resources (such as secrets) are configured for encryption, and what provider is used. Make sure that the preferred provider for that resource type is not identity; you only set identity (no encryption) as default when you want to disable encryption at rest. Verify that the first-listed provider for a resource is something other than identity, which means that an


Question: How does Decrypt Confidential Data that is Already Encrypted at Rest work?

Answer: Decrypt Confidential Data that is Already Encrypted at RestAll of the APIs in Kubernetes that let you write persistent API resource data support at-rest encryption. For example, you can enable at-rest encryption for Secrets. This at-rest encryption is additional to any system-level encryption for the etcd cluster or for the filesystem(s) on hosts where you are running the kube-apiserver.This page shows how to switch from encryption of API data at rest, so that API data are stored unencrypted. You might want to do this to improve performance; usually, though, if it was a good idea to encrypt some data, it's also a good idea to leave them encrypted.Note:This task covers encryption for resource data stored using the Kubernetes API. For example, you can encrypt Secret objects, including the key-value data they contain.If you wanted to manage encryption for data in filesystems that are mounted into containers, you instead need to either:use a storage integration that provides encrypted volumesencrypt the data within your own applicationBefore you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesThis task assumes that you are running the Kubernetes API server as a static pod on each control plane node.Your cluster's control plane must use etcd v3.x (major version 3, any minor version).To encrypt a custom resource, your cluster must be running Kubernetes v1.26 or newer.You should have some API data that are already encrypted.To check the version, enter kubectl version.Determine whether encryption at rest is already enabledBy default, the API server uses an identity provider that stores plain-text representations of resources. The default identity provider does not provide any confidentiality protection.The kube-apiserver process accepts an argument --encryption-provider-config that specifies a path to a configuration file. The contents of that file, if you specify one, control how Kubernetes API data is encrypted in etcd. If it is not specified, you do not have encryption at rest enabled.The format of that configuration file is YAML, representing a configuration API kind named EncryptionConfiguration. You can see an example configuration in Encryption at rest configuration.If --encryption-provider-config is set, check which resources (such as secrets) are configured for encryption, and what provider is used. Make sure that the preferred provider for that resource type is not identity; you only set identity (no encryption) as default when you want to disable encryption at rest. Verify that the first-listed provider for a resource is something other than identity, which means that an


Question: When would you use Decrypt Confidential Data that is Already Encrypted at Rest?

Answer: Decrypt Confidential Data that is Already Encrypted at RestAll of the APIs in Kubernetes that let you write persistent API resource data support at-rest encryption. For example, you can enable at-rest encryption for Secrets. This at-rest encryption is additional to any system-level encryption for the etcd cluster or for the filesystem(s) on hosts where you are running the kube-apiserver.This page shows how to switch from encryption of API data at rest, so that API data are stored unencrypted. You might want to do this to improve performance; usually, though, if it was a good idea to encrypt some data, it's also a good idea to leave them encrypted.Note:This task covers encryption for resource data stored using the Kubernetes API. For example, you can encrypt Secret objects, including the key-value data they contain.If you wanted to manage encryption for data in filesystems that are mounted into containers, you instead need to either:use a storage integration that provides encrypted volumesencrypt the data within your own applicationBefore you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesThis task assumes that you are running the Kubernetes API server as a static pod on each control plane node.Your cluster's control plane must use etcd v3.x (major version 3, any minor version).To encrypt a custom resource, your cluster must be running Kubernetes v1.26 or newer.You should have some API data that are already encrypted.To check the version, enter kubectl version.Determine whether encryption at rest is already enabledBy default, the API server uses an identity provider that stores plain-text representations of resources. The default identity provider does not provide any confidentiality protection.The kube-apiserver process accepts an argument --encryption-provider-config that specifies a path to a configuration file. The contents of that file, if you specify one, control how Kubernetes API data is encrypted in etcd. If it is not specified, you do not have encryption at rest enabled.The format of that configuration file is YAML, representing a configuration API kind named EncryptionConfiguration. You can see an example configuration in Encryption at rest configuration.If --encryption-provider-config is set, check which resources (such as secrets) are configured for encryption, and what provider is used. Make sure that the preferred provider for that resource type is not identity; you only set identity (no encryption) as default when you want to disable encryption at rest. Verify that the first-listed provider for a resource is something other than identity, which means that an


Question: What are the benefits of Decrypt Confidential Data that is Already Encrypted at Rest?

Answer: Decrypt Confidential Data that is Already Encrypted at RestAll of the APIs in Kubernetes that let you write persistent API resource data support at-rest encryption. For example, you can enable at-rest encryption for Secrets. This at-rest encryption is additional to any system-level encryption for the etcd cluster or for the filesystem(s) on hosts where you are running the kube-apiserver.This page shows how to switch from encryption of API data at rest, so that API data are stored unencrypted. You might want to do this to improve performance; usually, though, if it was a good idea to encrypt some data, it's also a good idea to leave them encrypted.Note:This task covers encryption for resource data stored using the Kubernetes API. For example, you can encrypt Secret objects, including the key-value data they contain.If you wanted to manage encryption for data in filesystems that are mounted into containers, you instead need to either:use a storage integration that provides encrypted volumesencrypt the data within your own applicationBefore you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesThis task assumes that you are running the Kubernetes API server as a static pod on each control plane node.Your cluster's control plane must use etcd v3.x (major version 3, any minor version).To encrypt a custom resource, your cluster must be running Kubernetes v1.26 or newer.You should have some API data that are already encrypted.To check the version, enter kubectl version.Determine whether encryption at rest is already enabledBy default, the API server uses an identity provider that stores plain-text representations of resources. The default identity provider does not provide any confidentiality protection.The kube-apiserver process accepts an argument --encryption-provider-config that specifies a path to a configuration file. The contents of that file, if you specify one, control how Kubernetes API data is encrypted in etcd. If it is not specified, you do not have encryption at rest enabled.The format of that configuration file is YAML, representing a configuration API kind named EncryptionConfiguration. You can see an example configuration in Encryption at rest configuration.If --encryption-provider-config is set, check which resources (such as secrets) are configured for encryption, and what provider is used. Make sure that the preferred provider for that resource type is not identity; you only set identity (no encryption) as default when you want to disable encryption at rest. Verify that the first-listed provider for a resource is something other than identity, which means that an


Question: What is Decrypt Confidential Data that is Already Encrypted at Rest?

Answer: A l l   o f   t h e   A P I s   i n   K u b e r n e t e s   t h a t   l e t   y o u   w r i t e   p e r s i s t e n t   A P I   r e s o u r c e   d a t a   s u p p o r t   a t - r e s t   e n c r y p t i o n .   F o r   e x a m p l e ,   y o u   c a n   e n a b l e   a t - r e s t   e n c r y p t i o n   f o r   S e c r e t s .   T h i s   a t - r e s t   e n c r y p t i o n   i s   a d d i t i o n a l   t o   a n y   s y s t e m - l e v e l   e n c r y p t i o n   f o r   t h e   e t c d   c l u s t e r   o r   f o r   t h e   f i l e s y s t e m ( s )   o n   h o s t s   w h e r e   y o u   a r e   r u n n i n g   t h e   k u b e - a p i s e r v e r .   T h i s   p a g e   s h o w s   h o w   t o   s w i t c h   f r o m   e n c r y p t i o n   o f   A P I   d a t a   a t   r e s t ,   s o   t h a t   A P I   d a t a   a r e   s t o r e d   u n e n c r y p t e d .   Y o u   m i g h t   w a n t   t o   d o   t h i s   t o   i m p r o v e   p e r f o r m a n c e ;   u s u a l l y ,   t h o u g h ,   i f   i t   w a s   a   g o o d   i d e a   t o   e n c r y p t   s o m e   d a t a ,   i t ' s   a l s o   a   g o o d   i d e a   t o   l e a v e   t h e m   e n c r y p t e d .


Question: What is Note:?

Answer: T h i s   t a s k   c o v e r s   e n c r y p t i o n   f o r   r e s o u r c e   d a t a   s t o r e d   u s i n g   t h e   K u b e r n e t e s   A P I .   F o r   e x a m p l e ,   y o u   c a n   e n c r y p t   S e c r e t   o b j e c t s ,   i n c l u d i n g   t h e   k e y - v a l u e   d a t a   t h e y   c o n t a i n .   I f   y o u   w a n t e d   t o   m a n a g e   e n c r y p t i o n   f o r   d a t a   i n   f i l e s y s t e m s   t h a t   a r e   m o u n t e d   i n t o   c o n t a i n e r s ,   y o u   i n s t e a d   n e e d   t o   e i t h e r :   u s e   a   s t o r a g e   i n t e g r a t i o n   t h a t   p r o v i d e s   e n c r y p t e d   v o l u m e s e n c r y p t   t h e   d a t a   w i t h i n   y o u r   o w n   a p p l i c a t i o n


Question: What is Before you begin?

Answer: Y o u   n e e d   t o   h a v e   a   K u b e r n e t e s   c l u s t e r ,   a n d   t h e   k u b e c t l   c o m m a n d - l i n e   t o o l   m u s t   b e   c o n f i g u r e d   t o   c o m m u n i c a t e   w i t h   y o u r   c l u s t e r .   I t   i s   r e c o m m e n d e d   t o   r u n   t h i s   t u t o r i a l   o n   a   c l u s t e r   w i t h   a t   l e a s t   t w o   n o d e s   t h a t   a r e   n o t   a c t i n g   a s   c o n t r o l   p l a n e   h o s t s .   I f   y o u   d o   n o t   a l r e a d y   h a v e   a   c l u s t e r ,   y o u   c a n   c r e a t e   o n e   b y   u s i n g   m i n i k u b e   o r   y o u   c a n   u s e   o n e   o f   t h e s e   K u b e r n e t e s   p l a y g r o u n d s : i x i m i u z   L a b s K i l l e r c o d a K o d e K l o u d P l a y   w i t h   K u b e r n e t e s T h i s   t a s k   a s s u m e s   t h a t   y o u   a r e   r u n n i n g   t h e   K u b e r n e t e s   A P I   s e r v e r   a s   a   s t a t i c   p o d   o n   e a c h   c o n t r o l   p l a n e   n o d e . Y o u r   c l u s t e r ' s   c o n t r o l   p l a n e   m u s t   u s e   e t c d   v 3 . x   ( m a j o r   v e r s i o n   3 ,   a n y   m i n o r   v e r s i o n ) . T o   e n c r y p t   a   c u s t o m   r e s o u r c e ,   y o u r   c l u s t e r   m u s t   b e   r u n n i n g   K u b e r n e t e s   v 1 . 2 6   o r   n e w e r . Y o u   s h o u l d   h a v e   s o m e   A P I   d a t a   t h a t   a r e   a l r e a d y   e n c r y p t e d .   Y o u   n e e d   t o   h a v e   a   K u b e r n e t e s   c l u s t e r ,   a n d   t h e   k u b e c t l   c o m m a n d - l i n e   t o o l   m u s t   b e   c o n f i g u r e d   t o   c o m m u n i c a t e   w i t h   y o u r   c l u s t e r .   I t   i s   r e c o m m e n d e d   t o   r u n   t h i s   t u t o r i a l   o n   a   c l u s t e r   w i t h   a t   l e a s t   t w o   n o d e s   t h a t   a r e   n o t   a c t i n g   a s   c o n t r o l   p l a n e   h o


Question: What is Determine whether encryption at rest is already enabled?

Answer: B y   d e f a u l t ,   t h e   A P I   s e r v e r   u s e s   a n   i d e n t i t y   p r o v i d e r   t h a t   s t o r e s   p l a i n - t e x t   r e p r e s e n t a t i o n s   o f   r e s o u r c e s .   T h e   d e f a u l t   i d e n t i t y   p r o v i d e r   d o e s   n o t   p r o v i d e   a n y   c o n f i d e n t i a l i t y   p r o t e c t i o n .   T h e   k u b e - a p i s e r v e r   p r o c e s s   a c c e p t s   a n   a r g u m e n t   - - e n c r y p t i o n - p r o v i d e r - c o n f i g   t h a t   s p e c i f i e s   a   p a t h   t o   a   c o n f i g u r a t i o n   f i l e .   T h e   c o n t e n t s   o f   t h a t   f i l e ,   i f   y o u   s p e c i f y   o n e ,   c o n t r o l   h o w   K u b e r n e t e s   A P I   d a t a   i s   e n c r y p t e d   i n   e t c d .   I f   i t   i s   n o t   s p e c i f i e d ,   y o u   d o   n o t   h a v e   e n c r y p t i o n   a t   r e s t   e n a b l e d .   T h e   f o r m a t   o f   t h a t   c o n f i g u r a t i o n   f i l e   i s   Y A M L ,   r e p r e s e n t i n g   a   c o n f i g u r a t i o n   A P I   k i n d   n a m e d   E n c r y p t i o n C o n f i g u r a t i o n .   Y o u   c a n   s e e   a n   e x a m p l e   c o n f i g u r a t i o n   i n   E n c r y p t i o n   a t   r e s t   c o n f i g u r a t i o n .   I f   - - e n c r y p t i o n - p r o v i d e r - c o n f i g   i s   s e t ,   c h e c k   w h i c h   r e s o u r c e s   ( s u c h   a s   s e c r e t s )   a r e   c o n f i g u r e d   f o r   e n c r y p t i o n ,   a n d   w h a t   p r o v i d e r   i s   u s e d .   M a k e   s u r e   t h a t   t h e   p r e f e r r e d   p r o v i d e r   f o r   t h a t   r e s o u r c e   t y p e   i s   n o t   i d e n t i t y ;   y o u   o n l y   s e t   i d e n t i t y   ( n o   e n c r y p t i o n )   a s   d e f a u l t   w h e n   y o u   w a n t   t o   d i s a b l e   e n c r y p t i o n   a t   r e s t .   V e r i f y   t h a t   t h e   f i r s t - l


Question: What is Decrypt all data?

Answer: T h i s   e x a m p l e   s h o w s   h o w   t o   s t o p   e n c r y p t i n g   t h e   S e c r e t   A P I   a t   r e s t .   I f   y o u   a r e   e n c r y p t i n g   o t h e r   A P I   k i n d s ,   a d j u s t   t h e   s t e p s   t o   m a t c h .


Question: What is Locate the encryption configuration file?

Answer: F i r s t ,   f i n d   t h e   A P I   s e r v e r   c o n f i g u r a t i o n   f i l e s .   O n   e a c h   c o n t r o l   p l a n e   n o d e ,   s t a t i c   P o d   m a n i f e s t   f o r   t h e   k u b e - a p i s e r v e r   s p e c i f i e s   a   c o m m a n d   l i n e   a r g u m e n t ,   - - e n c r y p t i o n - p r o v i d e r - c o n f i g .   Y o u   a r e   l i k e l y   t o   f i n d   t h a t   t h i s   f i l e   i s   m o u n t e d   i n t o   t h e   s t a t i c   P o d   u s i n g   a   h o s t P a t h   v o l u m e   m o u n t .   O n c e   y o u   l o c a t e   t h e   v o l u m e   y o u   c a n   f i n d   t h e   f i l e   o n   t h e   n o d e   f i l e s y s t e m   a n d   i n s p e c t   i t .


Question: What is Configure the API server to decrypt objects?

Answer: T o   d i s a b l e   e n c r y p t i o n   a t   r e s t ,   p l a c e   t h e   i d e n t i t y   p r o v i d e r   a s   t h e   f i r s t   e n t r y   i n   y o u r   e n c r y p t i o n   c o n f i g u r a t i o n   f i l e .   F o r   e x a m p l e ,   i f   y o u r   e x i s t i n g   E n c r y p t i o n C o n f i g u r a t i o n   f i l e   r e a d s :   - - -   a p i V e r s i o n :   a p i s e r v e r . c o n f i g . k 8 s . i o / v 1   k i n d :   E n c r y p t i o n C o n f i g u r a t i o n   r e s o u r c e s :   -   r e s o u r c e s :   -   s e c r e t s   p r o v i d e r s :   -   a e s c b c :   k e y s :   #   D o   n o t   u s e   t h i s   ( i n v a l i d )   e x a m p l e   k e y   f o r   e n c r y p t i o n   -   n a m e :   e x a m p l e   s e c r e t :   2 K f Z g d i q 2 K 0 g 2 Y r Y p y D Y s 9 m F 2 L P Z h Q = =   t h e n   c h a n g e   i t   t o :   - - -   a p i V e r s i o n :   a p i s e r v e r . c o n f i g . k 8 s . i o / v 1   k i n d :   E n c r y p t i o n C o n f i g u r a t i o n   r e s o u r c e s :   -   r e s o u r c e s :   -   s e c r e t s   p r o v i d e r s :   -   i d e n t i t y :   { }   #   a d d   t h i s   l i n e   -   a e s c b c :   k e y s :   -   n a m e :   e x a m p l e   s e c r e t :   2 K f Z g d i q 2 K 0 g 2 Y r Y p y D Y s 9 m F 2 L P Z h Q = =   a n d   r e s t a r t   t h e   k u b e - a p i s e r v e r   P o d   o n   t h i s   n o d e .


Question: What is Reconfigure other control plane hosts?

Answer: I f   y o u   h a v e   m u l t i p l e   A P I   s e r v e r s   i n   y o u r   c l u s t e r ,   y o u   s h o u l d   d e p l o y   t h e   c h a n g e s   i n   t u r n   t o   e a c h   A P I   s e r v e r .   M a k e   s u r e   t h a t   y o u   u s e   t h e   s a m e   e n c r y p t i o n   c o n f i g u r a t i o n   o n   e a c h   c o n t r o l   p l a n e   h o s t .


Question: What is Force decryption?

Answer: T h e n   r u n   t h e   f o l l o w i n g   c o m m a n d   t o   f o r c e   d e c r y p t i o n   o f   a l l   S e c r e t s :   #   I f   y o u   a r e   d e c r y p t i n g   a   d i f f e r e n t   k i n d   o f   o b j e c t ,   c h a n g e   " s e c r e t s "   t o   m a t c h .   k u b e c t l   g e t   s e c r e t s   - - a l l - n a m e s p a c e s   - o   j s o n   |   k u b e c t l   r e p l a c e   - f   -   O n c e   y o u   h a v e   r e p l a c e d   a l l   e x i s t i n g   e n c r y p t e d   r e s o u r c e s   w i t h   b a c k i n g   d a t a   t h a t   d o n ' t   u s e   e n c r y p t i o n ,   y o u   c a n   r e m o v e   t h e   e n c r y p t i o n   s e t t i n g s   f r o m   t h e   k u b e - a p i s e r v e r .   T h e   c o m m a n d   l i n e   o p t i o n s   t o   r e m o v e   a r e :   - - e n c r y p t i o n - p r o v i d e r - c o n f i g - - e n c r y p t i o n - p r o v i d e r - c o n f i g - a u t o m a t i c - r e l o a d   R e s t a r t   t h e   k u b e - a p i s e r v e r   P o d   a g a i n   t o   a p p l y   t h e   n e w   c o n f i g u r a t i o n .


Question: What is Reconfigure other control plane hosts?

Answer: I f   y o u   h a v e   m u l t i p l e   A P I   s e r v e r s   i n   y o u r   c l u s t e r ,   y o u   s h o u l d   a g a i n   d e p l o y   t h e   c h a n g e s   i n   t u r n   t o   e a c h   A P I   s e r v e r .   M a k e   s u r e   t h a t   y o u   u s e   t h e   s a m e   e n c r y p t i o n   c o n f i g u r a t i o n   o n   e a c h   c o n t r o l   p l a n e   h o s t .


Question: What is What's next?

Answer: L e a r n   m o r e   a b o u t   t h e   E n c r y p t i o n C o n f i g u r a t i o n   c o n f i g u r a t i o n   A P I   ( v 1 ) .


Question: What is Kubelet Configuration (v1)?

Answer: Kubelet Configuration (v1)Resource TypesCredentialProviderConfigCredentialProviderConfigCredentialProviderConfig is the configuration containing information about each exec credential provider. Kubelet reads this configuration from disk and enables each provider as specified by the CredentialProvider type.FieldDescriptionapiVersionstringkubelet.config.k8s.io/v1kindstringCredentialProviderConfigproviders [Required][]CredentialProviderproviders is a list of credential provider plugins that will be enabled by the kubelet. Multiple providers may match against a single image, in which case credentials from all providers will be returned to the kubelet. If multiple providers are called for a single image, the results are combined. If providers return overlapping auth keys, the value from the provider earlier in this list is attempted first.CredentialProviderAppears in:CredentialProviderConfigCredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only invoked when an image being pulled matches the images handled by the plugin (see matchImages).FieldDescriptionname [Required]stringname is the required name of the credential provider. It must match the name of the provider executable as seen by the kubelet. The executable must be in the kubelet's bin directory (set by the --image-credential-provider-bin-dir flag). Required to be unique across all providers.matchImages [Required][]stringmatchImages is a required list of strings used to match against images in order to determine if this provider should be invoked. If one of the strings matches the requested image from the kubelet, the plugin will be invoked and given a chance to provide credentials. Images are expected to contain the registry domain and URL path.Each entry in matchImages is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like '*.k8s.io' or 'k8s.*.io', and top-level-domains such as 'k8s.*'. Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match a single subdomain segment, so *.io does not match *.k8s.io.A match exists between an image and a matchImage when all of the below are true:Both contain the same number of domain parts and each part matches.The URL path of an imageMatch must be a prefix of the target image URL path.If the imageMatch contains a port, then the port must match in the image as well.Example values of matchImages:123456789.dkr.ecr.us-east-1.amazonaws.com*.azurecr.iogcr.io*.*.registry.ioregistry.io:8080/pathdefaultCacheDuration [Required]meta/v1.DurationdefaultCacheDuration is the default duration the plugin will cache credentials in-memory if a cache duration is not provided in the plugin response. This field is required.apiVersion [Required]stringRequired input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse MUST use the same encoding version as the input. Cu


Question: How does Kubelet Configuration (v1) work?

Answer: Kubelet Configuration (v1)Resource TypesCredentialProviderConfigCredentialProviderConfigCredentialProviderConfig is the configuration containing information about each exec credential provider. Kubelet reads this configuration from disk and enables each provider as specified by the CredentialProvider type.FieldDescriptionapiVersionstringkubelet.config.k8s.io/v1kindstringCredentialProviderConfigproviders [Required][]CredentialProviderproviders is a list of credential provider plugins that will be enabled by the kubelet. Multiple providers may match against a single image, in which case credentials from all providers will be returned to the kubelet. If multiple providers are called for a single image, the results are combined. If providers return overlapping auth keys, the value from the provider earlier in this list is attempted first.CredentialProviderAppears in:CredentialProviderConfigCredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only invoked when an image being pulled matches the images handled by the plugin (see matchImages).FieldDescriptionname [Required]stringname is the required name of the credential provider. It must match the name of the provider executable as seen by the kubelet. The executable must be in the kubelet's bin directory (set by the --image-credential-provider-bin-dir flag). Required to be unique across all providers.matchImages [Required][]stringmatchImages is a required list of strings used to match against images in order to determine if this provider should be invoked. If one of the strings matches the requested image from the kubelet, the plugin will be invoked and given a chance to provide credentials. Images are expected to contain the registry domain and URL path.Each entry in matchImages is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like '*.k8s.io' or 'k8s.*.io', and top-level-domains such as 'k8s.*'. Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match a single subdomain segment, so *.io does not match *.k8s.io.A match exists between an image and a matchImage when all of the below are true:Both contain the same number of domain parts and each part matches.The URL path of an imageMatch must be a prefix of the target image URL path.If the imageMatch contains a port, then the port must match in the image as well.Example values of matchImages:123456789.dkr.ecr.us-east-1.amazonaws.com*.azurecr.iogcr.io*.*.registry.ioregistry.io:8080/pathdefaultCacheDuration [Required]meta/v1.DurationdefaultCacheDuration is the default duration the plugin will cache credentials in-memory if a cache duration is not provided in the plugin response. This field is required.apiVersion [Required]stringRequired input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse MUST use the same encoding version as the input. Cu


Question: When would you use Kubelet Configuration (v1)?

Answer: Kubelet Configuration (v1)Resource TypesCredentialProviderConfigCredentialProviderConfigCredentialProviderConfig is the configuration containing information about each exec credential provider. Kubelet reads this configuration from disk and enables each provider as specified by the CredentialProvider type.FieldDescriptionapiVersionstringkubelet.config.k8s.io/v1kindstringCredentialProviderConfigproviders [Required][]CredentialProviderproviders is a list of credential provider plugins that will be enabled by the kubelet. Multiple providers may match against a single image, in which case credentials from all providers will be returned to the kubelet. If multiple providers are called for a single image, the results are combined. If providers return overlapping auth keys, the value from the provider earlier in this list is attempted first.CredentialProviderAppears in:CredentialProviderConfigCredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only invoked when an image being pulled matches the images handled by the plugin (see matchImages).FieldDescriptionname [Required]stringname is the required name of the credential provider. It must match the name of the provider executable as seen by the kubelet. The executable must be in the kubelet's bin directory (set by the --image-credential-provider-bin-dir flag). Required to be unique across all providers.matchImages [Required][]stringmatchImages is a required list of strings used to match against images in order to determine if this provider should be invoked. If one of the strings matches the requested image from the kubelet, the plugin will be invoked and given a chance to provide credentials. Images are expected to contain the registry domain and URL path.Each entry in matchImages is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like '*.k8s.io' or 'k8s.*.io', and top-level-domains such as 'k8s.*'. Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match a single subdomain segment, so *.io does not match *.k8s.io.A match exists between an image and a matchImage when all of the below are true:Both contain the same number of domain parts and each part matches.The URL path of an imageMatch must be a prefix of the target image URL path.If the imageMatch contains a port, then the port must match in the image as well.Example values of matchImages:123456789.dkr.ecr.us-east-1.amazonaws.com*.azurecr.iogcr.io*.*.registry.ioregistry.io:8080/pathdefaultCacheDuration [Required]meta/v1.DurationdefaultCacheDuration is the default duration the plugin will cache credentials in-memory if a cache duration is not provided in the plugin response. This field is required.apiVersion [Required]stringRequired input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse MUST use the same encoding version as the input. Cu


Question: What are the benefits of Kubelet Configuration (v1)?

Answer: Kubelet Configuration (v1)Resource TypesCredentialProviderConfigCredentialProviderConfigCredentialProviderConfig is the configuration containing information about each exec credential provider. Kubelet reads this configuration from disk and enables each provider as specified by the CredentialProvider type.FieldDescriptionapiVersionstringkubelet.config.k8s.io/v1kindstringCredentialProviderConfigproviders [Required][]CredentialProviderproviders is a list of credential provider plugins that will be enabled by the kubelet. Multiple providers may match against a single image, in which case credentials from all providers will be returned to the kubelet. If multiple providers are called for a single image, the results are combined. If providers return overlapping auth keys, the value from the provider earlier in this list is attempted first.CredentialProviderAppears in:CredentialProviderConfigCredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only invoked when an image being pulled matches the images handled by the plugin (see matchImages).FieldDescriptionname [Required]stringname is the required name of the credential provider. It must match the name of the provider executable as seen by the kubelet. The executable must be in the kubelet's bin directory (set by the --image-credential-provider-bin-dir flag). Required to be unique across all providers.matchImages [Required][]stringmatchImages is a required list of strings used to match against images in order to determine if this provider should be invoked. If one of the strings matches the requested image from the kubelet, the plugin will be invoked and given a chance to provide credentials. Images are expected to contain the registry domain and URL path.Each entry in matchImages is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like '*.k8s.io' or 'k8s.*.io', and top-level-domains such as 'k8s.*'. Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match a single subdomain segment, so *.io does not match *.k8s.io.A match exists between an image and a matchImage when all of the below are true:Both contain the same number of domain parts and each part matches.The URL path of an imageMatch must be a prefix of the target image URL path.If the imageMatch contains a port, then the port must match in the image as well.Example values of matchImages:123456789.dkr.ecr.us-east-1.amazonaws.com*.azurecr.iogcr.io*.*.registry.ioregistry.io:8080/pathdefaultCacheDuration [Required]meta/v1.DurationdefaultCacheDuration is the default duration the plugin will cache credentials in-memory if a cache duration is not provided in the plugin response. This field is required.apiVersion [Required]stringRequired input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse MUST use the same encoding version as the input. Cu


Question: What is Resource Types?

Answer: C r e d e n t i a l P r o v i d e r C o n f i g


Question: What is CredentialProviderConfig?

Answer: C r e d e n t i a l P r o v i d e r C o n f i g   i s   t h e   c o n f i g u r a t i o n   c o n t a i n i n g   i n f o r m a t i o n   a b o u t   e a c h   e x e c   c r e d e n t i a l   p r o v i d e r .   K u b e l e t   r e a d s   t h i s   c o n f i g u r a t i o n   f r o m   d i s k   a n d   e n a b l e s   e a c h   p r o v i d e r   a s   s p e c i f i e d   b y   t h e   C r e d e n t i a l P r o v i d e r   t y p e .   p r o v i d e r s   i s   a   l i s t   o f   c r e d e n t i a l   p r o v i d e r   p l u g i n s   t h a t   w i l l   b e   e n a b l e d   b y   t h e   k u b e l e t .   M u l t i p l e   p r o v i d e r s   m a y   m a t c h   a g a i n s t   a   s i n g l e   i m a g e ,   i n   w h i c h   c a s e   c r e d e n t i a l s   f r o m   a l l   p r o v i d e r s   w i l l   b e   r e t u r n e d   t o   t h e   k u b e l e t .   I f   m u l t i p l e   p r o v i d e r s   a r e   c a l l e d   f o r   a   s i n g l e   i m a g e ,   t h e   r e s u l t s   a r e   c o m b i n e d .   I f   p r o v i d e r s   r e t u r n   o v e r l a p p i n g   a u t h   k e y s ,   t h e   v a l u e   f r o m   t h e   p r o v i d e r   e a r l i e r   i n   t h i s   l i s t   i s   a t t e m p t e d   f i r s t .


Question: What is CredentialProvider?

Answer: A p p e a r s   i n :   C r e d e n t i a l P r o v i d e r C o n f i g   C r e d e n t i a l P r o v i d e r   r e p r e s e n t s   a n   e x e c   p l u g i n   t o   b e   i n v o k e d   b y   t h e   k u b e l e t .   T h e   p l u g i n   i s   o n l y   i n v o k e d   w h e n   a n   i m a g e   b e i n g   p u l l e d   m a t c h e s   t h e   i m a g e s   h a n d l e d   b y   t h e   p l u g i n   ( s e e   m a t c h I m a g e s ) .   n a m e   i s   t h e   r e q u i r e d   n a m e   o f   t h e   c r e d e n t i a l   p r o v i d e r .   I t   m u s t   m a t c h   t h e   n a m e   o f   t h e   p r o v i d e r   e x e c u t a b l e   a s   s e e n   b y   t h e   k u b e l e t .   T h e   e x e c u t a b l e   m u s t   b e   i n   t h e   k u b e l e t ' s   b i n   d i r e c t o r y   ( s e t   b y   t h e   - - i m a g e - c r e d e n t i a l - p r o v i d e r - b i n - d i r   f l a g ) .   R e q u i r e d   t o   b e   u n i q u e   a c r o s s   a l l   p r o v i d e r s .   m a t c h I m a g e s   i s   a   r e q u i r e d   l i s t   o f   s t r i n g s   u s e d   t o   m a t c h   a g a i n s t   i m a g e s   i n   o r d e r   t o   d e t e r m i n e   i f   t h i s   p r o v i d e r   s h o u l d   b e   i n v o k e d .   I f   o n e   o f   t h e   s t r i n g s   m a t c h e s   t h e   r e q u e s t e d   i m a g e   f r o m   t h e   k u b e l e t ,   t h e   p l u g i n   w i l l   b e   i n v o k e d   a n d   g i v e n   a   c h a n c e   t o   p r o v i d e   c r e d e n t i a l s .   I m a g e s   a r e   e x p e c t e d   t o   c o n t a i n   t h e   r e g i s t r y   d o m a i n   a n d   U R L   p a t h .   E a c h   e n t r y   i n   m a t c h I m a g e s   i s   a   p a t t e r n   w h i c h   c a n   o p t i o n a l l y   c o n t a i n   a   p o r t   a n d   a   p a t h .   G l o b s   c a n   b e   u s e d   i n   t h e   d o m a i n ,   b u t   n o t   i n   t h e   p o r t   o r   t h e   p a t h .   G l o b s   a r


Question: What is ExecEnvVar?

Answer: A p p e a r s   i n :   C r e d e n t i a l P r o v i d e r   E x e c E n v V a r   i s   u s e d   f o r   s e t t i n g   e n v i r o n m e n t   v a r i a b l e s   w h e n   e x e c u t i n g   a n   e x e c - b a s e d   c r e d e n t i a l   p l u g i n .


Question: What is ServiceAccountTokenAttributes?

Answer: A p p e a r s   i n :   C r e d e n t i a l P r o v i d e r   S e r v i c e A c c o u n t T o k e n A t t r i b u t e s   i s   t h e   c o n f i g u r a t i o n   f o r   t h e   s e r v i c e   a c c o u n t   t o k e n   t h a t   w i l l   b e   p a s s e d   t o   t h e   p l u g i n .   s e r v i c e A c c o u n t T o k e n A u d i e n c e   i s   t h e   i n t e n d e d   a u d i e n c e   f o r   t h e   p r o j e c t e d   s e r v i c e   a c c o u n t   t o k e n .   r e q u i r e S e r v i c e A c c o u n t   i n d i c a t e s   w h e t h e r   t h e   p l u g i n   r e q u i r e s   t h e   p o d   t o   h a v e   a   s e r v i c e   a c c o u n t .   I f   s e t   t o   t r u e ,   k u b e l e t   w i l l   o n l y   i n v o k e   t h e   p l u g i n   i f   t h e   p o d   h a s   a   s e r v i c e   a c c o u n t .   I f   s e t   t o   f a l s e ,   k u b e l e t   w i l l   i n v o k e   t h e   p l u g i n   e v e n   i f   t h e   p o d   d o e s   n o t   h a v e   a   s e r v i c e   a c c o u n t   a n d   w i l l   n o t   i n c l u d e   a   t o k e n   i n   t h e   C r e d e n t i a l P r o v i d e r R e q u e s t   i n   t h a t   s c e n a r i o .   T h i s   i s   u s e f u l   f o r   p l u g i n s   t h a t   a r e   u s e d   t o   p u l l   i m a g e s   f o r   p o d s   w i t h o u t   s e r v i c e   a c c o u n t s   ( e . g . ,   s t a t i c   p o d s ) .   r e q u i r e d S e r v i c e A c c o u n t A n n o t a t i o n K e y s   i s   t h e   l i s t   o f   a n n o t a t i o n   k e y s   t h a t   t h e   p l u g i n   i s   i n t e r e s t e d   i n   a n d   t h a t   a r e   r e q u i r e d   t o   b e   p r e s e n t   i n   t h e   s e r v i c e   a c c o u n t .   T h e   k e y s   d e f i n e d   i n   t h i s   l i s t   w i l l   b e   e x t r a c t e d   f r o m   t h e   c o r r e s p o n d i n g   s e r v i c e   a c c o u n t   a n d   p a s s e d   t o   t h e   p l u g i n   a s   p a r t   o f   t h e   C r e


Question: What is Storage Capacity?

Answer: Storage CapacityStorage capacity is limited and may vary depending on the node on which a pod runs: network-attached storage might not be accessible by all nodes, or storage is local to a node to begin with.FEATURE STATE: Kubernetes v1.24 [stable]This page describes how Kubernetes keeps track of storage capacity and how the scheduler uses that information to schedule Pods onto nodes that have access to enough storage capacity for the remaining missing volumes. Without storage capacity tracking, the scheduler may choose a node that doesn't have enough capacity to provision a volume and multiple scheduling retries will be needed.Before you beginKubernetes v1.34 includes cluster-level API support for storage capacity tracking. To use this you must also be using a CSI driver that supports capacity tracking. Consult the documentation for the CSI drivers that you use to find out whether this support is available and, if so, how to use it. If you are not running Kubernetes v1.34, check the documentation for that version of Kubernetes.APIThere are two API extensions for this feature:CSIStorageCapacity objects: these get produced by a CSI driver in the namespace where the driver is installed. Each object contains capacity information for one storage class and defines which nodes have access to that storage.The CSIDriverSpec.StorageCapacity field: when set to true, the Kubernetes scheduler will consider storage capacity for volumes that use the CSI driver.SchedulingStorage capacity information is used by the Kubernetes scheduler if:a Pod uses a volume that has not been created yet,that volume uses a StorageClass which references a CSI driver and uses WaitForFirstConsumer volume binding mode, andthe CSIDriver object for the driver has StorageCapacity set to true.In that case, the scheduler only considers nodes for the Pod which have enough storage available to them. This check is very simplistic and only compares the size of the volume against the capacity listed in CSIStorageCapacity objects with a topology that includes the node.For volumes with Immediate volume binding mode, the storage driver decides where to create the volume, independently of Pods that will use the volume. The scheduler then schedules Pods onto nodes where the volume is available after the volume has been created.For CSI ephemeral volumes, scheduling always happens without considering storage capacity. This is based on the assumption that this volume type is only used by special CSI drivers which are local to a node and do not need significant resources there.ReschedulingWhen a node has been selected for a Pod with WaitForFirstConsumer volumes, that decision is still tentative. The next step is that the CSI storage driver gets asked to create the volume with a hint that the volume is supposed to be available on the selected node.Because Kubernetes might have chosen a node based on out-dated capacity information, it is possible that the volume cannot really be created. The node selecti


Question: How does Storage Capacity work?

Answer: Storage CapacityStorage capacity is limited and may vary depending on the node on which a pod runs: network-attached storage might not be accessible by all nodes, or storage is local to a node to begin with.FEATURE STATE: Kubernetes v1.24 [stable]This page describes how Kubernetes keeps track of storage capacity and how the scheduler uses that information to schedule Pods onto nodes that have access to enough storage capacity for the remaining missing volumes. Without storage capacity tracking, the scheduler may choose a node that doesn't have enough capacity to provision a volume and multiple scheduling retries will be needed.Before you beginKubernetes v1.34 includes cluster-level API support for storage capacity tracking. To use this you must also be using a CSI driver that supports capacity tracking. Consult the documentation for the CSI drivers that you use to find out whether this support is available and, if so, how to use it. If you are not running Kubernetes v1.34, check the documentation for that version of Kubernetes.APIThere are two API extensions for this feature:CSIStorageCapacity objects: these get produced by a CSI driver in the namespace where the driver is installed. Each object contains capacity information for one storage class and defines which nodes have access to that storage.The CSIDriverSpec.StorageCapacity field: when set to true, the Kubernetes scheduler will consider storage capacity for volumes that use the CSI driver.SchedulingStorage capacity information is used by the Kubernetes scheduler if:a Pod uses a volume that has not been created yet,that volume uses a StorageClass which references a CSI driver and uses WaitForFirstConsumer volume binding mode, andthe CSIDriver object for the driver has StorageCapacity set to true.In that case, the scheduler only considers nodes for the Pod which have enough storage available to them. This check is very simplistic and only compares the size of the volume against the capacity listed in CSIStorageCapacity objects with a topology that includes the node.For volumes with Immediate volume binding mode, the storage driver decides where to create the volume, independently of Pods that will use the volume. The scheduler then schedules Pods onto nodes where the volume is available after the volume has been created.For CSI ephemeral volumes, scheduling always happens without considering storage capacity. This is based on the assumption that this volume type is only used by special CSI drivers which are local to a node and do not need significant resources there.ReschedulingWhen a node has been selected for a Pod with WaitForFirstConsumer volumes, that decision is still tentative. The next step is that the CSI storage driver gets asked to create the volume with a hint that the volume is supposed to be available on the selected node.Because Kubernetes might have chosen a node based on out-dated capacity information, it is possible that the volume cannot really be created. The node selecti


Question: When would you use Storage Capacity?

Answer: Storage CapacityStorage capacity is limited and may vary depending on the node on which a pod runs: network-attached storage might not be accessible by all nodes, or storage is local to a node to begin with.FEATURE STATE: Kubernetes v1.24 [stable]This page describes how Kubernetes keeps track of storage capacity and how the scheduler uses that information to schedule Pods onto nodes that have access to enough storage capacity for the remaining missing volumes. Without storage capacity tracking, the scheduler may choose a node that doesn't have enough capacity to provision a volume and multiple scheduling retries will be needed.Before you beginKubernetes v1.34 includes cluster-level API support for storage capacity tracking. To use this you must also be using a CSI driver that supports capacity tracking. Consult the documentation for the CSI drivers that you use to find out whether this support is available and, if so, how to use it. If you are not running Kubernetes v1.34, check the documentation for that version of Kubernetes.APIThere are two API extensions for this feature:CSIStorageCapacity objects: these get produced by a CSI driver in the namespace where the driver is installed. Each object contains capacity information for one storage class and defines which nodes have access to that storage.The CSIDriverSpec.StorageCapacity field: when set to true, the Kubernetes scheduler will consider storage capacity for volumes that use the CSI driver.SchedulingStorage capacity information is used by the Kubernetes scheduler if:a Pod uses a volume that has not been created yet,that volume uses a StorageClass which references a CSI driver and uses WaitForFirstConsumer volume binding mode, andthe CSIDriver object for the driver has StorageCapacity set to true.In that case, the scheduler only considers nodes for the Pod which have enough storage available to them. This check is very simplistic and only compares the size of the volume against the capacity listed in CSIStorageCapacity objects with a topology that includes the node.For volumes with Immediate volume binding mode, the storage driver decides where to create the volume, independently of Pods that will use the volume. The scheduler then schedules Pods onto nodes where the volume is available after the volume has been created.For CSI ephemeral volumes, scheduling always happens without considering storage capacity. This is based on the assumption that this volume type is only used by special CSI drivers which are local to a node and do not need significant resources there.ReschedulingWhen a node has been selected for a Pod with WaitForFirstConsumer volumes, that decision is still tentative. The next step is that the CSI storage driver gets asked to create the volume with a hint that the volume is supposed to be available on the selected node.Because Kubernetes might have chosen a node based on out-dated capacity information, it is possible that the volume cannot really be created. The node selecti


Question: What are the benefits of Storage Capacity?

Answer: Storage CapacityStorage capacity is limited and may vary depending on the node on which a pod runs: network-attached storage might not be accessible by all nodes, or storage is local to a node to begin with.FEATURE STATE: Kubernetes v1.24 [stable]This page describes how Kubernetes keeps track of storage capacity and how the scheduler uses that information to schedule Pods onto nodes that have access to enough storage capacity for the remaining missing volumes. Without storage capacity tracking, the scheduler may choose a node that doesn't have enough capacity to provision a volume and multiple scheduling retries will be needed.Before you beginKubernetes v1.34 includes cluster-level API support for storage capacity tracking. To use this you must also be using a CSI driver that supports capacity tracking. Consult the documentation for the CSI drivers that you use to find out whether this support is available and, if so, how to use it. If you are not running Kubernetes v1.34, check the documentation for that version of Kubernetes.APIThere are two API extensions for this feature:CSIStorageCapacity objects: these get produced by a CSI driver in the namespace where the driver is installed. Each object contains capacity information for one storage class and defines which nodes have access to that storage.The CSIDriverSpec.StorageCapacity field: when set to true, the Kubernetes scheduler will consider storage capacity for volumes that use the CSI driver.SchedulingStorage capacity information is used by the Kubernetes scheduler if:a Pod uses a volume that has not been created yet,that volume uses a StorageClass which references a CSI driver and uses WaitForFirstConsumer volume binding mode, andthe CSIDriver object for the driver has StorageCapacity set to true.In that case, the scheduler only considers nodes for the Pod which have enough storage available to them. This check is very simplistic and only compares the size of the volume against the capacity listed in CSIStorageCapacity objects with a topology that includes the node.For volumes with Immediate volume binding mode, the storage driver decides where to create the volume, independently of Pods that will use the volume. The scheduler then schedules Pods onto nodes where the volume is available after the volume has been created.For CSI ephemeral volumes, scheduling always happens without considering storage capacity. This is based on the assumption that this volume type is only used by special CSI drivers which are local to a node and do not need significant resources there.ReschedulingWhen a node has been selected for a Pod with WaitForFirstConsumer volumes, that decision is still tentative. The next step is that the CSI storage driver gets asked to create the volume with a hint that the volume is supposed to be available on the selected node.Because Kubernetes might have chosen a node based on out-dated capacity information, it is possible that the volume cannot really be created. The node selecti


Question: What is Storage Capacity?

Answer: S t o r a g e   c a p a c i t y   i s   l i m i t e d   a n d   m a y   v a r y   d e p e n d i n g   o n   t h e   n o d e   o n   w h i c h   a   p o d   r u n s :   n e t w o r k - a t t a c h e d   s t o r a g e   m i g h t   n o t   b e   a c c e s s i b l e   b y   a l l   n o d e s ,   o r   s t o r a g e   i s   l o c a l   t o   a   n o d e   t o   b e g i n   w i t h .   T h i s   p a g e   d e s c r i b e s   h o w   K u b e r n e t e s   k e e p s   t r a c k   o f   s t o r a g e   c a p a c i t y   a n d   h o w   t h e   s c h e d u l e r   u s e s   t h a t   i n f o r m a t i o n   t o   s c h e d u l e   P o d s   o n t o   n o d e s   t h a t   h a v e   a c c e s s   t o   e n o u g h   s t o r a g e   c a p a c i t y   f o r   t h e   r e m a i n i n g   m i s s i n g   v o l u m e s .   W i t h o u t   s t o r a g e   c a p a c i t y   t r a c k i n g ,   t h e   s c h e d u l e r   m a y   c h o o s e   a   n o d e   t h a t   d o e s n ' t   h a v e   e n o u g h   c a p a c i t y   t o   p r o v i s i o n   a   v o l u m e   a n d   m u l t i p l e   s c h e d u l i n g   r e t r i e s   w i l l   b e   n e e d e d .


Question: What is Before you begin?

Answer: K u b e r n e t e s   v 1 . 3 4   i n c l u d e s   c l u s t e r - l e v e l   A P I   s u p p o r t   f o r   s t o r a g e   c a p a c i t y   t r a c k i n g .   T o   u s e   t h i s   y o u   m u s t   a l s o   b e   u s i n g   a   C S I   d r i v e r   t h a t   s u p p o r t s   c a p a c i t y   t r a c k i n g .   C o n s u l t   t h e   d o c u m e n t a t i o n   f o r   t h e   C S I   d r i v e r s   t h a t   y o u   u s e   t o   f i n d   o u t   w h e t h e r   t h i s   s u p p o r t   i s   a v a i l a b l e   a n d ,   i f   s o ,   h o w   t o   u s e   i t .   I f   y o u   a r e   n o t   r u n n i n g   K u b e r n e t e s   v 1 . 3 4 ,   c h e c k   t h e   d o c u m e n t a t i o n   f o r   t h a t   v e r s i o n   o f   K u b e r n e t e s .


Question: What is API?

Answer: T h e r e   a r e   t w o   A P I   e x t e n s i o n s   f o r   t h i s   f e a t u r e :   C S I S t o r a g e C a p a c i t y   o b j e c t s :   t h e s e   g e t   p r o d u c e d   b y   a   C S I   d r i v e r   i n   t h e   n a m e s p a c e   w h e r e   t h e   d r i v e r   i s   i n s t a l l e d .   E a c h   o b j e c t   c o n t a i n s   c a p a c i t y   i n f o r m a t i o n   f o r   o n e   s t o r a g e   c l a s s   a n d   d e f i n e s   w h i c h   n o d e s   h a v e   a c c e s s   t o   t h a t   s t o r a g e . T h e   C S I D r i v e r S p e c . S t o r a g e C a p a c i t y   f i e l d :   w h e n   s e t   t o   t r u e ,   t h e   K u b e r n e t e s   s c h e d u l e r   w i l l   c o n s i d e r   s t o r a g e   c a p a c i t y   f o r   v o l u m e s   t h a t   u s e   t h e   C S I   d r i v e r .


Question: What is Scheduling?

Answer: S t o r a g e   c a p a c i t y   i n f o r m a t i o n   i s   u s e d   b y   t h e   K u b e r n e t e s   s c h e d u l e r   i f :   a   P o d   u s e s   a   v o l u m e   t h a t   h a s   n o t   b e e n   c r e a t e d   y e t , t h a t   v o l u m e   u s e s   a   S t o r a g e C l a s s   w h i c h   r e f e r e n c e s   a   C S I   d r i v e r   a n d   u s e s   W a i t F o r F i r s t C o n s u m e r   v o l u m e   b i n d i n g   m o d e ,   a n d t h e   C S I D r i v e r   o b j e c t   f o r   t h e   d r i v e r   h a s   S t o r a g e C a p a c i t y   s e t   t o   t r u e .   I n   t h a t   c a s e ,   t h e   s c h e d u l e r   o n l y   c o n s i d e r s   n o d e s   f o r   t h e   P o d   w h i c h   h a v e   e n o u g h   s t o r a g e   a v a i l a b l e   t o   t h e m .   T h i s   c h e c k   i s   v e r y   s i m p l i s t i c   a n d   o n l y   c o m p a r e s   t h e   s i z e   o f   t h e   v o l u m e   a g a i n s t   t h e   c a p a c i t y   l i s t e d   i n   C S I S t o r a g e C a p a c i t y   o b j e c t s   w i t h   a   t o p o l o g y   t h a t   i n c l u d e s   t h e   n o d e .   F o r   v o l u m e s   w i t h   I m m e d i a t e   v o l u m e   b i n d i n g   m o d e ,   t h e   s t o r a g e   d r i v e r   d e c i d e s   w h e r e   t o   c r e a t e   t h e   v o l u m e ,   i n d e p e n d e n t l y   o f   P o d s   t h a t   w i l l   u s e   t h e   v o l u m e .   T h e   s c h e d u l e r   t h e n   s c h e d u l e s   P o d s   o n t o   n o d e s   w h e r e   t h e   v o l u m e   i s   a v a i l a b l e   a f t e r   t h e   v o l u m e   h a s   b e e n   c r e a t e d .   F o r   C S I   e p h e m e r a l   v o l u m e s ,   s c h e d u l i n g   a l w a y s   h a p p e n s   w i t h o u t   c o n s i d e r i n g   s t o r a g e   c a p a c i t y .   T h i s   i s   b a s e d   o n   t h e   a s s u m p t i o n   t h a t   t h i s   v o l u m e   t y p e   i s   o n l y   u s e d   b y


Question: What is Rescheduling?

Answer: W h e n   a   n o d e   h a s   b e e n   s e l e c t e d   f o r   a   P o d   w i t h   W a i t F o r F i r s t C o n s u m e r   v o l u m e s ,   t h a t   d e c i s i o n   i s   s t i l l   t e n t a t i v e .   T h e   n e x t   s t e p   i s   t h a t   t h e   C S I   s t o r a g e   d r i v e r   g e t s   a s k e d   t o   c r e a t e   t h e   v o l u m e   w i t h   a   h i n t   t h a t   t h e   v o l u m e   i s   s u p p o s e d   t o   b e   a v a i l a b l e   o n   t h e   s e l e c t e d   n o d e .   B e c a u s e   K u b e r n e t e s   m i g h t   h a v e   c h o s e n   a   n o d e   b a s e d   o n   o u t - d a t e d   c a p a c i t y   i n f o r m a t i o n ,   i t   i s   p o s s i b l e   t h a t   t h e   v o l u m e   c a n n o t   r e a l l y   b e   c r e a t e d .   T h e   n o d e   s e l e c t i o n   i s   t h e n   r e s e t   a n d   t h e   K u b e r n e t e s   s c h e d u l e r   t r i e s   a g a i n   t o   f i n d   a   n o d e   f o r   t h e   P o d .


Question: What is Limitations?

Answer: S t o r a g e   c a p a c i t y   t r a c k i n g   i n c r e a s e s   t h e   c h a n c e   t h a t   s c h e d u l i n g   w o r k s   o n   t h e   f i r s t   t r y ,   b u t   c a n n o t   g u a r a n t e e   t h i s   b e c a u s e   t h e   s c h e d u l e r   h a s   t o   d e c i d e   b a s e d   o n   p o t e n t i a l l y   o u t - d a t e d   i n f o r m a t i o n .   U s u a l l y ,   t h e   s a m e   r e t r y   m e c h a n i s m   a s   f o r   s c h e d u l i n g   w i t h o u t   a n y   s t o r a g e   c a p a c i t y   i n f o r m a t i o n   h a n d l e s   s c h e d u l i n g   f a i l u r e s .   O n e   s i t u a t i o n   w h e r e   s c h e d u l i n g   c a n   f a i l   p e r m a n e n t l y   i s   w h e n   a   P o d   u s e s   m u l t i p l e   v o l u m e s :   o n e   v o l u m e   m i g h t   h a v e   b e e n   c r e a t e d   a l r e a d y   i n   a   t o p o l o g y   s e g m e n t   w h i c h   t h e n   d o e s   n o t   h a v e   e n o u g h   c a p a c i t y   l e f t   f o r   a n o t h e r   v o l u m e .   M a n u a l   i n t e r v e n t i o n   i s   n e c e s s a r y   t o   r e c o v e r   f r o m   t h i s ,   f o r   e x a m p l e   b y   i n c r e a s i n g   c a p a c i t y   o r   d e l e t i n g   t h e   v o l u m e   t h a t   w a s   a l r e a d y   c r e a t e d .


Question: What is What's next?

Answer: F o r   m o r e   i n f o r m a t i o n   o n   t h e   d e s i g n ,   s e e   t h e   S t o r a g e   C a p a c i t y   C o n s t r a i n t s   f o r   P o d   S c h e d u l i n g   K E P .


Question: What is Using sysctls in a Kubernetes Cluster?

Answer: Using sysctls in a Kubernetes ClusterFEATURE STATE: Kubernetes v1.21 [stable]This document describes how to configure and use kernel parameters within a Kubernetes cluster using the sysctl interface.Note:Starting from Kubernetes version 1.23, the kubelet supports the use of either / or . as separators for sysctl names. Starting from Kubernetes version 1.25, setting Sysctls for a Pod supports setting sysctls with slashes. For example, you can represent the same sysctl name as kernel.shm_rmid_forced using a period as the separator, or as kernel/shm_rmid_forced using a slash as a separator. For more sysctl parameter conversion method details, please refer to the page sysctl.d(5) from the Linux man-pages project.Before you beginNote:sysctl is a Linux-specific command-line tool used to configure various kernel parameters and it is not available on non-Linux operating systems.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesFor some steps, you also need to be able to reconfigure the command line options for the kubelets running on your cluster.Listing all Sysctl ParametersIn Linux, the sysctl interface allows an administrator to modify kernel parameters at runtime. Parameters are available via the /proc/sys/ virtual process file system. The parameters cover various subsystems such as:kernel (common prefix: kernel.)networking (common prefix: net.)virtual memory (common prefix: vm.)MDADM (common prefix: dev.)More subsystems are described in Kernel docs.To get a list of all parameters, you can runsudo sysctl -a Safe and Unsafe SysctlsKubernetes classes sysctls as either safe or unsafe. In addition to proper namespacing, a safe sysctl must be properly isolated between pods on the same node. This means that setting a safe sysctl for one podmust not have any influence on any other pod on the nodemust not allow to harm the node's healthmust not allow to gain CPU or memory resources outside of the resource limits of a pod.By far, most of the namespaced sysctls are not necessarily considered safe. The following sysctls are supported in the safe set:kernel.shm_rmid_forced;net.ipv4.ip_local_port_range;net.ipv4.tcp_syncookies;net.ipv4.ping_group_range (since Kubernetes 1.18);net.ipv4.ip_unprivileged_port_start (since Kubernetes 1.22);net.ipv4.ip_local_reserved_ports (since Kubernetes 1.27, needs kernel 3.16+);net.ipv4.tcp_keepalive_time (since Kubernetes 1.29, needs kernel 4.5+);net.ipv4.tcp_fin_timeout (since Kubernetes 1.29, needs kernel 4.6+);net.ipv4.tcp_keepalive_intvl (since Kubernetes 1.29, needs kernel 4.5+);net.ipv4.tcp_keepalive_probes (since Kubernetes 1.29, needs kernel 4.5+).net.ipv4.


Question: How does Using sysctls in a Kubernetes Cluster work?

Answer: Using sysctls in a Kubernetes ClusterFEATURE STATE: Kubernetes v1.21 [stable]This document describes how to configure and use kernel parameters within a Kubernetes cluster using the sysctl interface.Note:Starting from Kubernetes version 1.23, the kubelet supports the use of either / or . as separators for sysctl names. Starting from Kubernetes version 1.25, setting Sysctls for a Pod supports setting sysctls with slashes. For example, you can represent the same sysctl name as kernel.shm_rmid_forced using a period as the separator, or as kernel/shm_rmid_forced using a slash as a separator. For more sysctl parameter conversion method details, please refer to the page sysctl.d(5) from the Linux man-pages project.Before you beginNote:sysctl is a Linux-specific command-line tool used to configure various kernel parameters and it is not available on non-Linux operating systems.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesFor some steps, you also need to be able to reconfigure the command line options for the kubelets running on your cluster.Listing all Sysctl ParametersIn Linux, the sysctl interface allows an administrator to modify kernel parameters at runtime. Parameters are available via the /proc/sys/ virtual process file system. The parameters cover various subsystems such as:kernel (common prefix: kernel.)networking (common prefix: net.)virtual memory (common prefix: vm.)MDADM (common prefix: dev.)More subsystems are described in Kernel docs.To get a list of all parameters, you can runsudo sysctl -a Safe and Unsafe SysctlsKubernetes classes sysctls as either safe or unsafe. In addition to proper namespacing, a safe sysctl must be properly isolated between pods on the same node. This means that setting a safe sysctl for one podmust not have any influence on any other pod on the nodemust not allow to harm the node's healthmust not allow to gain CPU or memory resources outside of the resource limits of a pod.By far, most of the namespaced sysctls are not necessarily considered safe. The following sysctls are supported in the safe set:kernel.shm_rmid_forced;net.ipv4.ip_local_port_range;net.ipv4.tcp_syncookies;net.ipv4.ping_group_range (since Kubernetes 1.18);net.ipv4.ip_unprivileged_port_start (since Kubernetes 1.22);net.ipv4.ip_local_reserved_ports (since Kubernetes 1.27, needs kernel 3.16+);net.ipv4.tcp_keepalive_time (since Kubernetes 1.29, needs kernel 4.5+);net.ipv4.tcp_fin_timeout (since Kubernetes 1.29, needs kernel 4.6+);net.ipv4.tcp_keepalive_intvl (since Kubernetes 1.29, needs kernel 4.5+);net.ipv4.tcp_keepalive_probes (since Kubernetes 1.29, needs kernel 4.5+).net.ipv4.


Question: When would you use Using sysctls in a Kubernetes Cluster?

Answer: Using sysctls in a Kubernetes ClusterFEATURE STATE: Kubernetes v1.21 [stable]This document describes how to configure and use kernel parameters within a Kubernetes cluster using the sysctl interface.Note:Starting from Kubernetes version 1.23, the kubelet supports the use of either / or . as separators for sysctl names. Starting from Kubernetes version 1.25, setting Sysctls for a Pod supports setting sysctls with slashes. For example, you can represent the same sysctl name as kernel.shm_rmid_forced using a period as the separator, or as kernel/shm_rmid_forced using a slash as a separator. For more sysctl parameter conversion method details, please refer to the page sysctl.d(5) from the Linux man-pages project.Before you beginNote:sysctl is a Linux-specific command-line tool used to configure various kernel parameters and it is not available on non-Linux operating systems.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesFor some steps, you also need to be able to reconfigure the command line options for the kubelets running on your cluster.Listing all Sysctl ParametersIn Linux, the sysctl interface allows an administrator to modify kernel parameters at runtime. Parameters are available via the /proc/sys/ virtual process file system. The parameters cover various subsystems such as:kernel (common prefix: kernel.)networking (common prefix: net.)virtual memory (common prefix: vm.)MDADM (common prefix: dev.)More subsystems are described in Kernel docs.To get a list of all parameters, you can runsudo sysctl -a Safe and Unsafe SysctlsKubernetes classes sysctls as either safe or unsafe. In addition to proper namespacing, a safe sysctl must be properly isolated between pods on the same node. This means that setting a safe sysctl for one podmust not have any influence on any other pod on the nodemust not allow to harm the node's healthmust not allow to gain CPU or memory resources outside of the resource limits of a pod.By far, most of the namespaced sysctls are not necessarily considered safe. The following sysctls are supported in the safe set:kernel.shm_rmid_forced;net.ipv4.ip_local_port_range;net.ipv4.tcp_syncookies;net.ipv4.ping_group_range (since Kubernetes 1.18);net.ipv4.ip_unprivileged_port_start (since Kubernetes 1.22);net.ipv4.ip_local_reserved_ports (since Kubernetes 1.27, needs kernel 3.16+);net.ipv4.tcp_keepalive_time (since Kubernetes 1.29, needs kernel 4.5+);net.ipv4.tcp_fin_timeout (since Kubernetes 1.29, needs kernel 4.6+);net.ipv4.tcp_keepalive_intvl (since Kubernetes 1.29, needs kernel 4.5+);net.ipv4.tcp_keepalive_probes (since Kubernetes 1.29, needs kernel 4.5+).net.ipv4.


Question: What are the benefits of Using sysctls in a Kubernetes Cluster?

Answer: Using sysctls in a Kubernetes ClusterFEATURE STATE: Kubernetes v1.21 [stable]This document describes how to configure and use kernel parameters within a Kubernetes cluster using the sysctl interface.Note:Starting from Kubernetes version 1.23, the kubelet supports the use of either / or . as separators for sysctl names. Starting from Kubernetes version 1.25, setting Sysctls for a Pod supports setting sysctls with slashes. For example, you can represent the same sysctl name as kernel.shm_rmid_forced using a period as the separator, or as kernel/shm_rmid_forced using a slash as a separator. For more sysctl parameter conversion method details, please refer to the page sysctl.d(5) from the Linux man-pages project.Before you beginNote:sysctl is a Linux-specific command-line tool used to configure various kernel parameters and it is not available on non-Linux operating systems.You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesFor some steps, you also need to be able to reconfigure the command line options for the kubelets running on your cluster.Listing all Sysctl ParametersIn Linux, the sysctl interface allows an administrator to modify kernel parameters at runtime. Parameters are available via the /proc/sys/ virtual process file system. The parameters cover various subsystems such as:kernel (common prefix: kernel.)networking (common prefix: net.)virtual memory (common prefix: vm.)MDADM (common prefix: dev.)More subsystems are described in Kernel docs.To get a list of all parameters, you can runsudo sysctl -a Safe and Unsafe SysctlsKubernetes classes sysctls as either safe or unsafe. In addition to proper namespacing, a safe sysctl must be properly isolated between pods on the same node. This means that setting a safe sysctl for one podmust not have any influence on any other pod on the nodemust not allow to harm the node's healthmust not allow to gain CPU or memory resources outside of the resource limits of a pod.By far, most of the namespaced sysctls are not necessarily considered safe. The following sysctls are supported in the safe set:kernel.shm_rmid_forced;net.ipv4.ip_local_port_range;net.ipv4.tcp_syncookies;net.ipv4.ping_group_range (since Kubernetes 1.18);net.ipv4.ip_unprivileged_port_start (since Kubernetes 1.22);net.ipv4.ip_local_reserved_ports (since Kubernetes 1.27, needs kernel 3.16+);net.ipv4.tcp_keepalive_time (since Kubernetes 1.29, needs kernel 4.5+);net.ipv4.tcp_fin_timeout (since Kubernetes 1.29, needs kernel 4.6+);net.ipv4.tcp_keepalive_intvl (since Kubernetes 1.29, needs kernel 4.5+);net.ipv4.tcp_keepalive_probes (since Kubernetes 1.29, needs kernel 4.5+).net.ipv4.


Question: What is Using sysctls in a Kubernetes Cluster?

Answer: T h i s   d o c u m e n t   d e s c r i b e s   h o w   t o   c o n f i g u r e   a n d   u s e   k e r n e l   p a r a m e t e r s   w i t h i n   a   K u b e r n e t e s   c l u s t e r   u s i n g   t h e   s y s c t l   i n t e r f a c e .


Question: What is Note:?

Answer: Y o u   n e e d   t o   h a v e   a   K u b e r n e t e s   c l u s t e r ,   a n d   t h e   k u b e c t l   c o m m a n d - l i n e   t o o l   m u s t   b e   c o n f i g u r e d   t o   c o m m u n i c a t e   w i t h   y o u r   c l u s t e r .   I t   i s   r e c o m m e n d e d   t o   r u n   t h i s   t u t o r i a l   o n   a   c l u s t e r   w i t h   a t   l e a s t   t w o   n o d e s   t h a t   a r e   n o t   a c t i n g   a s   c o n t r o l   p l a n e   h o s t s .   I f   y o u   d o   n o t   a l r e a d y   h a v e   a   c l u s t e r ,   y o u   c a n   c r e a t e   o n e   b y   u s i n g   m i n i k u b e   o r   y o u   c a n   u s e   o n e   o f   t h e s e   K u b e r n e t e s   p l a y g r o u n d s :   i x i m i u z   L a b s K i l l e r c o d a K o d e K l o u d P l a y   w i t h   K u b e r n e t e s   F o r   s o m e   s t e p s ,   y o u   a l s o   n e e d   t o   b e   a b l e   t o   r e c o n f i g u r e   t h e   c o m m a n d   l i n e   o p t i o n s   f o r   t h e   k u b e l e t s   r u n n i n g   o n   y o u r   c l u s t e r .


Question: What is Listing all Sysctl Parameters?

Answer: I n   L i n u x ,   t h e   s y s c t l   i n t e r f a c e   a l l o w s   a n   a d m i n i s t r a t o r   t o   m o d i f y   k e r n e l   p a r a m e t e r s   a t   r u n t i m e .   P a r a m e t e r s   a r e   a v a i l a b l e   v i a   t h e   / p r o c / s y s /   v i r t u a l   p r o c e s s   f i l e   s y s t e m .   T h e   p a r a m e t e r s   c o v e r   v a r i o u s   s u b s y s t e m s   s u c h   a s :   k e r n e l   ( c o m m o n   p r e f i x :   k e r n e l . ) n e t w o r k i n g   ( c o m m o n   p r e f i x :   n e t . ) v i r t u a l   m e m o r y   ( c o m m o n   p r e f i x :   v m . ) M D A D M   ( c o m m o n   p r e f i x :   d e v . ) M o r e   s u b s y s t e m s   a r e   d e s c r i b e d   i n   K e r n e l   d o c s .   T o   g e t   a   l i s t   o f   a l l   p a r a m e t e r s ,   y o u   c a n   r u n   s u d o   s y s c t l   - a


Question: What is Safe and Unsafe Sysctls?

Answer: K u b e r n e t e s   c l a s s e s   s y s c t l s   a s   e i t h e r   s a f e   o r   u n s a f e .   I n   a d d i t i o n   t o   p r o p e r   n a m e s p a c i n g ,   a   s a f e   s y s c t l   m u s t   b e   p r o p e r l y   i s o l a t e d   b e t w e e n   p o d s   o n   t h e   s a m e   n o d e .   T h i s   m e a n s   t h a t   s e t t i n g   a   s a f e   s y s c t l   f o r   o n e   p o d   m u s t   n o t   h a v e   a n y   i n f l u e n c e   o n   a n y   o t h e r   p o d   o n   t h e   n o d e m u s t   n o t   a l l o w   t o   h a r m   t h e   n o d e ' s   h e a l t h m u s t   n o t   a l l o w   t o   g a i n   C P U   o r   m e m o r y   r e s o u r c e s   o u t s i d e   o f   t h e   r e s o u r c e   l i m i t s   o f   a   p o d .   B y   f a r ,   m o s t   o f   t h e   n a m e s p a c e d   s y s c t l s   a r e   n o t   n e c e s s a r i l y   c o n s i d e r e d   s a f e .   T h e   f o l l o w i n g   s y s c t l s   a r e   s u p p o r t e d   i n   t h e   s a f e   s e t :   k e r n e l . s h m _ r m i d _ f o r c e d ; n e t . i p v 4 . i p _ l o c a l _ p o r t _ r a n g e ; n e t . i p v 4 . t c p _ s y n c o o k i e s ; n e t . i p v 4 . p i n g _ g r o u p _ r a n g e   ( s i n c e   K u b e r n e t e s   1 . 1 8 ) ; n e t . i p v 4 . i p _ u n p r i v i l e g e d _ p o r t _ s t a r t   ( s i n c e   K u b e r n e t e s   1 . 2 2 ) ; n e t . i p v 4 . i p _ l o c a l _ r e s e r v e d _ p o r t s   ( s i n c e   K u b e r n e t e s   1 . 2 7 ,   n e e d s   k e r n e l   3 . 1 6 + ) ; n e t . i p v 4 . t c p _ k e e p a l i v e _ t i m e   ( s i n c e   K u b e r n e t e s   1 . 2 9 ,   n e e d s   k e r n e l   4 . 5 + ) ; n e t . i p v 4 . t c p _ f i n _ t i m e o u t   ( s i n c e   K u b e r n e t e s   1 . 2 9 ,   n e e d s   k e r n e l   4 . 6 + ) ; n e t . i p v 4 . t c p _ k e e p a l i v e _ i n t v l   ( s i n c e   K u b e r n e t e s   1 . 2 9 ,   n e e d s   k e r n e l   4 . 5 + ) ; n e t .


Question: What is Note:?

Answer: T h e r e   a r e   s o m e   e x c e p t i o n s   t o   t h e   s e t   o f   s a f e   s y s c t l s :   T h e   n e t . *   s y s c t l s   a r e   n o t   a l l o w e d   w i t h   h o s t   n e t w o r k i n g   e n a b l e d . T h e   n e t . i p v 4 . t c p _ s y n c o o k i e s   s y s c t l   i s   n o t   n a m e s p a c e d   o n   L i n u x   k e r n e l   v e r s i o n   4 . 5   o r   l o w e r .   T h i s   l i s t   w i l l   b e   e x t e n d e d   i n   f u t u r e   K u b e r n e t e s   v e r s i o n s   w h e n   t h e   k u b e l e t   s u p p o r t s   b e t t e r   i s o l a t i o n   m e c h a n i s m s .


Question: What is Enabling Unsafe Sysctls?

Answer: A l l   s a f e   s y s c t l s   a r e   e n a b l e d   b y   d e f a u l t .   A l l   u n s a f e   s y s c t l s   a r e   d i s a b l e d   b y   d e f a u l t   a n d   m u s t   b e   a l l o w e d   m a n u a l l y   b y   t h e   c l u s t e r   a d m i n   o n   a   p e r - n o d e   b a s i s .   P o d s   w i t h   d i s a b l e d   u n s a f e   s y s c t l s   w i l l   b e   s c h e d u l e d ,   b u t   w i l l   f a i l   t o   l a u n c h .   W i t h   t h e   w a r n i n g   a b o v e   i n   m i n d ,   t h e   c l u s t e r   a d m i n   c a n   a l l o w   c e r t a i n   u n s a f e   s y s c t l s   f o r   v e r y   s p e c i a l   s i t u a t i o n s   s u c h   a s   h i g h - p e r f o r m a n c e   o r   r e a l - t i m e   a p p l i c a t i o n   t u n i n g .   U n s a f e   s y s c t l s   a r e   e n a b l e d   o n   a   n o d e - b y - n o d e   b a s i s   w i t h   a   f l a g   o f   t h e   k u b e l e t ;   f o r   e x a m p l e :   k u b e l e t   - - a l l o w e d - u n s a f e - s y s c t l s   \   ' k e r n e l . m s g * , n e t . c o r e . s o m a x c o n n '   . . .   F o r   M i n i k u b e ,   t h i s   c a n   b e   d o n e   v i a   t h e   e x t r a - c o n f i g   f l a g :   m i n i k u b e   s t a r t   - - e x t r a - c o n f i g = " k u b e l e t . a l l o w e d - u n s a f e - s y s c t l s = k e r n e l . m s g * , n e t . c o r e . s o m a x c o n n " . . .   O n l y   n a m e s p a c e d   s y s c t l s   c a n   b e   e n a b l e d   t h i s   w a y .


Question: What is Setting Sysctls for a Pod?

Answer: A   n u m b e r   o f   s y s c t l s   a r e   n a m e s p a c e d   i n   t o d a y ' s   L i n u x   k e r n e l s .   T h i s   m e a n s   t h a t   t h e y   c a n   b e   s e t   i n d e p e n d e n t l y   f o r   e a c h   p o d   o n   a   n o d e .   O n l y   n a m e s p a c e d   s y s c t l s   a r e   c o n f i g u r a b l e   v i a   t h e   p o d   s e c u r i t y C o n t e x t   w i t h i n   K u b e r n e t e s .   T h e   f o l l o w i n g   s y s c t l s   a r e   k n o w n   t o   b e   n a m e s p a c e d .   T h i s   l i s t   c o u l d   c h a n g e   i n   f u t u r e   v e r s i o n s   o f   t h e   L i n u x   k e r n e l .   k e r n e l . s h m * , k e r n e l . m s g * , k e r n e l . s e m , f s . m q u e u e . * , T h o s e   n e t . *   t h a t   c a n   b e   s e t   i n   c o n t a i n e r   n e t w o r k i n g   n a m e s p a c e .   H o w e v e r ,   t h e r e   a r e   e x c e p t i o n s   ( e . g . ,   n e t . n e t f i l t e r . n f _ c o n n t r a c k _ m a x   a n d   n e t . n e t f i l t e r . n f _ c o n n t r a c k _ e x p e c t _ m a x   c a n   b e   s e t   i n   c o n t a i n e r   n e t w o r k i n g   n a m e s p a c e   b u t   a r e   u n n a m e s p a c e d   b e f o r e   L i n u x   5 . 1 2 . 2 ) .   S y s c t l s   w i t h   n o   n a m e s p a c e   a r e   c a l l e d   n o d e - l e v e l   s y s c t l s .   I f   y o u   n e e d   t o   s e t   t h e m ,   y o u   m u s t   m a n u a l l y   c o n f i g u r e   t h e m   o n   e a c h   n o d e ' s   o p e r a t i n g   s y s t e m ,   o r   b y   u s i n g   a   D a e m o n S e t   w i t h   p r i v i l e g e d   c o n t a i n e r s .   U s e   t h e   p o d   s e c u r i t y C o n t e x t   t o   c o n f i g u r e   n a m e s p a c e d   s y s c t l s .   T h e   s e c u r i t y C o n t e x t   a p p l i e s   t o   a l l   c o n t a i n e r s   i n   t h e   s a m e   p o d .   T h i s   e x a m p l e   u s e s   t h e   p o d   s e c u r i t y C o


Question: What is Warning:?

Answer: a p i V e r s i o n :   v 1   k i n d :   P o d   m e t a d a t a :   n a m e :   s y s c t l - e x a m p l e   s p e c :   s e c u r i t y C o n t e x t :   s y s c t l s :   -   n a m e :   k e r n e l . s h m _ r m i d _ f o r c e d   v a l u e :   " 0 "   -   n a m e :   n e t . c o r e . s o m a x c o n n   v a l u e :   " 1 0 2 4 "   -   n a m e :   k e r n e l . m s g m a x   v a l u e :   " 6 5 5 3 6 "   . . .


Question: What is Warning:?

Answer: I t   i s   g o o d   p r a c t i c e   t o   c o n s i d e r   n o d e s   w i t h   s p e c i a l   s y s c t l   s e t t i n g s   a s   t a i n t e d   w i t h i n   a   c l u s t e r ,   a n d   o n l y   s c h e d u l e   p o d s   o n t o   t h e m   w h i c h   n e e d   t h o s e   s y s c t l   s e t t i n g s .   I t   i s   s u g g e s t e d   t o   u s e   t h e   K u b e r n e t e s   t a i n t s   a n d   t o l e r a t i o n   f e a t u r e   t o   i m p l e m e n t   t h i s .   A   p o d   w i t h   t h e   u n s a f e   s y s c t l s   w i l l   f a i l   t o   l a u n c h   o n   a n y   n o d e   w h i c h   h a s   n o t   e n a b l e d   t h o s e   t w o   u n s a f e   s y s c t l s   e x p l i c i t l y .   A s   w i t h   n o d e - l e v e l   s y s c t l s   i t   i s   r e c o m m e n d e d   t o   u s e   t a i n t s   a n d   t o l e r a t i o n   f e a t u r e   o r   t a i n t s   o n   n o d e s   t o   s c h e d u l e   t h o s e   p o d s   o n t o   t h e   r i g h t   n o d e s .


Question: What is Determine the Reason for Pod Failure?

Answer: Determine the Reason for Pod FailureThis page shows how to write and read a Container termination message.Termination messages provide a way for containers to write information about fatal events to a location where it can be easily retrieved and surfaced by tools like dashboards and monitoring software. In most cases, information that you put in a termination message should also be written to the general Kubernetes logs.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesWriting and reading a termination messageIn this exercise, you create a Pod that runs one container. The manifest for that Pod specifies a command that runs when the container starts:debug/termination.yaml apiVersion: v1 kind: Pod metadata: name: termination-demo spec: containers: - name: termination-demo-container image: debian command: ["/bin/sh"] args: ["-c", "sleep 10 && echo Sleep expired > /dev/termination-log"] Create a Pod based on the YAML configuration file:kubectl apply -f https://k8s.io/examples/debug/termination.yaml In the YAML file, in the command and args fields, you can see that the container sleeps for 10 seconds and then writes "Sleep expired" to the /dev/termination-log file. After the container writes the "Sleep expired" message, it terminates.Display information about the Pod:kubectl get pod termination-demo Repeat the preceding command until the Pod is no longer running.Display detailed information about the Pod:kubectl get pod termination-demo --output=yaml The output includes the "Sleep expired" message:apiVersion: v1 kind: Pod ... lastState: terminated: containerID: ... exitCode: 0 finishedAt: ... message: | Sleep expired ... Use a Go template to filter the output so that it includes only the termination message:kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" If you are running a multi-container Pod, you can use a Go template to include the container's name. By doing so, you can discover which of the containers is failing:kubectl get pod multi-container-pod -o go-template='{{range .status.containerStatuses}}{{printf "%s:\n%s\n\n" .name .lastState.terminated.message}}{{end}}' Customizing the termination messageKubernetes retrieves termination messages from the termination message file specified in the terminationMessagePath field of a Container, which has a default value of /dev/termination-log. By customizing this field, you can tell Kubernetes to use a different file. Kubernetes use the contents from the specified file to populate the Container's status message on both success and failure.The ter


Question: How does Determine the Reason for Pod Failure work?

Answer: Determine the Reason for Pod FailureThis page shows how to write and read a Container termination message.Termination messages provide a way for containers to write information about fatal events to a location where it can be easily retrieved and surfaced by tools like dashboards and monitoring software. In most cases, information that you put in a termination message should also be written to the general Kubernetes logs.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesWriting and reading a termination messageIn this exercise, you create a Pod that runs one container. The manifest for that Pod specifies a command that runs when the container starts:debug/termination.yaml apiVersion: v1 kind: Pod metadata: name: termination-demo spec: containers: - name: termination-demo-container image: debian command: ["/bin/sh"] args: ["-c", "sleep 10 && echo Sleep expired > /dev/termination-log"] Create a Pod based on the YAML configuration file:kubectl apply -f https://k8s.io/examples/debug/termination.yaml In the YAML file, in the command and args fields, you can see that the container sleeps for 10 seconds and then writes "Sleep expired" to the /dev/termination-log file. After the container writes the "Sleep expired" message, it terminates.Display information about the Pod:kubectl get pod termination-demo Repeat the preceding command until the Pod is no longer running.Display detailed information about the Pod:kubectl get pod termination-demo --output=yaml The output includes the "Sleep expired" message:apiVersion: v1 kind: Pod ... lastState: terminated: containerID: ... exitCode: 0 finishedAt: ... message: | Sleep expired ... Use a Go template to filter the output so that it includes only the termination message:kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" If you are running a multi-container Pod, you can use a Go template to include the container's name. By doing so, you can discover which of the containers is failing:kubectl get pod multi-container-pod -o go-template='{{range .status.containerStatuses}}{{printf "%s:\n%s\n\n" .name .lastState.terminated.message}}{{end}}' Customizing the termination messageKubernetes retrieves termination messages from the termination message file specified in the terminationMessagePath field of a Container, which has a default value of /dev/termination-log. By customizing this field, you can tell Kubernetes to use a different file. Kubernetes use the contents from the specified file to populate the Container's status message on both success and failure.The ter


Question: When would you use Determine the Reason for Pod Failure?

Answer: Determine the Reason for Pod FailureThis page shows how to write and read a Container termination message.Termination messages provide a way for containers to write information about fatal events to a location where it can be easily retrieved and surfaced by tools like dashboards and monitoring software. In most cases, information that you put in a termination message should also be written to the general Kubernetes logs.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesWriting and reading a termination messageIn this exercise, you create a Pod that runs one container. The manifest for that Pod specifies a command that runs when the container starts:debug/termination.yaml apiVersion: v1 kind: Pod metadata: name: termination-demo spec: containers: - name: termination-demo-container image: debian command: ["/bin/sh"] args: ["-c", "sleep 10 && echo Sleep expired > /dev/termination-log"] Create a Pod based on the YAML configuration file:kubectl apply -f https://k8s.io/examples/debug/termination.yaml In the YAML file, in the command and args fields, you can see that the container sleeps for 10 seconds and then writes "Sleep expired" to the /dev/termination-log file. After the container writes the "Sleep expired" message, it terminates.Display information about the Pod:kubectl get pod termination-demo Repeat the preceding command until the Pod is no longer running.Display detailed information about the Pod:kubectl get pod termination-demo --output=yaml The output includes the "Sleep expired" message:apiVersion: v1 kind: Pod ... lastState: terminated: containerID: ... exitCode: 0 finishedAt: ... message: | Sleep expired ... Use a Go template to filter the output so that it includes only the termination message:kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" If you are running a multi-container Pod, you can use a Go template to include the container's name. By doing so, you can discover which of the containers is failing:kubectl get pod multi-container-pod -o go-template='{{range .status.containerStatuses}}{{printf "%s:\n%s\n\n" .name .lastState.terminated.message}}{{end}}' Customizing the termination messageKubernetes retrieves termination messages from the termination message file specified in the terminationMessagePath field of a Container, which has a default value of /dev/termination-log. By customizing this field, you can tell Kubernetes to use a different file. Kubernetes use the contents from the specified file to populate the Container's status message on both success and failure.The ter


Question: What are the benefits of Determine the Reason for Pod Failure?

Answer: Determine the Reason for Pod FailureThis page shows how to write and read a Container termination message.Termination messages provide a way for containers to write information about fatal events to a location where it can be easily retrieved and surfaced by tools like dashboards and monitoring software. In most cases, information that you put in a termination message should also be written to the general Kubernetes logs.Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:iximiuz LabsKillercodaKodeKloudPlay with KubernetesWriting and reading a termination messageIn this exercise, you create a Pod that runs one container. The manifest for that Pod specifies a command that runs when the container starts:debug/termination.yaml apiVersion: v1 kind: Pod metadata: name: termination-demo spec: containers: - name: termination-demo-container image: debian command: ["/bin/sh"] args: ["-c", "sleep 10 && echo Sleep expired > /dev/termination-log"] Create a Pod based on the YAML configuration file:kubectl apply -f https://k8s.io/examples/debug/termination.yaml In the YAML file, in the command and args fields, you can see that the container sleeps for 10 seconds and then writes "Sleep expired" to the /dev/termination-log file. After the container writes the "Sleep expired" message, it terminates.Display information about the Pod:kubectl get pod termination-demo Repeat the preceding command until the Pod is no longer running.Display detailed information about the Pod:kubectl get pod termination-demo --output=yaml The output includes the "Sleep expired" message:apiVersion: v1 kind: Pod ... lastState: terminated: containerID: ... exitCode: 0 finishedAt: ... message: | Sleep expired ... Use a Go template to filter the output so that it includes only the termination message:kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}" If you are running a multi-container Pod, you can use a Go template to include the container's name. By doing so, you can discover which of the containers is failing:kubectl get pod multi-container-pod -o go-template='{{range .status.containerStatuses}}{{printf "%s:\n%s\n\n" .name .lastState.terminated.message}}{{end}}' Customizing the termination messageKubernetes retrieves termination messages from the termination message file specified in the terminationMessagePath field of a Container, which has a default value of /dev/termination-log. By customizing this field, you can tell Kubernetes to use a different file. Kubernetes use the contents from the specified file to populate the Container's status message on both success and failure.The ter


Question: What is Determine the Reason for Pod Failure?

Answer: T h i s   p a g e   s h o w s   h o w   t o   w r i t e   a n d   r e a d   a   C o n t a i n e r   t e r m i n a t i o n   m e s s a g e .   T e r m i n a t i o n   m e s s a g e s   p r o v i d e   a   w a y   f o r   c o n t a i n e r s   t o   w r i t e   i n f o r m a t i o n   a b o u t   f a t a l   e v e n t s   t o   a   l o c a t i o n   w h e r e   i t   c a n   b e   e a s i l y   r e t r i e v e d   a n d   s u r f a c e d   b y   t o o l s   l i k e   d a s h b o a r d s   a n d   m o n i t o r i n g   s o f t w a r e .   I n   m o s t   c a s e s ,   i n f o r m a t i o n   t h a t   y o u   p u t   i n   a   t e r m i n a t i o n   m e s s a g e   s h o u l d   a l s o   b e   w r i t t e n   t o   t h e   g e n e r a l   K u b e r n e t e s   l o g s .