TLS and Certificate Management

Kubernetes PKI Architecture

Kubernetes relies heavily on TLS certificates for securing communication between all cluster components. A typical kubeadm-provisioned cluster has a two-tier Certificate Authority (CA) structure: one for the Kubernetes components and one for etcd.

CKS Exam Relevance

You need to know where certificates are stored, how to check their expiry, how to rotate them, and how to configure TLS for etcd communication. You should also understand the Certificate Signing Request (CSR) process.

Certificate Locations on the Control Plane

All certificates for a kubeadm cluster are stored under /etc/kubernetes/pki/:

Certificate	File Path	Purpose
Kubernetes CA	`/etc/kubernetes/pki/ca.crt`, `ca.key`	Root CA for the cluster
API Server	`/etc/kubernetes/pki/apiserver.crt`, `apiserver.key`	API server serving certificate
API Server Kubelet Client	`/etc/kubernetes/pki/apiserver-kubelet-client.crt`, `.key`	API server authentication to kubelets
Front Proxy CA	`/etc/kubernetes/pki/front-proxy-ca.crt`, `.key`	CA for front proxy (aggregation layer)
Front Proxy Client	`/etc/kubernetes/pki/front-proxy-client.crt`, `.key`	Client cert for front proxy
Service Account	`/etc/kubernetes/pki/sa.key`, `sa.pub`	Signing and verifying SA tokens
etcd CA	`/etc/kubernetes/pki/etcd/ca.crt`, `ca.key`	Root CA for etcd
etcd Server	`/etc/kubernetes/pki/etcd/server.crt`, `server.key`	etcd serving certificate
etcd Peer	`/etc/kubernetes/pki/etcd/peer.crt`, `peer.key`	etcd peer-to-peer communication
etcd Healthcheck	`/etc/kubernetes/pki/etcd/healthcheck-client.crt`, `.key`	etcd health check client
API Server etcd Client	`/etc/kubernetes/pki/apiserver-etcd-client.crt`, `.key`	API server to etcd communication

Kubeconfig Files

Kubeconfig files embed client certificates for component authentication:

Kubeconfig	Path	Used By
`admin.conf`	`/etc/kubernetes/admin.conf`	Cluster administrator
`kubelet.conf`	`/etc/kubernetes/kubelet.conf`	Kubelet
`controller-manager.conf`	`/etc/kubernetes/controller-manager.conf`	Controller Manager
`scheduler.conf`	`/etc/kubernetes/scheduler.conf`	Scheduler

Checking Certificate Expiry

Using kubeadm

bash

# Check expiration of all certificates
kubeadm certs check-expiration

Example output:

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jan 15, 2027 10:30 UTC   364d            ca                      no
apiserver                  Jan 15, 2027 10:30 UTC   364d            ca                      no
apiserver-etcd-client      Jan 15, 2027 10:30 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Jan 15, 2027 10:30 UTC   364d            ca                      no
controller-manager.conf    Jan 15, 2027 10:30 UTC   364d            ca                      no
etcd-healthcheck-client    Jan 15, 2027 10:30 UTC   364d            etcd-ca                 no
etcd-peer                  Jan 15, 2027 10:30 UTC   364d            etcd-ca                 no
etcd-server                Jan 15, 2027 10:30 UTC   364d            etcd-ca                 no
front-proxy-client         Jan 15, 2027 10:30 UTC   364d            front-proxy-ca          no
scheduler.conf             Jan 15, 2027 10:30 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jan 13, 2036 10:30 UTC   10y             no
etcd-ca                 Jan 13, 2036 10:30 UTC   10y             no
front-proxy-ca          Jan 13, 2036 10:30 UTC   10y             no

Using openssl

bash

# Check a specific certificate
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text

# Check expiry date only
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate

# Check Subject Alternative Names (SANs)
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A1 "Subject Alternative Name"

# Check issuer
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -issuer

# Check full certificate details
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -dates

Quick Certificate Inspection

bash

# One-liner to check expiry of all certs in pki directory
for cert in /etc/kubernetes/pki/*.crt; do
  echo "=== $cert ==="
  openssl x509 -in "$cert" -noout -subject -enddate
done

Rotating Certificates with kubeadm

Renew All Certificates

bash

# Renew all certificates at once
sudo kubeadm certs renew all

# Renew specific certificates
sudo kubeadm certs renew apiserver
sudo kubeadm certs renew apiserver-kubelet-client
sudo kubeadm certs renew apiserver-etcd-client
sudo kubeadm certs renew front-proxy-client
sudo kubeadm certs renew etcd-server
sudo kubeadm certs renew etcd-peer
sudo kubeadm certs renew etcd-healthcheck-client

After Renewal

After renewing certificates, you must restart the affected components:

bash

# For static pod components (API server, controller manager, scheduler, etcd),
# the kubelet will automatically restart them when manifests change.
# However, you may need to force a restart:

# Option 1: Move manifests out and back in
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# Wait a few seconds for the pod to stop
sleep 10
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

# Option 2: Restart kubelet to pick up new certs
sudo systemctl restart kubelet

Renew Before Expiry

kubeadm certificates are valid for 1 year by default. CA certificates are valid for 10 years. Kubernetes automatically renews certificates during kubeadm upgrade. If you do not upgrade regularly, you must renew manually.

API Server Certificate SANs

The API server certificate must include all names and IPs through which the API server can be reached. These are called Subject Alternative Names (SANs).

Checking Current SANs

bash

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A1 "Subject Alternative Name"

Typical output:

X509v3 Subject Alternative Name:
    DNS:controlplane, DNS:kubernetes, DNS:kubernetes.default,
    DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local,
    IP Address:10.96.0.1, IP Address:192.168.1.100

Adding Custom SANs

If you need to add additional SANs (for example, a load balancer hostname), update the kubeadm configuration:

yaml

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  certSANs:
    - "kubernetes"
    - "kubernetes.default"
    - "kubernetes.default.svc"
    - "kubernetes.default.svc.cluster.local"
    - "10.96.0.1"
    - "192.168.1.100"
    - "my-custom-dns.example.com"    # Custom SAN
    - "10.0.0.50"                     # Custom IP SAN

Then regenerate the API server certificate:

bash

# Remove old certificate
sudo rm /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.key

# Regenerate with new SANs
sudo kubeadm init phase certs apiserver --config kubeadm-config.yaml

# Restart API server
sudo crictl pods --name kube-apiserver -q | xargs sudo crictl rmp

TLS Handshake in Kubernetes Context

Securing etcd Communication

etcd must use TLS for both client and peer communication.

Client-to-Server TLS

The API server connects to etcd as a client. This connection must be secured:

yaml

# In the API server manifest (/etc/kubernetes/manifests/kube-apiserver.yaml)
spec:
  containers:
    - command:
        - kube-apiserver
        - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
        - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
        - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
        - --etcd-servers=https://127.0.0.1:2379

Peer-to-Peer TLS

In multi-node etcd clusters, peer communication must also be encrypted:

yaml

# In the etcd manifest (/etc/kubernetes/manifests/etcd.yaml)
spec:
  containers:
    - command:
        - etcd
        - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
        - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
        - --peer-client-cert-auth=true
        - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

Verifying etcd TLS

bash

# Test etcd connectivity with TLS
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

# Without TLS certs, this should fail
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  endpoint health
# Expected: connection refused or certificate error

Kubernetes Certificate Signing Requests (CSR)

Kubernetes has a built-in CSR API for managing certificate requests.

Creating a CSR for a New User

bash

# Step 1: Generate a private key
openssl genrsa -out jane.key 2048

# Step 2: Create a Certificate Signing Request
openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=developers"

# Step 3: Encode the CSR
CSR_CONTENT=$(cat jane.csr | base64 | tr -d '\n')

# Step 4: Create a Kubernetes CSR object
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane
spec:
  request: ${CSR_CONTENT}
  signerName: kubernetes.io/kube-apiserver-client
  usages:
    - client auth
EOF

# Step 5: Approve the CSR
kubectl certificate approve jane

# Step 6: Retrieve the signed certificate
kubectl get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt

# Step 7: Verify the certificate
openssl x509 -in jane.crt -noout -text

Viewing and Managing CSRs

bash

# List all CSRs
kubectl get csr

# Describe a CSR
kubectl describe csr jane

# Approve a CSR
kubectl certificate approve jane

# Deny a CSR
kubectl certificate deny jane

# Delete a CSR
kubectl delete csr jane

TLS Configuration Best Practices

Security Checklist

All etcd communication uses TLS (client and peer)
API server uses valid TLS certificates with correct SANs
Kubelet uses certificate rotation (--rotate-certificates)
No self-signed certificates in production (use a proper CA)
Certificate expiration is monitored
mTLS (mutual TLS) is enforced where possible
Minimum TLS version is set to 1.2 or higher

Setting Minimum TLS Version

yaml

# In the API server manifest
spec:
  containers:
    - command:
        - kube-apiserver
        - --tls-min-version=VersionTLS12
        - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Enabling Kubelet Certificate Rotation

yaml

# In kubelet config (/var/lib/kubelet/config.yaml)
rotateCertificates: true
serverTLSBootstrap: true

bash

# Verify rotation is enabled
ps aux | grep kubelet | grep rotate

Certificate Expiry = Cluster Down

If certificates expire, cluster components cannot communicate. The API server will not start, kubectl will not work, and workloads cannot be scheduled. Always monitor certificate expiry and renew well before the deadline. Set calendar reminders or use monitoring tools like cert-manager.

Common Certificate Troubleshooting

bash

# Check if API server certificate is valid for a specific hostname
openssl s_client -connect 192.168.1.100:6443 -showcerts </dev/null 2>/dev/null | \
  openssl x509 -noout -text | grep -A1 "Subject Alternative Name"

# Verify a certificate chain
openssl verify -CAfile /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/apiserver.crt

# Check certificate and key match
diff <(openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -modulus) \
     <(openssl rsa -in /etc/kubernetes/pki/apiserver.key -noout -modulus)
# Should show no differences

# Debug TLS connection issues
openssl s_client -connect 127.0.0.1:6443 \
  -CAfile /etc/kubernetes/pki/ca.crt \
  -cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
  -key /etc/kubernetes/pki/apiserver-kubelet-client.key

TLS and Certificate Management ​

Kubernetes PKI Architecture ​

Certificate Locations on the Control Plane ​

Kubeconfig Files ​

Checking Certificate Expiry ​

Using kubeadm ​

Using openssl ​

Rotating Certificates with kubeadm ​

Renew All Certificates ​

After Renewal ​

API Server Certificate SANs ​

Checking Current SANs ​

Adding Custom SANs ​

TLS Handshake in Kubernetes Context ​

Securing etcd Communication ​

Client-to-Server TLS ​

Peer-to-Peer TLS ​

Verifying etcd TLS ​

Kubernetes Certificate Signing Requests (CSR) ​

Creating a CSR for a New User ​

Viewing and Managing CSRs ​

TLS Configuration Best Practices ​

Setting Minimum TLS Version ​

Enabling Kubelet Certificate Rotation ​

Common Certificate Troubleshooting ​

TLS and Certificate Management

Kubernetes PKI Architecture

Certificate Locations on the Control Plane

Kubeconfig Files

Checking Certificate Expiry

Using kubeadm

Using openssl

Rotating Certificates with kubeadm

Renew All Certificates

After Renewal

API Server Certificate SANs

Checking Current SANs

Adding Custom SANs

TLS Handshake in Kubernetes Context

Securing etcd Communication

Client-to-Server TLS

Peer-to-Peer TLS

Verifying etcd TLS

Kubernetes Certificate Signing Requests (CSR)

Creating a CSR for a New User

Viewing and Managing CSRs

TLS Configuration Best Practices

Setting Minimum TLS Version

Enabling Kubelet Certificate Rotation

Common Certificate Troubleshooting