Skip to content

How to Approach Control Plane Issues

Prerequisite Knowledge

  • /etc/kubernetes/manifests Location of control plane static pod manifests.

Places to Look for Logs

Kubelet Logs

Use these when the kubelet itself is failing due to syntax errors, invalid flags, or startup issues.

bash
journalctl -u kubelet
journalctl -u kubelet -n 100
journalctl -u kubelet -f
journalctl -u kubelet -b
journalctl -u kubelet -p err
journalctl -u kubelet --since "10 minutes ago"

To inspect kubelet service flags and configuration location:

bash
systemctl cat kubelet

System logs (if applicable):

bash
/var/log/syslog | grep <keyword>

Static Pod Logs

Use these when the kubelet is running and able to create static pods, but the pods are crashing.

  • Pod logs:

    • /var/log/pods/

      • 0.log → current log
      • 1.log → rotated log
  • Container logs:

    • /var/log/containers/
    • This directory contains symlinks to pod logs.

Container Runtime

Use these when the kubelet created the static pod but the container is failing.

bash
crictl ps
crictl ps -a
crictl logs <container-id>

Health and Probe Endpoints

kube-apiserver

Probe TypeEndpointPort
Startup/livez6443
Liveness/livez6443
Readiness/readyz6443

etcd

Probe TypeEndpointPort
Liveness/health2379
Readiness/health2379

kube-controller-manager

Probe TypeEndpointPort
Liveness/healthz10257
Readiness/healthz10257

kube-scheduler

Probe TypeEndpointPort
Liveness/healthz10259
Readiness/healthz10259

kubelet

EndpointAddress
/healthz127.0.0.1:10248

Key Component Ports

ComponentPurposePort
kube-apiserverAPI + health6443
etcdClient API2379
kubeletLocal health10248

kube-apiserver Minimal Required Flags

  • --advertise-address=<node-ip>
  • --secure-port=6443
  • --etcd-servers=https://127.0.0.1:2379

TLS Certificate Paths (Do Not Invent)

API Server

/etc/kubernetes/pki/apiserver.crt
/etc/kubernetes/pki/apiserver.key
/etc/kubernetes/pki/ca.crt

etcd

/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/server.crt
/etc/kubernetes/pki/etcd/server.key

If these files exist, do not modify them.


Released under the MIT License.