Managing Kubernetes Cluster via kubeadm

kubeadm Overview

kubeadm is the official tool for bootstrapping Kubernetes clusters.

Command	Purpose
`kubeadm init`	Initialize control plane
`kubeadm join`	Join node to cluster
`kubeadm upgrade`	Upgrade cluster version
`kubeadm token`	Manage bootstrap tokens
`kubeadm reset`	Tear down cluster on node
`kubeadm certs`	Manage cluster certificates

Cluster Creation

Pre-requisites

bash

# Disable swap (required)
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab

# Load kernel modules
cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

# Sysctl settings
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

Initialize Control Plane

bash

# Basic init
kubeadm init

# With specific version
kubeadm init --kubernetes-version=1.31.0

# With pod network CIDR (required for CNI)
kubeadm init --pod-network-cidr=192.168.0.0/16

# Full example
kubeadm init \
  --kubernetes-version=1.31.0 \
  --pod-network-cidr=10.244.0.0/16 \
  --service-cidr=10.96.0.0/12 \
  --apiserver-advertise-address=192.168.1.10 \
  --control-plane-endpoint=k8s-api.example.com:6443

Common Init Flags

Flag	Purpose
`--kubernetes-version`	Specific K8s version
`--pod-network-cidr`	Pod network CIDR (for CNI)
`--service-cidr`	Service network CIDR
`--apiserver-advertise-address`	API server bind address
`--control-plane-endpoint`	Stable endpoint for HA
`--upload-certs`	Upload certs to kubeadm-certs secret
`--ignore-preflight-errors`	Skip specific checks

Configure kubectl After Init

bash

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# Or for root user
export KUBECONFIG=/etc/kubernetes/admin.conf

Join Nodes to Cluster

Generate Join Command

bash

# Print join command for workers
kubeadm token create --print-join-command

Join Worker Node

bash

kubeadm join <control-plane-ip>:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Join Additional Control Plane

bash

# On first control plane, generate certificate key
kubeadm init phase upload-certs --upload-certs

# Join as control plane
kubeadm join <control-plane-ip>:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <certificate-key>

Token Management

bash

# List tokens
kubeadm token list

# Create new token (24h default)
kubeadm token create

# Create token with custom TTL
kubeadm token create --ttl 2h

# Delete token
kubeadm token delete <token>

# Get CA cert hash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | \
  openssl rsa -pubin -outform der 2>/dev/null | \
  openssl dgst -sha256 -hex | sed 's/^.* //'

Cluster Upgrade

Upgrade Order (CRITICAL)

kubeadm first
Control plane nodes (one at a time)
Worker nodes (one at a time)
Always upgrade one minor version at a time (1.30 → 1.31, not 1.30 → 1.32)

Upgrade Control Plane

bash

# 1. Check available versions
apt-cache madison kubeadm

# 2. Upgrade kubeadm
apt-mark unhold kubeadm
apt-get update && apt-get install -y kubeadm=1.31.0-1.1
apt-mark hold kubeadm

# 3. Verify kubeadm version
kubeadm version

# 4. Check upgrade plan
kubeadm upgrade plan

# 5. Apply upgrade (first control plane)
kubeadm upgrade apply v1.31.0

# 6. Drain the node
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# 7. Upgrade kubelet and kubectl
apt-mark unhold kubelet kubectl
apt-get update && apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
apt-mark hold kubelet kubectl

# 8. Restart kubelet
systemctl daemon-reload
systemctl restart kubelet

# 9. Uncordon node
kubectl uncordon <node-name>

Upgrade Additional Control Planes

bash

# Use upgrade node instead of upgrade apply
kubeadm upgrade node

# Then upgrade kubelet/kubectl same as above

Upgrade Worker Nodes

bash

# 1. From control plane, drain the worker
kubectl drain <worker-name> --ignore-daemonsets --delete-emptydir-data

# 2. On worker: upgrade kubeadm
apt-mark unhold kubeadm
apt-get update && apt-get install -y kubeadm=1.31.0-1.1
apt-mark hold kubeadm

# 3. Upgrade node configuration
kubeadm upgrade node

# 4. Upgrade kubelet and kubectl
apt-mark unhold kubelet kubectl
apt-get update && apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
apt-mark hold kubelet kubectl

# 5. Restart kubelet
systemctl daemon-reload
systemctl restart kubelet

# 6. Uncordon from control plane
kubectl uncordon <worker-name>

etcd Backup and Restore

Backup etcd

bash

# Find etcd cert paths
cat /etc/kubernetes/manifests/etcd.yaml | grep -E "cert|key|ca"

# Backup
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify backup
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table

Restore etcd

bash

# Stop kubelet
systemctl stop kubelet

# Move current etcd data
mv /var/lib/etcd /var/lib/etcd.bak

# Restore from snapshot
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd \
  --name=<node-name> \
  --initial-cluster=<node-name>=https://<node-ip>:2380 \
  --initial-cluster-token=etcd-cluster-1 \
  --initial-advertise-peer-urls=https://<node-ip>:2380

# Fix ownership
chown -R etcd:etcd /var/lib/etcd

# Start kubelet
systemctl start kubelet

etcd Key-Value Operations

bash

# Get all keys
ETCDCTL_API=3 etcdctl get / --prefix --keys-only \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Get specific key
ETCDCTL_API=3 etcdctl get /registry/pods/default/nginx ...

# Check etcd health
ETCDCTL_API=3 etcdctl endpoint health ...
ETCDCTL_API=3 etcdctl endpoint status ...

Certificate Management

Check Certificate Expiration

bash

kubeadm certs check-expiration

Renew All Certificates

bash

kubeadm certs renew all

Renew Specific Certificate

bash

kubeadm certs renew apiserver
kubeadm certs renew apiserver-kubelet-client
kubeadm certs renew front-proxy-client

Certificate Locations

Certificate	Location
CA	`/etc/kubernetes/pki/ca.crt`
API Server	`/etc/kubernetes/pki/apiserver.crt`
etcd CA	`/etc/kubernetes/pki/etcd/ca.crt`
etcd Server	`/etc/kubernetes/pki/etcd/server.crt`
Front Proxy CA	`/etc/kubernetes/pki/front-proxy-ca.crt`

kubeadm reset

bash

# Tear down kubeadm setup on a node
kubeadm reset

# Clean up iptables
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

# Clean up CNI
rm -rf /etc/cni/net.d

# Clean up .kube
rm -rf $HOME/.kube/config

High Availability Setup

HA Topologies

Stacked etcd (default):

etcd runs on control plane nodes
Simpler setup
etcd and control plane share fate

External etcd:

etcd runs on dedicated nodes
More resilient
Requires more nodes

HA Requirements

Minimum 3 control plane nodes
Load balancer for API server
Shared --control-plane-endpoint

HA Init Example

bash

# First control plane
kubeadm init \
  --control-plane-endpoint "k8s-lb.example.com:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16

# Additional control planes use kubeadm join --control-plane

Troubleshooting

Common Issues

Issue	Check
kubelet not starting	`journalctl -xeu kubelet`
API server down	`/var/log/containers/kube-apiserver-*`
Node NotReady	`kubectl describe node <name>`
etcd issues	`crictl logs <etcd-container>`
Join fails	Token expired? CA hash correct?

Useful Commands

bash

# Check kubelet status
systemctl status kubelet
journalctl -xeu kubelet

# Check static pod manifests
ls -la /etc/kubernetes/manifests/

# Check container runtime
crictl ps
crictl logs <container-id>

# Verify cluster health
kubectl get componentstatuses  # deprecated but useful
kubectl get nodes
kubectl cluster-info

Quick Reference

bash

# Create cluster
kubeadm init --pod-network-cidr=10.244.0.0/16

# Get join command
kubeadm token create --print-join-command

# Upgrade cluster
kubeadm upgrade plan
kubeadm upgrade apply v1.31.0

# Backup etcd
ETCDCTL_API=3 etcdctl snapshot save /backup/snap.db --endpoints=... --cacert=... --cert=... --key=...

# Check certs
kubeadm certs check-expiration

# Renew certs
kubeadm certs renew all

Managing Kubernetes Cluster via kubeadm ​

kubeadm Overview ​

Cluster Creation ​

Pre-requisites ​

Initialize Control Plane ​

Common Init Flags ​

Configure kubectl After Init ​

Join Nodes to Cluster ​

Generate Join Command ​

Join Worker Node ​

Join Additional Control Plane ​

Token Management ​

Cluster Upgrade ​

Upgrade Order (CRITICAL) ​

Upgrade Control Plane ​

Upgrade Additional Control Planes ​

Upgrade Worker Nodes ​

etcd Backup and Restore ​

Backup etcd ​

Restore etcd ​

etcd Key-Value Operations ​

Certificate Management ​

Check Certificate Expiration ​

Renew All Certificates ​

Renew Specific Certificate ​

Certificate Locations ​

kubeadm reset ​

High Availability Setup ​

HA Topologies ​

HA Requirements ​

HA Init Example ​

Troubleshooting ​

Common Issues ​

Useful Commands ​

Quick Reference ​

Managing Kubernetes Cluster via kubeadm

kubeadm Overview

Cluster Creation

Pre-requisites

Initialize Control Plane

Common Init Flags

Configure kubectl After Init

Join Nodes to Cluster

Generate Join Command

Join Worker Node

Join Additional Control Plane

Token Management

Cluster Upgrade

Upgrade Order (CRITICAL)

Upgrade Control Plane

Upgrade Additional Control Planes

Upgrade Worker Nodes

etcd Backup and Restore

Backup etcd

Restore etcd

etcd Key-Value Operations

Certificate Management

Check Certificate Expiration

Renew All Certificates

Renew Specific Certificate

Certificate Locations

kubeadm reset

High Availability Setup

HA Topologies

HA Requirements

HA Init Example

Troubleshooting

Common Issues

Useful Commands

Quick Reference