Securing etcd

Why etcd Security Matters

etcd is the key-value store that holds the entire state of the Kubernetes cluster -- every object, secret, configuration, and RBAC rule. If an attacker gains access to etcd, they have access to everything, including secrets in plain text (unless encryption at rest is configured).

Critical Component

Compromising etcd is equivalent to compromising the entire cluster. An attacker with etcd access can:

Read all secrets (including service account tokens, TLS certificates, passwords)
Modify any object (create admin users, deploy malicious workloads)
Delete all cluster data
Bypass all RBAC controls entirely

CKS Exam Relevance

You should know how to configure encryption at rest for secrets, verify etcd TLS configuration, and perform secure backups. The EncryptionConfiguration resource is a high-probability exam topic.

Encryption at Rest

By default, Kubernetes stores data in etcd in plain text. Encryption at rest ensures that sensitive data (especially Secrets) is encrypted before being written to etcd.

Encryption Flow

Step 1: Create the EncryptionConfiguration

yaml

# /etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets                    # Encrypt secrets
    providers:
      - aescbc:                    # AES-CBC encryption (recommended for CKS)
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}               # Fallback: no encryption (for reading old unencrypted data)

Generating the Encryption Key

bash

# Generate a random 32-byte key, base64 encoded
head -c 32 /dev/urandom | base64

This produces a string like aGVsbG93b3JsZGhlbGxvd29ybGRoZWxsb3dvcmxkaGU= that you use as the secret value.

Encryption Providers

Provider	Strength	Speed	Key Management
identity	None (plain text)	Fastest	N/A
aescbc	Strong (AES-256-CBC)	Moderate	Static key in config
aesgcm	Strong (AES-256-GCM)	Fast	Must rotate key frequently
secretbox	Strong (XSalsa20+Poly1305)	Fast	Static key in config
kms	Strongest	Varies	External KMS (Vault, AWS KMS, etc.)

Provider Order Matters

The first provider in the list is used for writing (encrypting new data). All providers in the list are tried for reading (decrypting existing data). Always keep identity: {} as the last provider to ensure old unencrypted data can still be read.

Step 2: Configure the API Server

Edit the API server manifest to use the encryption configuration:

yaml

# /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
        # ... other flags
      volumeMounts:
        - name: enc-config
          mountPath: /etc/kubernetes/enc
          readOnly: true
  volumes:
    - name: enc-config
      hostPath:
        path: /etc/kubernetes/enc
        type: DirectoryOrCreate

Do Not Forget Volume Mounts

If you add --encryption-provider-config without the corresponding volume mount, the API server will not start. Always add both the flag AND the volume mount.

Step 3: Encrypt Existing Secrets

After enabling encryption, only new secrets are encrypted. Existing secrets remain unencrypted until they are re-written:

bash

# Re-encrypt all secrets in all namespaces
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Step 4: Verify Encryption

bash

# Read a secret directly from etcd to verify it is encrypted
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret

# If encrypted, the output will contain garbled binary data prefixed with "k8s:enc:aescbc:v1:key1:"
# If NOT encrypted, you will see the secret value in plain text

Verification Is Key

In the exam, always verify that encryption is working by reading data directly from etcd. Look for the encryption prefix (k8s:enc:aescbc:v1:key1:) in the output. If you see plain text, the encryption configuration is not applied correctly.

Complete Encryption Setup Example

bash

# Step 1: Generate encryption key
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
echo "Generated key: $ENCRYPTION_KEY"

# Step 2: Create the encryption config directory and file
sudo mkdir -p /etc/kubernetes/enc

sudo cat > /etc/kubernetes/enc/encryption-config.yaml <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

# Step 3: Set proper permissions
sudo chmod 600 /etc/kubernetes/enc/encryption-config.yaml
sudo chown root:root /etc/kubernetes/enc/encryption-config.yaml

# Step 4: Edit the API server manifest (add flag + volume mount)
# (See the YAML above)

# Step 5: Wait for API server to restart
sleep 30
kubectl get nodes   # Verify API server is back

# Step 6: Create a test secret
kubectl create secret generic test-encryption -n default \
  --from-literal=mykey=myvalue

# Step 7: Verify encryption in etcd
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/test-encryption | hexdump -C | head -20

# Step 8: Re-encrypt all existing secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

etcd TLS Configuration

Client-Server TLS

yaml

# /etc/kubernetes/manifests/etcd.yaml
spec:
  containers:
    - command:
        - etcd
        # Client-facing TLS
        - --cert-file=/etc/kubernetes/pki/etcd/server.crt
        - --key-file=/etc/kubernetes/pki/etcd/server.key
        - --client-cert-auth=true
        - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
        # Listen on HTTPS only
        - --listen-client-urls=https://127.0.0.1:2379,https://192.168.1.100:2379
        - --advertise-client-urls=https://192.168.1.100:2379

Peer TLS (Multi-Node etcd)

yaml

        # Peer-to-peer TLS
        - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
        - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
        - --peer-client-cert-auth=true
        - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
        - --listen-peer-urls=https://192.168.1.100:2380
        - --initial-advertise-peer-urls=https://192.168.1.100:2380

Verifying TLS Configuration

bash

# Check etcd is listening on HTTPS
ss -tlnp | grep 2379

# Test connection with TLS
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

# Should output:
# https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.5ms

# Test WITHOUT TLS (should fail)
ETCDCTL_API=3 etcdctl \
  --endpoints=http://127.0.0.1:2379 \
  endpoint health
# Should fail with connection refused

Backup and Restore with Security

Secure Backup

bash

# Create an etcd snapshot (requires TLS certs)
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /tmp/etcd-backup.db

# Verify the snapshot
ETCDCTL_API=3 etcdctl snapshot status /tmp/etcd-backup.db --write-table

Secure the Backup File

bash

# Set restrictive permissions on the backup
chmod 600 /tmp/etcd-backup.db
chown root:root /tmp/etcd-backup.db

# Store in a secure location (not on the same node)
scp /tmp/etcd-backup.db backup-server:/backups/etcd/

Restore from Backup

bash

# Stop the API server and etcd
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sudo mv /etc/kubernetes/manifests/etcd.yaml /tmp/

# Wait for them to stop
sleep 15

# Restore the snapshot
ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-backup.db \
  --data-dir=/var/lib/etcd-restored

# Update etcd manifest to use the restored data directory
# Change --data-dir to /var/lib/etcd-restored in etcd.yaml

# Start etcd and API server
sudo mv /tmp/etcd.yaml /etc/kubernetes/manifests/
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

# Wait and verify
sleep 30
kubectl get nodes

Backup Security Considerations

etcd backups contain all cluster data in plain text, including secrets
Always encrypt backups at the storage level
Restrict access to backup files with proper permissions
Store backups in a separate, secure location
Regularly test restore procedures

etcd Security Checklist

Complete etcd Security Checklist

[ ] Client-to-server TLS enabled (--cert-file, --key-file, --trusted-ca-file)
[ ] Client certificate authentication enabled (--client-cert-auth=true)
[ ] Peer TLS enabled (--peer-cert-file, --peer-key-file, --peer-trusted-ca-file)
[ ] Peer client certificate authentication enabled (--peer-client-cert-auth=true)
[ ] Encryption at rest configured (--encryption-provider-config)
[ ] etcd only listens on HTTPS (no HTTP endpoints)
[ ] etcd data directory has restricted permissions (700)
[ ] etcd is only accessible from the API server (use firewall rules or network policies)
[ ] Regular backups with secure storage
[ ] Backup files have restrictive permissions (600)

etcdctl Command Reference

bash

# Set environment for convenience
export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key

# Now commands are shorter:
etcdctl endpoint health
etcdctl endpoint status --write-table
etcdctl member list --write-table

# Read a specific key
etcdctl get /registry/secrets/default/my-secret

# List all keys under a prefix
etcdctl get /registry/secrets/ --prefix --keys-only

# Count secrets stored in etcd
etcdctl get /registry/secrets/ --prefix --keys-only | wc -l

# Take a snapshot
etcdctl snapshot save /tmp/backup.db
etcdctl snapshot status /tmp/backup.db --write-table

Securing etcd ​

Why etcd Security Matters ​

Encryption at Rest ​

Encryption Flow ​

Step 1: Create the EncryptionConfiguration ​

Encryption Providers ​

Step 2: Configure the API Server ​

Step 3: Encrypt Existing Secrets ​

Step 4: Verify Encryption ​

Complete Encryption Setup Example ​

etcd TLS Configuration ​

Client-Server TLS ​

Peer TLS (Multi-Node etcd) ​

Verifying TLS Configuration ​

Backup and Restore with Security ​

Secure Backup ​

Secure the Backup File ​

Restore from Backup ​

etcd Security Checklist ​

etcdctl Command Reference ​

Securing etcd

Why etcd Security Matters

Encryption at Rest

Encryption Flow

Step 1: Create the EncryptionConfiguration

Encryption Providers

Step 2: Configure the API Server

Step 3: Encrypt Existing Secrets

Step 4: Verify Encryption

Complete Encryption Setup Example

etcd TLS Configuration

Client-Server TLS

Peer TLS (Multi-Node etcd)

Verifying TLS Configuration

Backup and Restore with Security

Secure Backup

Secure the Backup File

Restore from Backup

etcd Security Checklist

etcdctl Command Reference