Kubernetes Secrets Security

Understanding Kubernetes Secrets

Kubernetes Secrets store sensitive data such as passwords, tokens, and certificates. However, Secrets have a critical misconception that every CKS candidate must understand.

Base64 is NOT Encryption

By default, Kubernetes Secrets are stored as base64-encoded plain text in etcd. Base64 is an encoding scheme, not encryption. Anyone with access to etcd or the API can trivially decode secrets:

bash

echo "cGFzc3dvcmQxMjM=" | base64 -d
# password123

This is the single most important fact about Secrets security on the CKS exam.

Secret Lifecycle and Encryption Flow

Types of Secrets

bash

# View available secret types
kubectl create secret --help

Type	Description	Use Case
`Opaque`	Arbitrary key-value data	Passwords, API keys
`kubernetes.io/tls`	TLS certificate + key	Ingress TLS
`kubernetes.io/dockerconfigjson`	Docker registry credentials	Image pull secrets
`kubernetes.io/basic-auth`	Username + password	Basic authentication
`kubernetes.io/ssh-auth`	SSH private key	SSH access
`kubernetes.io/service-account-token`	SA token (legacy)	API authentication

Creating Secrets

From Literal Values

bash

kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password='S3cur3P@ss!'

From Files

bash

kubectl create secret generic tls-secret \
  --from-file=cert.pem \
  --from-file=key.pem

From YAML (values must be base64-encoded)

yaml

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
  namespace: production
type: Opaque
data:
  username: YWRtaW4=           # echo -n "admin" | base64
  password: UzNjdXIzUEBzcyE=  # echo -n "S3cur3P@ss!" | base64

Using stringData (plain text, auto-encoded)

yaml

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
stringData:                     # Plain text - Kubernetes encodes it
  username: admin
  password: S3cur3P@ss!

Exam Tip

Use stringData in YAML manifests to avoid manual base64 encoding during the exam. It saves time and reduces errors.

Consuming Secrets: Volumes vs Environment Variables

As Environment Variables

yaml

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: app
    image: nginx:1.25
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-creds
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-creds
          key: password
    # Or load ALL keys as env vars:
    envFrom:
    - secretRef:
        name: db-creds

As Volume Mounts

yaml

apiVersion: v1
kind: Pod
metadata:
  name: secret-vol-pod
spec:
  containers:
  - name: app
    image: nginx:1.25
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true             # Always mount secrets read-only
  volumes:
  - name: secret-volume
    secret:
      secretName: db-creds
      defaultMode: 0400          # Restrictive file permissions

Security Comparison

Aspect	Environment Variables	Volume Mounts
Visibility	Visible via `/proc/*/environ`	Files only accessible in mount path
Logging risk	Can leak in logs, error reports, crash dumps	Less likely to leak
Updates	Requires pod restart	Auto-updated by kubelet
Child processes	Inherited by all child processes	Not inherited
Recommendation	Avoid for sensitive data	Preferred for sensitive data

Security Best Practice

Always prefer volume mounts over environment variables for secrets. Environment variables can be exposed through:

Application error logs that dump the environment
kubectl describe pod output (shows env var names)
Child processes inheriting the full environment
/proc/<pid>/environ inside the container

Encryption at Rest

Encryption at rest protects secrets stored in etcd by encrypting them before they are written to disk.

Step 1: Create the EncryptionConfiguration

yaml

# /etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets                    # Encrypt secrets
    providers:
    - aescbc:                    # Primary: encrypt with AES-CBC
        keys:
        - name: key1
          secret: <base64-encoded-32-byte-key>
    - identity: {}               # Fallback: read unencrypted secrets

Generating an Encryption Key

bash

# Generate a 32-byte random key, base64-encoded
head -c 32 /dev/urandom | base64

Step 2: Configure the API Server

Add the encryption configuration to the kube-apiserver static pod manifest:

yaml

# /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
    # ... other flags ...
    volumeMounts:
    - name: enc
      mountPath: /etc/kubernetes/enc
      readOnly: true
  volumes:
  - name: enc
    hostPath:
      path: /etc/kubernetes/enc
      type: DirectoryOrCreate

Step 3: Re-encrypt Existing Secrets

Critical Step

Enabling encryption at rest only affects new secrets. Existing secrets remain unencrypted until you re-encrypt them:

bash

# Re-encrypt all secrets in all namespaces
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Encryption Providers Comparison

Provider	Algorithm	Key Size	Speed	Recommendation
`aescbc`	AES-CBC	32 bytes	Moderate	Recommended for CKS exam
`secretbox`	XSalsa20+Poly1305	32 bytes	Fast	Strong, modern choice
`aesgcm`	AES-GCM	16/24/32 bytes	Fast	Must rotate keys frequently
`identity`	None (plain text)	N/A	Fastest	No encryption -- fallback only
`kms`	Envelope encryption	External KMS	Varies	Production recommended

Provider Order Matters

The first provider in the list is used for encryption. The remaining providers are used for decryption only. Always put identity: {} last so it can decrypt old unencrypted secrets but is never used for new encryption.

Full EncryptionConfiguration Examples

AES-CBC (Most Common on CKS Exam)

yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: dGhpcy1pcy1hLXRlc3Qta2V5LW11c3QtYmUtMzItYnl0
    - identity: {}

Secretbox (Modern, Recommended)

yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - secretbox:
        keys:
        - name: key1
          secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
    - identity: {}

Multiple Resources

yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    - configmaps          # Can encrypt ConfigMaps too
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: dGhpcy1pcy1hLXRlc3Qta2V5LW11c3QtYmUtMzItYnl0
    - identity: {}

Verifying Encryption in etcd

This is a common CKS exam task: verify that secrets are actually encrypted in etcd.

Check if a Secret is Encrypted

bash

# Read the secret directly from etcd
ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-creds \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# If ENCRYPTED: output starts with "k8s:enc:aescbc:v1:key1:" followed by binary data
# If NOT encrypted: output shows readable base64 or plain text

Verify Encryption is Working

bash

# Create a test secret
kubectl create secret generic test-encryption \
  --from-literal=mykey=myvalue

# Read it from etcd
ETCDCTL_API=3 etcdctl get /registry/secrets/default/test-encryption \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  | hexdump -C | head -20

# Look for "k8s:enc:aescbc" prefix in the output
# If you see plain "myvalue" text, encryption is NOT working

Exam Shortcut

The etcdctl command is long. On the exam, the etcd certificates are always at the standard paths. You can alias the command:

bash

alias etcdctl='ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'

Key Rotation

To rotate encryption keys, update the EncryptionConfiguration with a new key as the first entry:

yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key2                    # NEW key first (used for encryption)
          secret: <new-base64-key>
        - name: key1                    # OLD key second (used for decryption)
          secret: <old-base64-key>
    - identity: {}

After updating:

bash

# Restart the API server (it will pick up the new config)
# Then re-encrypt all secrets with the new key
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

# After verifying, remove the old key from the config

External Secret Management

While not directly tested with hands-on tasks, the CKS exam expects conceptual knowledge of external secret management.

HashiCorp Vault Concepts

Key concepts:

Vault stores secrets externally, not in etcd
Pods authenticate to Vault using Kubernetes Service Account tokens
Vault Agent sidecar injects secrets into pods
Secrets are never stored in Kubernetes objects
Dynamic secrets with automatic rotation

Secrets Store CSI Driver

yaml

# SecretProviderClass example (conceptual)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-secrets
spec:
  provider: vault
  parameters:
    roleName: "app-role"
    vaultAddress: "https://vault.example.com:8200"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/db"
        secretKey: "password"

Security Best Practices Summary

Practice	Implementation
Enable encryption at rest	EncryptionConfiguration with aescbc/secretbox
Use volume mounts, not env vars	`volumes[].secret` instead of `env[].valueFrom.secretKeyRef`
Restrict secret access with RBAC	Minimal roles -- only grant `get` on specific secrets
Mount secrets read-only	`readOnly: true` on volumeMounts
Set restrictive file permissions	`defaultMode: 0400` on secret volumes
Avoid secrets in pod specs	Use external secret management for production
Rotate encryption keys regularly	Update EncryptionConfiguration, re-encrypt
Verify encryption in etcd	Use etcdctl to confirm encrypted prefix
Disable automount of SA tokens	`automountServiceAccountToken: false`
Use short-lived tokens	Bound service account tokens (default in v1.24+)

Exam Priority

On the CKS exam, focus on:

Setting up EncryptionConfiguration (most likely task)
Verifying secrets in etcd (common verification step)
Fixing pods that expose secrets via environment variables
RBAC for secret access (covered in Domain 1)

Kubernetes Secrets Security ​

Understanding Kubernetes Secrets ​

Secret Lifecycle and Encryption Flow ​

Types of Secrets ​

Creating Secrets ​

From Literal Values ​

From Files ​

From YAML (values must be base64-encoded) ​

Using stringData (plain text, auto-encoded) ​

Consuming Secrets: Volumes vs Environment Variables ​

As Environment Variables ​

As Volume Mounts ​

Security Comparison ​

Encryption at Rest ​

Step 1: Create the EncryptionConfiguration ​

Step 2: Configure the API Server ​

Step 3: Re-encrypt Existing Secrets ​

Encryption Providers Comparison ​

Full EncryptionConfiguration Examples ​

AES-CBC (Most Common on CKS Exam) ​

Secretbox (Modern, Recommended) ​

Multiple Resources ​

Verifying Encryption in etcd ​

Check if a Secret is Encrypted ​

Verify Encryption is Working ​

Key Rotation ​

External Secret Management ​

HashiCorp Vault Concepts ​

Secrets Store CSI Driver ​

Security Best Practices Summary ​

Kubernetes Secrets Security

Understanding Kubernetes Secrets

Secret Lifecycle and Encryption Flow

Types of Secrets

Creating Secrets

From Literal Values

From Files

From YAML (values must be base64-encoded)

Using stringData (plain text, auto-encoded)

Consuming Secrets: Volumes vs Environment Variables

As Environment Variables

As Volume Mounts

Security Comparison

Encryption at Rest

Step 1: Create the EncryptionConfiguration

Step 2: Configure the API Server

Step 3: Re-encrypt Existing Secrets

Encryption Providers Comparison

Full EncryptionConfiguration Examples

AES-CBC (Most Common on CKS Exam)

Secretbox (Modern, Recommended)

Multiple Resources

Verifying Encryption in etcd

Check if a Secret is Encrypted

Verify Encryption is Working

Key Rotation

External Secret Management

HashiCorp Vault Concepts

Secrets Store CSI Driver

Security Best Practices Summary