Skip to content

Minimize IAM Roles and External Access

Overview

Kubernetes nodes exist at the intersection of three security domains: cloud provider IAM, operating system access control, and Kubernetes RBAC. A weakness in any one of these layers can compromise the others. This section covers minimizing the permissions and exposure at each layer -- ensuring nodes have only the cloud permissions they need, OS-level access is tightly controlled, and network exposure is limited to essential ports.

The principle of least privilege applies at every layer: cloud IAM roles attached to nodes, OS user accounts, SSH access, and firewall rules.

CKS Exam Relevance

The CKS exam tests your ability to restrict access to nodes and minimize attack surface. You may be asked to:

  • Block access to the cloud metadata API from pods using NetworkPolicies or iptables
  • Harden SSH configuration on a node
  • Configure firewall rules to allow only required Kubernetes ports
  • Verify file permissions on Kubernetes PKI and configuration files
  • Restrict which OS users can access the node

While cloud-specific IAM configurations (AWS IRSA, GCP Workload Identity) are unlikely to appear directly, understanding the concepts behind IMDS attacks and least-privilege IAM is expected.

Security Layers Around a Kubernetes Node

IAM Least Privilege for Kubernetes Nodes

Cloud IAM Roles on Node Instances

In cloud environments, every Kubernetes node (EC2 instance, GCE VM, Azure VM) has an IAM identity attached to it. This identity determines what cloud APIs the node can call.

Cloud ProviderNode IAM MechanismService Account Mechanism
AWSEC2 Instance Profile (IAM Role)IAM Roles for Service Accounts (IRSA)
GCPGCE Service AccountWorkload Identity
AzureManaged IdentityAzure AD Workload Identity

Why Nodes Should Have Minimal IAM Permissions

Kubernetes nodes need cloud IAM permissions for a limited set of operations:

OperationWhy It's Needed
Pull container images from private registryECR, GCR, ACR authentication
Manage network routesCNI plugin route configuration
Attach/detach storage volumesPersistent volume provisioning
Describe instancesNode registration and discovery
Write logs to cloud loggingCentralized log collection

Overly Permissive Node IAM Roles

If a node has broad IAM permissions (e.g., AdministratorAccess on AWS, Owner on GCP), any pod on that node can potentially access those permissions through the Instance Metadata Service. This means:

  • A compromised pod could read secrets from cloud secret managers
  • A pod could create/delete cloud resources (VMs, databases, storage)
  • A pod could escalate privileges by modifying IAM policies
  • A pod could exfiltrate data from cloud storage (S3, GCS, Blob Storage)

Always apply the principle of least privilege to node IAM roles.

The Instance Metadata Service (IMDS) Attack

Every major cloud provider exposes an Instance Metadata Service at a well-known link-local IP address. This service provides instance information and temporary credentials for the attached IAM role.

bash
# AWS -- retrieve IAM credentials from IMDS
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-name>

# GCP -- retrieve access token from metadata server
curl -H "Metadata-Flavor: Google" \
  http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token

# Azure -- retrieve access token from IMDS
curl -H "Metadata: true" \
  "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/"

IMDS Attack Path

Any pod running on a node can access the metadata service by default, because 169.254.169.254 is routable from within the pod network namespace. This is one of the most common cloud-native attack vectors:

  1. Attacker compromises a pod (e.g., via application vulnerability)
  2. Attacker queries 169.254.169.254 from inside the pod
  3. Attacker retrieves temporary cloud credentials for the node's IAM role
  4. Attacker uses those credentials to access cloud resources

Blocking IMDS Access from Pods

Method 1: iptables Rules on the Node

The most direct method is to block traffic to 169.254.169.254 from pod network interfaces at the node level.

bash
# Block all pod traffic to the metadata service
# This rule should be inserted BEFORE any ACCEPT rules in the FORWARD chain
sudo iptables -I FORWARD 1 \
  -d 169.254.169.254 \
  -j DROP \
  -m comment --comment "Block pod access to IMDS"

# Alternatively, block from specific pod CIDR
sudo iptables -I FORWARD 1 \
  -s 10.244.0.0/16 \
  -d 169.254.169.254 \
  -j DROP \
  -m comment --comment "Block pod CIDR to IMDS"

# Verify the rule is in place
sudo iptables -L FORWARD -n -v --line-numbers | grep 169.254.169.254

# Make the rule persistent (Debian/Ubuntu)
sudo apt install iptables-persistent
sudo netfilter-persistent save

AWS IMDSv2

AWS supports IMDSv2, which requires a PUT request to obtain a session token before querying metadata. While this mitigates some SSRF-based attacks, it does not prevent a pod with shell access from querying IMDS. Blocking at the network level is still necessary.

bash
# AWS: Enforce IMDSv2 on the instance (requires token-based access)
aws ec2 modify-instance-metadata-options \
  --instance-id i-1234567890abcdef0 \
  --http-tokens required \
  --http-endpoint enabled

Method 2: Kubernetes NetworkPolicy

If your CNI plugin supports it, you can block egress to 169.254.169.254 using a NetworkPolicy.

yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-metadata-access
  namespace: default
spec:
  podSelector: {}          # Apply to ALL pods in namespace
  policyTypes:
    - Egress
  egress:
    # Allow all egress EXCEPT metadata service
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32

NetworkPolicy Limitation

This approach requires a NetworkPolicy to be applied in every namespace. If you miss a namespace, pods in that namespace can still reach the metadata API. The iptables approach at the node level is more comprehensive.

Method 3: Per-Pod IAM with IRSA / Workload Identity

The best long-term solution is to decouple pod identity from node identity. Instead of pods inheriting the node's IAM role, each pod gets its own scoped credentials.

yaml
# AWS: Pod with IAM Role for Service Account (IRSA)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-reader
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/S3ReadOnlyRole
---
apiVersion: v1
kind: Pod
metadata:
  name: s3-app
  namespace: default
spec:
  serviceAccountName: s3-reader    # Pod uses this SA's IAM role, NOT the node role
  containers:
  - name: app
    image: amazon/aws-cli
    command: ["aws", "s3", "ls"]
yaml
# GCP: Pod with Workload Identity
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gcs-reader
  namespace: default
  annotations:
    iam.gke.io/gcp-service-account: gcs-reader@my-project.iam.gserviceaccount.com
---
apiVersion: v1
kind: Pod
metadata:
  name: gcs-app
  namespace: default
spec:
  serviceAccountName: gcs-reader
  containers:
  - name: app
    image: gcr.io/google.com/cloudsdktool/cloud-sdk
    command: ["gsutil", "ls"]

OS-Level User Access Control

Principle of Least Privilege for Node Users

Kubernetes nodes should have the minimum number of user accounts necessary. Each account should have only the permissions required for its function.

bash
# List all user accounts on the node
cat /etc/passwd | grep -v nologin | grep -v /bin/false

# List accounts with login shells
grep -E '/bin/(bash|sh|zsh)' /etc/passwd

# List users who can use sudo
getent group sudo
# or on RHEL-based systems
getent group wheel

# Check sudoers configuration
sudo cat /etc/sudoers
sudo ls -la /etc/sudoers.d/

Restricting SSH Access to Nodes

SSH is the primary remote access method for Kubernetes nodes. It must be carefully hardened.

Disable Direct Root Login

bash
# In /etc/ssh/sshd_config
PermitRootLogin no

Use Key-Based Authentication Only

bash
# In /etc/ssh/sshd_config
PasswordAuthentication no
PubkeyAuthentication yes
PermitEmptyPasswords no
ChallengeResponseAuthentication no

Limit SSH Access to Specific Users

bash
# In /etc/ssh/sshd_config
AllowUsers kubeadmin deploybot
# Or restrict by group
AllowGroups kubernetes-admins

SSH Hardening Settings

bash
# Recommended /etc/ssh/sshd_config settings for Kubernetes nodes

# Authentication limits
MaxAuthTries 3
LoginGraceTime 30
MaxSessions 3

# Disable unused features
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no

# Idle timeout (disconnect after 5 minutes of inactivity)
ClientAliveInterval 300
ClientAliveCountMax 0

# Use only strong algorithms
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com

# Logging
LogLevel VERBOSE
bash
# Apply SSH configuration changes
sudo sshd -t                    # Test configuration for syntax errors
sudo systemctl restart sshd     # Restart to apply

# Verify active settings
sudo sshd -T | grep -E '(permitrootlogin|passwordauthentication|maxauthtries|logingracetime|allowusers)'

Limiting sudo Access

bash
# Only allow specific users to sudo
# /etc/sudoers.d/kubernetes-admins

# Allow kubeadmin full sudo with password
kubeadmin ALL=(ALL) ALL

# Allow deploybot to only restart kubelet (no other commands)
deploybot ALL=(root) NOPASSWD: /usr/bin/systemctl restart kubelet

# Deny sudo for all other users (ensure they are NOT in the sudo/wheel group)
bash
# Remove a user from the sudo group
sudo deluser username sudo          # Debian/Ubuntu
sudo gpasswd -d username wheel      # RHEL/CentOS

# Verify sudo access for a user
sudo -l -U username

Using Bastion / Jump Hosts

Kubernetes nodes should never be directly accessible from the internet. All SSH access should go through a bastion (jump) host.

bash
# SSH through a bastion host using ProxyJump
ssh -J bastion-user@bastion.example.com kubeadmin@10.0.1.100

# Or configure in ~/.ssh/config
# Host k8s-node-*
#     ProxyJump bastion.example.com
#     User kubeadmin
#     IdentityFile ~/.ssh/k8s_ed25519
#
# Host bastion.example.com
#     User bastion-user
#     IdentityFile ~/.ssh/bastion_ed25519

Minimizing External Network Access to Nodes

Required Ports for Kubernetes

Kubernetes components communicate over specific ports. Only these ports should be open, and access should be restricted to the minimum set of sources.

Control Plane Node Ports

PortProtocolComponentPurposeSource
6443TCPkube-apiserverKubernetes APIWorkers, admins, LB
2379-2380TCPetcdetcd client and peerControl plane only
10250TCPkubeletKubelet APIControl plane
10259TCPkube-schedulerScheduler metricslocalhost
10257TCPkube-controller-managerController manager metricslocalhost

Worker Node Ports

PortProtocolComponentPurposeSource
10250TCPkubeletKubelet APIControl plane
10256TCPkube-proxyHealth checklocalhost
30000-32767TCPNodePort ServicesService exposureExternal (if needed)

Firewall Configuration with iptables

bash
# Flush existing rules (CAUTION: only on initial setup)
# sudo iptables -F

# Allow established connections
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow loopback
sudo iptables -A INPUT -i lo -j ACCEPT

# Allow SSH (restrict to bastion IP)
sudo iptables -A INPUT -p tcp --dport 22 -s 10.0.0.5/32 -j ACCEPT

# Allow kubelet API from control plane
sudo iptables -A INPUT -p tcp --dport 10250 -s 10.0.1.0/24 -j ACCEPT

# Allow NodePort range (if needed)
sudo iptables -A INPUT -p tcp --dport 30000:32767 -j ACCEPT

# Drop everything else
sudo iptables -A INPUT -j DROP

# Save rules
sudo netfilter-persistent save

Firewall Configuration for Control Plane Nodes

bash
# Allow API server from anywhere (or restrict to known CIDRs)
sudo iptables -A INPUT -p tcp --dport 6443 -j ACCEPT

# Allow etcd only from other control plane nodes
sudo iptables -A INPUT -p tcp --dport 2379:2380 -s 10.0.1.10/32 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 2379:2380 -s 10.0.1.11/32 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 2379:2380 -s 10.0.1.12/32 -j ACCEPT

# Allow kubelet API from within cluster
sudo iptables -A INPUT -p tcp --dport 10250 -s 10.0.0.0/16 -j ACCEPT

# Allow scheduler and controller-manager only on localhost
sudo iptables -A INPUT -p tcp --dport 10259 -s 127.0.0.1 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 10257 -s 127.0.0.1 -j ACCEPT

Firewall Configuration with ufw

bash
# Enable ufw
sudo ufw enable

# Default deny incoming
sudo ufw default deny incoming

# Allow SSH from bastion only
sudo ufw allow from 10.0.0.5 to any port 22

# Allow Kubernetes API server
sudo ufw allow 6443/tcp

# Allow kubelet
sudo ufw allow from 10.0.1.0/24 to any port 10250

# Allow NodePort range
sudo ufw allow 30000:32767/tcp

# Check status
sudo ufw status verbose

Service Account Considerations at OS Level

Running kubelet as a Dedicated System User

The kubelet should run as a dedicated system user with only the permissions it needs. On most distributions, kubeadm configures this correctly, but you should verify.

bash
# Check what user kubelet runs as
ps aux | grep kubelet

# Check kubelet service file
systemctl cat kubelet | grep -E '(User|Group)'

# Verify kubelet binary ownership
ls -la /usr/bin/kubelet

File Ownership and Permissions for Kubernetes

Critical Kubernetes files must have restrictive permissions. If an attacker gains access to these files, they can take over the cluster.

bash
# Control plane configuration files
sudo chmod 600 /etc/kubernetes/admin.conf
sudo chmod 600 /etc/kubernetes/scheduler.conf
sudo chmod 600 /etc/kubernetes/controller-manager.conf
sudo chmod 644 /etc/kubernetes/kubelet.conf

# Verify ownership (should be root:root)
sudo chown root:root /etc/kubernetes/*.conf

# Kubernetes PKI directory
sudo chmod 700 /etc/kubernetes/pki
sudo chmod 600 /etc/kubernetes/pki/*.key
sudo chmod 644 /etc/kubernetes/pki/*.crt

# etcd data directory
sudo chmod 700 /var/lib/etcd
sudo chown etcd:etcd /var/lib/etcd

# Check all permissions at once
ls -la /etc/kubernetes/
ls -la /etc/kubernetes/pki/
ls -la /var/lib/etcd/

Kubernetes PKI Directory

The /etc/kubernetes/pki directory contains private keys for the API server, etcd, and all cluster certificates. Anyone with read access to these keys can:

  • Impersonate the API server to any client
  • Decrypt etcd data (including all Secrets)
  • Issue certificates for any user or component
  • Take full control of the cluster

Always ensure permissions are 700 for the directory and 600 for .key files.

CIS Benchmark Checks for File Permissions

The CIS Kubernetes Benchmark specifies exact permissions for critical files:

File / DirectoryOwnerPermissions
/etc/kubernetes/manifests/root:root644 or more restrictive
/etc/kubernetes/admin.confroot:root600
/etc/kubernetes/scheduler.confroot:root600
/etc/kubernetes/controller-manager.confroot:root600
/etc/kubernetes/kubelet.confroot:root600
/etc/kubernetes/pki/root:root700
/etc/kubernetes/pki/*.keyroot:root600
/etc/kubernetes/pki/*.crtroot:root644
/var/lib/etcd/etcd:etcd700
/var/lib/kubelet/config.yamlroot:root644
bash
# Quick audit script for file permissions
echo "=== Kubernetes Config Files ==="
stat -c '%a %U:%G %n' /etc/kubernetes/*.conf 2>/dev/null

echo "=== PKI Directory ==="
stat -c '%a %U:%G %n' /etc/kubernetes/pki/ 2>/dev/null
stat -c '%a %U:%G %n' /etc/kubernetes/pki/*.key 2>/dev/null

echo "=== etcd Data ==="
stat -c '%a %U:%G %n' /var/lib/etcd/ 2>/dev/null

echo "=== Static Pod Manifests ==="
stat -c '%a %U:%G %n' /etc/kubernetes/manifests/*.yaml 2>/dev/null

Restricting Access to Cloud Metadata API (Detailed)

This section provides a comprehensive set of methods for blocking the cloud metadata API, which is critical for defense in depth.

Understanding the Attack

Defense Layer 1: iptables on Every Node

bash
# Create a script to apply IMDS blocking rules
cat <<'EOF' | sudo tee /usr/local/bin/block-imds.sh
#!/bin/bash
# Block pod access to cloud metadata service
# Run on every Kubernetes node

# Block in FORWARD chain (pod traffic goes through FORWARD)
iptables -C FORWARD -d 169.254.169.254/32 -j DROP 2>/dev/null || \
  iptables -I FORWARD 1 -d 169.254.169.254/32 \
    -j DROP \
    -m comment --comment "Block pod access to cloud IMDS"

# Also block in OUTPUT chain for host-network pods
iptables -C OUTPUT -d 169.254.169.254/32 -m owner --uid-owner 0 -j ACCEPT 2>/dev/null || \
  iptables -I OUTPUT 1 -d 169.254.169.254/32 -m owner --uid-owner 0 -j ACCEPT

iptables -C OUTPUT -d 169.254.169.254/32 -j DROP 2>/dev/null || \
  iptables -A OUTPUT -d 169.254.169.254/32 \
    -j DROP \
    -m comment --comment "Block non-root access to IMDS"

echo "IMDS blocking rules applied successfully"
iptables -L FORWARD -n --line-numbers | head -5
EOF

sudo chmod +x /usr/local/bin/block-imds.sh
sudo /usr/local/bin/block-imds.sh

Defense Layer 2: NetworkPolicy in Every Namespace

yaml
# Apply this NetworkPolicy in every namespace to block metadata access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-cloud-metadata
  namespace: default          # Repeat for every namespace
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    # Allow all egress EXCEPT the metadata IP
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32
bash
# Apply to all namespaces at once
for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
  cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-cloud-metadata
  namespace: $ns
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32
EOF
done

Defense Layer 3: Use Per-Pod IAM (IRSA / Workload Identity)

When pods need cloud access, use per-pod identity instead of node-level identity:

ApproachSecurityComplexityRecommendation
Node IAM role (default)Low -- all pods share one roleNoneAvoid for production
Block IMDS (iptables/NetworkPolicy)Medium -- blocks credential theftLowMinimum baseline
Per-pod IAM (IRSA/Workload Identity)High -- each pod gets scoped credentialsMediumBest practice
Block IMDS + Per-pod IAMHighest -- defense in depthMediumRecommended

Verifying IMDS Is Blocked

bash
# Deploy a test pod
kubectl run imds-test --image=curlimages/curl --rm -it --restart=Never -- \
  curl -s --max-time 3 http://169.254.169.254/latest/meta-data/

# Expected output: connection timeout or no response
# If you see metadata, IMDS is NOT blocked

# Test from a specific namespace
kubectl run imds-test -n production --image=curlimages/curl --rm -it --restart=Never -- \
  curl -s --max-time 3 http://169.254.169.254/latest/meta-data/

Hardening Checklist

Node Access Hardening Checklist

Cloud IAM:

  • [ ] Node IAM roles have only the minimum required permissions
  • [ ] IMDS access is blocked for pods (iptables and/or NetworkPolicy)
  • [ ] IMDSv2 is enforced (AWS)
  • [ ] Pods that need cloud access use per-pod identity (IRSA/Workload Identity)

SSH & OS Access:

  • [ ] Root login disabled (PermitRootLogin no)
  • [ ] Password authentication disabled (PasswordAuthentication no)
  • [ ] SSH limited to specific users (AllowUsers)
  • [ ] MaxAuthTries set to 3 or fewer
  • [ ] SSH idle timeout configured
  • [ ] Access via bastion/jump host only
  • [ ] Minimum number of user accounts on nodes
  • [ ] sudo restricted to authorized users only

Network / Firewall:

  • [ ] Only required Kubernetes ports are open
  • [ ] etcd ports restricted to control plane nodes only
  • [ ] Scheduler and controller-manager metrics on localhost only
  • [ ] NodePort range restricted to necessary sources
  • [ ] Default deny policy for inbound traffic

File Permissions:

  • [ ] /etc/kubernetes/pki/ is 700 and owned by root:root
  • [ ] Private keys (.key) are 600
  • [ ] admin.conf, scheduler.conf, controller-manager.conf are 600
  • [ ] /var/lib/etcd/ is 700 and owned by etcd:etcd

Quick Reference

Exam Speed Reference

bash
# Block IMDS from pods (iptables)
sudo iptables -I FORWARD 1 -d 169.254.169.254 -j DROP

# Verify IMDS is blocked
kubectl run test --image=curlimages/curl --rm -it --restart=Never -- \
  curl -s --max-time 3 http://169.254.169.254/

# Check SSH configuration
sudo sshd -T | grep -E '(permitrootlogin|passwordauthentication|maxauthtries)'

# Harden SSH quickly
sudo sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo sed -i 's/^#*MaxAuthTries.*/MaxAuthTries 3/' /etc/ssh/sshd_config
sudo systemctl restart sshd

# Check file permissions
stat -c '%a %U:%G %n' /etc/kubernetes/pki/*.key
stat -c '%a %U:%G %n' /etc/kubernetes/*.conf

# Fix PKI permissions
sudo chmod 700 /etc/kubernetes/pki
sudo chmod 600 /etc/kubernetes/pki/*.key

# List open ports on the node
sudo ss -tlnp

# Check firewall rules
sudo iptables -L -n -v --line-numbers

NetworkPolicy to block metadata API:

yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-cloud-metadata
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32

Key Exam Takeaways

  1. Block the metadata API -- use iptables (iptables -I FORWARD 1 -d 169.254.169.254 -j DROP) or NetworkPolicy to prevent pods from stealing cloud credentials via 169.254.169.254
  2. Harden SSH -- disable root login, enforce key-based auth, set MaxAuthTries 3, restrict with AllowUsers
  3. Least privilege IAM -- node IAM roles should only permit what the kubelet and node services need
  4. Firewall rules -- only open required ports (6443, 10250, 2379-2380, 30000-32767) and restrict source IPs
  5. File permissions -- PKI keys must be 600, PKI directory 700, config files 600, all owned by root:root
  6. Per-pod identity -- use IRSA (AWS) or Workload Identity (GCP) instead of relying on node IAM roles
  7. Bastion hosts -- nodes should not be directly accessible from the internet; use jump hosts for SSH
  8. On the exam, you may need to SSH into a node and fix SSH config, iptables rules, or file permissions

Released under the MIT License.