Container Forensics

Overview

Container forensics is the process of investigating security incidents in containerized environments. Unlike traditional forensics where you can image a hard drive, container environments are ephemeral -- containers can be killed and restarted at any time, destroying critical evidence. Effective forensic investigation requires speed, methodology, and the right tools.

CKS Exam Relevance

The CKS exam may present scenarios where:

A container has been compromised and you need to identify what happened
You need to find evidence of malicious activity using available tools
You need to use crictl, kubectl, and Linux commands for investigation
You need to analyze audit logs to trace an attack

Forensic Investigation Workflow

Investigating Compromised Containers

Step 1: Identify the Compromised Container

bash

# Check Falco alerts for the container
journalctl -u falco --since "1 hour ago" | grep -i "warning\|error\|critical"

# List all running containers
crictl ps

# List all pods with their status
kubectl get pods -A -o wide

# Check for pods with unusual resource usage
kubectl top pods -A --sort-by=cpu

# Check for recently created or restarted pods
kubectl get pods -A --sort-by=.metadata.creationTimestamp

Step 2: Preserve Evidence Before It Disappears

Critical

Container evidence is ephemeral. If the pod is deleted or restarted, runtime data (processes, network connections, memory) is lost permanently. Always preserve evidence first.

bash

# Create an evidence directory
mkdir -p /tmp/evidence/$(date +%Y%m%d-%H%M%S)
EVIDENCE_DIR=/tmp/evidence/$(date +%Y%m%d-%H%M%S)

# Save the pod specification
kubectl get pod <pod-name> -n <namespace> -o yaml > $EVIDENCE_DIR/pod-spec.yaml

# Save pod logs
kubectl logs <pod-name> -n <namespace> > $EVIDENCE_DIR/pod-logs.txt
kubectl logs <pod-name> -n <namespace> --previous > $EVIDENCE_DIR/pod-logs-previous.txt 2>/dev/null

# Save pod events
kubectl describe pod <pod-name> -n <namespace> > $EVIDENCE_DIR/pod-describe.txt

# Save all events in the namespace
kubectl get events -n <namespace> --sort-by=.lastTimestamp > $EVIDENCE_DIR/events.txt

# Save the container ID for crictl investigation
CONTAINER_ID=$(crictl ps --name <container-name> -q)
echo "Container ID: $CONTAINER_ID" > $EVIDENCE_DIR/container-id.txt

# Save container inspect output
crictl inspect $CONTAINER_ID > $EVIDENCE_DIR/container-inspect.json

Examining Container Processes

Using kubectl exec

bash

# List running processes
kubectl exec <pod-name> -n <namespace> -- ps aux

# List all processes with full command lines
kubectl exec <pod-name> -n <namespace> -- ps auxwww

# Check process tree
kubectl exec <pod-name> -n <namespace> -- pstree -p

# Check for processes running as root
kubectl exec <pod-name> -n <namespace> -- ps aux | grep "^root"

# Check for suspicious listening ports
kubectl exec <pod-name> -n <namespace> -- ss -tlnp
kubectl exec <pod-name> -n <namespace> -- netstat -tlnp

Using crictl

bash

# List running containers
crictl ps

# List all containers (including stopped)
crictl ps -a

# Get container PID on the host
CONTAINER_ID=$(crictl ps --name <container-name> -q)
PID=$(crictl inspect $CONTAINER_ID | jq .info.pid)

# Execute commands in the container
crictl exec $CONTAINER_ID ps aux
crictl exec $CONTAINER_ID ls -la /tmp

# View container logs
crictl logs $CONTAINER_ID
crictl logs --tail 100 $CONTAINER_ID
crictl logs --since "2024-01-15T10:00:00Z" $CONTAINER_ID

Using /proc from the Host

bash

# Get the container's main process PID
PID=$(crictl inspect $CONTAINER_ID | jq .info.pid)

# View all processes in the container's PID namespace
ls /proc/$PID/root/proc/*/cmdline 2>/dev/null | while read f; do
  pid=$(echo $f | cut -d/ -f6)
  cmd=$(cat $f 2>/dev/null | tr '\0' ' ')
  echo "PID $pid: $cmd"
done

# Check the main process
cat /proc/$PID/cmdline | tr '\0' ' '; echo

# View the process executable
ls -la /proc/$PID/exe

# Check environment variables for sensitive data
cat /proc/$PID/environ | tr '\0' '\n' | sort

# View open files
ls -la /proc/$PID/fd/

# Check memory maps for injected libraries
cat /proc/$PID/maps | grep -v "\.so" | grep "rwx"

Examining Container Network Activity

Active Connections

bash

# Using kubectl exec
kubectl exec <pod-name> -n <namespace> -- ss -tuanp
kubectl exec <pod-name> -n <namespace> -- netstat -tuanp

# From the host using /proc
PID=$(crictl inspect $CONTAINER_ID | jq .info.pid)
cat /proc/$PID/net/tcp
cat /proc/$PID/net/tcp6
cat /proc/$PID/net/udp

# Using nsenter to enter the container network namespace
nsenter -t $PID -n ss -tuanp
nsenter -t $PID -n netstat -tuanp

# Check DNS resolution log
nsenter -t $PID -n cat /etc/resolv.conf

Decoding /proc/net/tcp

bash

# Script to decode /proc/net/tcp entries
PID=$(crictl inspect $CONTAINER_ID | jq .info.pid)

cat /proc/$PID/net/tcp | tail -n +2 | while read line; do
  local_hex=$(echo $line | awk '{print $2}')
  remote_hex=$(echo $line | awk '{print $3}')
  state=$(echo $line | awk '{print $4}')
  
  # Decode local address
  local_ip_hex=${local_hex%:*}
  local_port_hex=${local_hex#*:}
  local_port=$((16#$local_port_hex))
  
  # Decode remote address
  remote_ip_hex=${remote_hex%:*}
  remote_port_hex=${remote_hex#*:}
  remote_port=$((16#$remote_port_hex))
  
  # Decode state
  case $state in
    01) state_str="ESTABLISHED" ;;
    02) state_str="SYN_SENT" ;;
    06) state_str="TIME_WAIT" ;;
    0A) state_str="LISTEN" ;;
    *) state_str="OTHER($state)" ;;
  esac
  
  echo "Local port: $local_port, Remote port: $remote_port, State: $state_str"
done

Checking Container Filesystem Changes

Using docker diff (Docker runtime)

bash

# Show files added, changed, or deleted in the container
docker diff <container-id>

# Output format:
# A = Added
# C = Changed
# D = Deleted

# Example output:
# A /tmp/malicious-binary
# C /etc/passwd
# A /dev/shm/payload.sh

Using crictl and /proc (Containerd runtime)

bash

# Access the container's root filesystem from the host
PID=$(crictl inspect $CONTAINER_ID | jq .info.pid)

# List files in common attack locations
ls -la /proc/$PID/root/tmp/
ls -la /proc/$PID/root/dev/shm/
ls -la /proc/$PID/root/var/tmp/
ls -la /proc/$PID/root/run/

# Find recently modified files
find /proc/$PID/root/ -mmin -60 -type f 2>/dev/null | \
  grep -v "proc\|sys\|dev"

# Find files with suspicious permissions (setuid/setgid)
find /proc/$PID/root/ -perm /6000 -type f 2>/dev/null

# Find executable files in /tmp
find /proc/$PID/root/tmp/ -executable -type f 2>/dev/null

# Check if /etc/passwd was modified
cat /proc/$PID/root/etc/passwd

# Check for new users added
diff <(docker run --rm <original-image> cat /etc/passwd) \
     <(cat /proc/$PID/root/etc/passwd)

# Check for modified binaries
sha256sum /proc/$PID/root/usr/bin/ls
# Compare against known-good hash from the original image

Examining Specific Suspicious Files

bash

# Check a suspicious binary
file /proc/$PID/root/tmp/suspicious-binary
strings /proc/$PID/root/tmp/suspicious-binary | head -50
sha256sum /proc/$PID/root/tmp/suspicious-binary

# Check a suspicious script
cat /proc/$PID/root/tmp/payload.sh

# Check bash history (if exists)
cat /proc/$PID/root/root/.bash_history 2>/dev/null
cat /proc/$PID/root/home/*/.bash_history 2>/dev/null

Using crictl for Container Investigation

crictl is the CLI for CRI-compatible container runtimes (containerd, CRI-O). It is the primary container investigation tool in modern Kubernetes clusters.

Essential crictl Commands

bash

# List all running containers
crictl ps

# List all containers including stopped ones
crictl ps -a

# List pods
crictl pods

# Filter containers by name
crictl ps --name nginx

# Filter containers by state
crictl ps --state exited

# Get detailed container information
crictl inspect <container-id>

# Get just the PID
crictl inspect <container-id> | jq .info.pid

# Get the image used
crictl inspect <container-id> | jq .status.image

# Get container labels (includes pod name, namespace)
crictl inspect <container-id> | jq .status.labels

# Execute a command in the container
crictl exec <container-id> cat /etc/hostname
crictl exec -it <container-id> sh

# View container logs
crictl logs <container-id>
crictl logs --tail 50 <container-id>
crictl logs -f <container-id>

# View container stats
crictl stats <container-id>

# List images
crictl images

# Inspect an image
crictl inspecti <image-id>

Extracting Key Information from crictl inspect

bash

CONTAINER_ID=$(crictl ps --name suspicious -q)

# Get full inspect output
crictl inspect $CONTAINER_ID > /tmp/inspect.json

# Extract key fields
echo "PID: $(jq .info.pid /tmp/inspect.json)"
echo "Image: $(jq -r .status.image.image /tmp/inspect.json)"
echo "State: $(jq -r .status.state /tmp/inspect.json)"
echo "Created: $(jq -r .status.createdAt /tmp/inspect.json)"
echo "Pod: $(jq -r '.status.labels["io.kubernetes.pod.name"]' /tmp/inspect.json)"
echo "Namespace: $(jq -r '.status.labels["io.kubernetes.pod.namespace"]' /tmp/inspect.json)"

# Check security context
jq '.info.config.linux.security_context' /tmp/inspect.json

# Check mounts
jq '.info.config.mounts' /tmp/inspect.json

# Check environment variables
jq '.info.config.envs' /tmp/inspect.json

Analyzing Audit Logs for Compromise Indicators

Timeline Reconstruction

bash

# Build a timeline of events for a specific pod
POD_NAME="compromised-pod"
NAMESPACE="default"

cat /var/log/kubernetes/audit/audit.log | jq -r "
  select(
    .objectRef.name==\"$POD_NAME\" and
    .objectRef.namespace==\"$NAMESPACE\"
  ) |
  \"\(.requestReceivedTimestamp) \(.verb) \(.objectRef.resource)/\(.objectRef.subresource // \"\"  ) by \(.user.username) from \(.sourceIPs[0])\"
" | sort

# Example output:
# 2024-01-15T10:00:00Z create pods/ by admin from 10.0.0.5
# 2024-01-15T10:05:00Z create pods/exec by attacker from 192.168.1.100
# 2024-01-15T10:06:00Z get secrets/ by system:serviceaccount:default:default from 10.244.1.5

Identify Who Created a Suspicious Pod

bash

cat /var/log/kubernetes/audit/audit.log | jq '
  select(
    .objectRef.resource=="pods" and
    .objectRef.name=="suspicious-pod" and
    .verb=="create" and
    .stage=="ResponseComplete"
  ) | {
    user: .user.username,
    groups: .user.groups,
    sourceIP: .sourceIPs[0],
    time: .requestReceivedTimestamp,
    responseCode: .responseStatus.code
  }'

Find All exec Sessions into a Pod

bash

cat /var/log/kubernetes/audit/audit.log | jq '
  select(
    .objectRef.subresource=="exec" and
    .objectRef.name=="target-pod" and
    .verb=="create"
  ) | {
    user: .user.username,
    sourceIP: .sourceIPs[0],
    time: .requestReceivedTimestamp,
    command: .requestObject.command
  }'

Find Secret Access by a Compromised Service Account

bash

SA_NAME="system:serviceaccount:default:compromised-sa"

cat /var/log/kubernetes/audit/audit.log | jq "
  select(
    .user.username==\"$SA_NAME\" and
    .objectRef.resource==\"secrets\"
  ) | {
    verb: .verb,
    secret: .objectRef.name,
    namespace: .objectRef.namespace,
    time: .requestReceivedTimestamp,
    responseCode: .responseStatus.code
  }"

Find RBAC Changes (Persistence Indicators)

bash

cat /var/log/kubernetes/audit/audit.log | jq '
  select(
    (.objectRef.resource | test("roles|rolebindings|clusterroles|clusterrolebindings")) and
    (.verb | test("create|update|patch|delete"))
  ) | {
    verb: .verb,
    resource: .objectRef.resource,
    name: .objectRef.name,
    user: .user.username,
    time: .requestReceivedTimestamp
  }'

Collecting Evidence Without Destroying It

Order of Volatility

Evidence should be collected in order from most volatile (disappears first) to least volatile:

Priority	Evidence Type	Volatility	Collection Method
1	Running processes	Very High	`ps aux`, `/proc`
2	Network connections	Very High	`ss -tuanp`, `/proc/net/tcp`
3	Memory contents	Very High	Core dump (advanced)
4	Container logs	High	`kubectl logs`, `crictl logs`
5	Filesystem state	Medium	`find`, `ls`, container overlay
6	Pod specification	Medium	`kubectl get pod -o yaml`
7	Audit logs	Low	File on control plane node
8	Falco alerts	Low	Journal or output file
9	Container image	Very Low	Stored in registry

Non-Destructive Investigation Checklist

bash

# 1. DO NOT delete the pod yet
# 2. DO NOT exec into the pod (this generates its own events and may alert the attacker)

# 3. From the HOST, gather evidence non-destructively:

# Get container PID
CONTAINER_ID=$(crictl ps --name <name> -q)
PID=$(crictl inspect $CONTAINER_ID | jq .info.pid)

# Record processes without entering the container
cat /proc/$PID/root/proc/*/cmdline 2>/dev/null | tr '\0' '\n'

# Record network connections from the host
nsenter -t $PID -n ss -tuanp > /tmp/evidence/connections.txt

# Record open files from the host
ls -la /proc/$PID/fd/ > /tmp/evidence/open-files.txt

# Record filesystem changes from the host
find /proc/$PID/root/tmp -type f > /tmp/evidence/tmp-files.txt
find /proc/$PID/root/dev/shm -type f > /tmp/evidence/shm-files.txt

# Copy suspicious files from the host
cp /proc/$PID/root/tmp/suspicious-file /tmp/evidence/

# Record container spec
crictl inspect $CONTAINER_ID > /tmp/evidence/container-inspect.json

# Record pod spec
kubectl get pod <pod-name> -n <namespace> -o yaml > /tmp/evidence/pod-spec.yaml

# 4. NOW you can take containment actions

Exam Tip

In the CKS exam, you may be given a scenario where a container is compromised and asked to investigate. Follow this approach:

Read the question carefully -- it will tell you what to find
Use crictl and kubectl to identify the container/pod
Use the appropriate investigation command to find the evidence
Record your findings as requested (specific file, specific process, etc.)

Common Investigation Commands Reference

Quick Reference Card

bash

# === CONTAINER IDENTIFICATION ===
crictl ps                                    # List running containers
crictl ps -a                                 # List all containers
crictl pods                                  # List pods
crictl inspect <container-id> | jq .info.pid # Get host PID

# === PROCESS INVESTIGATION ===
crictl exec <id> ps aux                      # List processes in container
cat /proc/<pid>/cmdline | tr '\0' ' '        # Get command line
ls -la /proc/<pid>/exe                       # Get executable path
cat /proc/<pid>/environ | tr '\0' '\n'       # Get environment

# === NETWORK INVESTIGATION ===
crictl exec <id> ss -tuanp                   # Network connections
nsenter -t <pid> -n ss -tuanp               # From host
cat /proc/<pid>/net/tcp                      # Raw TCP connections

# === FILESYSTEM INVESTIGATION ===
ls -la /proc/<pid>/root/tmp/                 # Check /tmp
ls -la /proc/<pid>/root/dev/shm/            # Check /dev/shm
find /proc/<pid>/root/ -mmin -30 -type f     # Recently modified files
find /proc/<pid>/root/ -perm /6000 -type f   # Setuid/setgid files

# === LOG INVESTIGATION ===
kubectl logs <pod> -n <ns>                   # Pod logs
crictl logs <container-id>                   # Container logs
journalctl -u falco --since "1h ago"         # Falco alerts

# === AUDIT LOG INVESTIGATION ===
cat audit.log | jq 'select(.verb=="delete")' # Filter by verb
cat audit.log | jq 'select(.user.username=="X")' # Filter by user
cat audit.log | jq 'select(.objectRef.resource=="secrets")' # Filter by resource

Summary

Phase	Key Actions	Tools
Detection	Identify the incident from alerts	Falco, audit logs, monitoring
Preservation	Capture volatile evidence immediately	kubectl, crictl, /proc
Process Investigation	Examine running processes	ps, /proc, crictl exec
Network Investigation	Check connections and listening ports	ss, netstat, /proc/net
Filesystem Investigation	Find modified or new files	find, ls, file, strings
Audit Analysis	Trace API actions and build timeline	jq, audit logs
Containment	Isolate and prevent further damage	NetworkPolicy, cordon, delete
Documentation	Record all findings and actions	Text files, screenshots

Container Forensics ​

Overview ​

Forensic Investigation Workflow ​

Investigating Compromised Containers ​

Step 1: Identify the Compromised Container ​

Step 2: Preserve Evidence Before It Disappears ​

Examining Container Processes ​

Using kubectl exec ​

Using crictl ​

Using /proc from the Host ​

Examining Container Network Activity ​

Active Connections ​

Decoding /proc/net/tcp ​

Checking Container Filesystem Changes ​

Using docker diff (Docker runtime) ​

Using crictl and /proc (Containerd runtime) ​

Examining Specific Suspicious Files ​

Using crictl for Container Investigation ​

Essential crictl Commands ​

Extracting Key Information from crictl inspect ​

Analyzing Audit Logs for Compromise Indicators ​

Timeline Reconstruction ​

Identify Who Created a Suspicious Pod ​

Find All exec Sessions into a Pod ​

Find Secret Access by a Compromised Service Account ​

Find RBAC Changes (Persistence Indicators) ​

Collecting Evidence Without Destroying It ​

Order of Volatility ​

Non-Destructive Investigation Checklist ​

Common Investigation Commands Reference ​

Quick Reference Card ​

Summary ​

Container Forensics

Overview

Forensic Investigation Workflow

Investigating Compromised Containers

Step 1: Identify the Compromised Container

Step 2: Preserve Evidence Before It Disappears

Examining Container Processes

Using kubectl exec

Using crictl

Using /proc from the Host

Examining Container Network Activity

Active Connections

Decoding /proc/net/tcp

Checking Container Filesystem Changes

Using docker diff (Docker runtime)

Using crictl and /proc (Containerd runtime)

Examining Specific Suspicious Files

Using crictl for Container Investigation

Essential crictl Commands

Extracting Key Information from crictl inspect

Analyzing Audit Logs for Compromise Indicators

Timeline Reconstruction

Identify Who Created a Suspicious Pod

Find All exec Sessions into a Pod

Find Secret Access by a Compromised Service Account

Find RBAC Changes (Persistence Indicators)

Collecting Evidence Without Destroying It

Order of Volatility

Non-Destructive Investigation Checklist

Common Investigation Commands Reference

Quick Reference Card

Summary