Practice Questions: Monitoring, Logging and Runtime Security

Practice Environment

These questions are designed for a Kind cluster or any local Kubernetes environment. Some questions require Falco to be installed on the node. Where node-level access is needed, instructions assume you can SSH into the node or use docker exec for Kind clusters.

Time Target: Aim to complete all 20 questions within 90 minutes to simulate exam pressure.

---

Question 1: Basic Audit Policy Creation

Difficulty: Easy

Scenario: Your cluster does not have audit logging enabled. The security team requires audit logging to track all API activity.

Task:

1. Create an audit policy file at /etc/kubernetes/audit/policy.yaml that:
   • Logs all Secret operations at the Metadata level
   • Logs all namespace create and delete operations at the RequestResponse level
   • Logs everything else at the Metadata level
   • Does not log RequestReceived stage events
2. The audit policy should be syntactically correct with the proper apiVersion and kind.

---

Question 2: Enable Audit Logging on API Server

Difficulty: Medium

Scenario: An audit policy has been created at /etc/kubernetes/audit/policy.yaml. You need to configure the API server to use it.

Task:

1. Configure the kube-apiserver to enable audit logging using the policy file at /etc/kubernetes/audit/policy.yaml
2. Audit logs should be written to /var/log/kubernetes/audit/audit.log
3. Set maximum log age to 30 days, maximum backup count to 5, and maximum log size to 100MB
4. Ensure all required volume mounts and volumes are configured
5. Verify the API server restarts successfully and audit events are being logged

---

Question 3: Advanced Audit Policy

Difficulty: Hard

Scenario: The security team needs a more granular audit policy. They want to minimize log volume while capturing security-relevant events.

Task: Create an audit policy file at /etc/kubernetes/audit/policy-advanced.yaml that:

1. Does NOT log get, list, or watch operations on endpoints or services
2. Does NOT log any requests to non-resource URLs like /api*, /healthz*, /version
3. Logs all Secret operations at the Metadata level (never log Secret contents)
4. Logs all RBAC resource modifications (create, update, patch, delete) at RequestResponse level
5. Logs pods/exec, pods/attach, and pods/portforward at Metadata level
6. Logs all other events at Metadata level
7. Omits the RequestReceived stage globally

---

Question 4: Audit Log Investigation

Difficulty: Medium

Scenario: The file /var/log/kubernetes/audit/audit.log contains audit events. A Deployment named webapp in the production namespace was recently deleted, and you need to find out who did it.

Task:

1. Write a jq command to filter the audit log and find who deleted the Deployment named webapp in the production namespace
2. Record the username, source IP, and timestamp of the deletion event
3. Save the filtered output to /tmp/investigation/deletion-event.json

---

Question 5: Write a Falco Rule for Shell Detection

Difficulty: Easy

Scenario: The security team wants Falco to detect when an interactive shell is spawned inside any container.

Task:

1. Create a Falco rule in /etc/falco/falco_rules.local.yaml that:
   • Has the name Detect Shell in Container
   • Detects when bash, sh, dash, or zsh is spawned in a container
   • Outputs the user name, shell name, container name, pod name, namespace, and command line
   • Has priority WARNING
   • Is tagged with shell and container
2. Validate the rule file
3. Restart Falco

---

Question 6: Falco Rule for Sensitive File Access

Difficulty: Medium

Scenario: Falco is running on the cluster nodes. You need to create a custom rule to detect when any process inside a container reads /etc/shadow.

Task:

1. Add a Falco rule to /etc/falco/falco_rules.local.yaml that:
   • Has the name Shadow File Read in Container
   • Detects reads of /etc/shadow inside containers
   • Outputs the process name, user, container name, pod name, namespace, and image
   • Has priority ERROR
2. Test the rule by exec-ing into a container and running cat /etc/shadow
3. Verify the Falco alert is generated

---

Question 7: Falco Rule for Package Manager Detection

Difficulty: Medium

Scenario: Container immutability is a security requirement. You need to detect any attempt to install packages inside running containers.

Task:

1. Create a Falco rule file at /etc/falco/rules.d/package-manager.yaml with:
   • A list named package_mgr_binaries containing: apt, apt-get, dpkg, yum, rpm, pip, pip3, npm, apk
   • A rule named Package Manager Launched in Container that detects execution of any package manager process inside a container
   • Priority CRITICAL
   • Output should include the package manager name, command line, container name, pod name, and namespace
2. Restart Falco and verify the rule is loaded

---

Question 8: Investigate Falco Alerts

Difficulty: Medium

Scenario: Falco has been generating alerts about a pod named web-app in the default namespace. The alerts indicate that a shell has been spawned in the container.

Task:

1. Check the Falco alerts from the last hour
2. Identify the specific container and process that triggered the alert
3. Investigate the pod and determine what processes are running
4. Check if any files have been modified in the container's /tmp directory
5. Record your findings in /tmp/investigation/falco-findings.txt

---

Question 9: Container Immutability - Fix a Pod

Difficulty: Easy

Scenario: A pod named mutable-app is running in the default namespace without readOnlyRootFilesystem. This violates the security policy.

Task:

1. Delete the existing mutable-app pod
2. Recreate it with the same image and configuration but with:
   • readOnlyRootFilesystem: true
   • An emptyDir volume mounted at /tmp
   • An emptyDir volume mounted at /var/cache
3. Verify the pod is running successfully

---

Question 10: Container Immutability - Nginx

Difficulty: Medium

Scenario: Deploy an Nginx pod that enforces container immutability while remaining fully functional.

Task:

1. Create a pod named immutable-nginx in the default namespace with:
   • Image: nginx:1.25
   • readOnlyRootFilesystem: true
   • All necessary writable directories provided via emptyDir volumes (nginx needs /tmp, /var/run, /var/cache/nginx, /var/log/nginx)
   • allowPrivilegeEscalation: false
2. Verify the pod is running and Nginx is serving on port 80
3. Verify that writing to the root filesystem (e.g., /usr/share/nginx/html/test.html) fails

---

Question 11: Forensic Investigation - Suspicious Container

Difficulty: Hard

Scenario: A container named suspicious-app in the investigation namespace is suspected of being compromised. You need to investigate without alerting the attacker.

Task:

1. Create the investigation namespace and deploy a pod named suspicious-app using nginx:1.25
2. Simulate compromise: exec into the pod and run:
   • apt-get update && apt-get install -y curl
   • curl -o /tmp/payload http://example.com
   • echo "malicious" > /tmp/backdoor.sh
3. Now investigate from the host:
   • Find the container ID using crictl
   • Find the host PID of the container
   • List processes running inside the container
   • Check what files exist in /tmp inside the container
   • Record your investigation results in /tmp/investigation/forensics-report.txt

---

Question 12: Audit Log Analysis - Secret Access

Difficulty: Medium

Scenario: You suspect that someone has been accessing Secrets they should not have access to. Audit logging is enabled.

Task:

1. Write commands to find all audit log entries where:
   • The resource type is secrets
   • The verb is get or list
   • Group the results by username
2. Identify any users or service accounts that accessed secrets in namespaces other than their own
3. Save the analysis to /tmp/investigation/secret-access.txt

---

Question 13: Behavioral Analytics - Detect Lateral Movement

Difficulty: Hard

Scenario: You need to set up detection for lateral movement attempts in your cluster.

Task:

1. Create a Falco rule in /etc/falco/rules.d/lateral-movement.yaml that detects:
   • Any container reading the service account token file (/var/run/secrets/kubernetes.io/serviceaccount/token)
   • With priority WARNING
   • Output includes process name, command, container name, pod name, namespace
2. Create another rule that detects:
   • Network scanning tools (nmap, masscan, nc, ncat, netcat) being executed in a container
   • With priority CRITICAL
   • Output includes the tool name, command line, container name, pod name, namespace
3. Restart Falco and verify both rules are loaded

---

Question 14: Combined - Audit Policy and Investigation

Difficulty: Hard

Scenario: A security incident occurred. You need to set up audit logging and investigate.

Task:

1. Create an audit policy at /etc/kubernetes/audit/incident-policy.yaml that:
   • Logs all pods/exec operations at RequestResponse level
   • Logs all Secret operations at Metadata level
   • Logs all RBAC changes at RequestResponse level
   • Logs everything else at Metadata level
2. Configure the API server to use this policy (logs to /var/log/kubernetes/audit/incident.log)
3. After the API server restarts, perform these actions:
   • Create a Secret named test-secret in the default namespace
   • Exec into a running pod
4. Verify these actions appear in the audit log

---

Question 15: Immutability Enforcement with Pod Security Standards

Difficulty: Medium

Scenario: You need to enforce security standards on a namespace to ensure all pods meet minimum security requirements.

Task:

1. Create a namespace named restricted-ns
2. Apply the restricted Pod Security Standard in enforce mode
3. Attempt to create a pod without security context -- verify it is rejected
4. Create a pod that meets the restricted standard with:
   • readOnlyRootFilesystem: true
   • runAsNonRoot: true
   • runAsUser: 1000
   • allowPrivilegeEscalation: false
   • capabilities: drop: [ALL]
   • seccompProfile: type: RuntimeDefault
   • Appropriate emptyDir volumes
5. Verify the pod runs successfully

---

Question 16: Falco - Modify Output Format

Difficulty: Easy

Scenario: The security team wants Falco alerts to be output in JSON format to a specific file.

Task:

1. Modify /etc/falco/falco.yaml to:
   • Enable JSON output
   • Enable file output to /var/log/falco/alerts.json
   • Keep stdout output enabled
2. Restart Falco
3. Trigger an alert (exec into a container)
4. Verify the alert appears in JSON format in the output file

---

Question 17: Forensic Investigation - Network Analysis

Difficulty: Hard

Scenario: A pod named network-suspect in the default namespace is suspected of making unauthorized outbound connections.

Task:

1. Create a pod network-suspect using nginx:1.25
2. Simulate suspicious activity: exec into the pod and run curl http://example.com
3. From the host, investigate:
   • Find the container's host PID
   • Check active network connections
   • Check listening ports
   • Check the processes that have network activity
4. Create a NetworkPolicy that isolates this pod (deny all ingress and egress)
5. Verify the pod can no longer make outbound connections

---

Question 18: sysdig Investigation

Difficulty: Medium

Scenario: You need to use sysdig to monitor system calls made by a specific container.

Task:

1. Create a pod named monitored-app in the default namespace using nginx:1.25
2. Use sysdig to:
   • Monitor all file open events in the monitored-app container
   • Monitor all process execution events in the monitored-app container
   • Monitor all network connection events in the monitored-app container
3. Write the sysdig commands that would produce each of these outputs
4. Save the commands to /tmp/investigation/sysdig-commands.txt

---

Question 19: Combined - Full Incident Response

Difficulty: Hard

Scenario: You receive a Falco alert indicating that a shell was spawned in a container in the production namespace. You need to perform a complete incident response.

Task:

1. Create the production namespace with a pod named app-server using nginx:1.25
2. Simulate the compromise: exec into the pod, create a file /tmp/malware.sh, and install curl
3. Perform incident response:
   • Collect evidence: pod logs, pod spec, running processes, filesystem changes
   • Save all evidence to /tmp/evidence/
   • Create a NetworkPolicy to isolate the pod
   • Document the timeline of events
4. Remediate:
   • Delete the compromised pod
   • Recreate it with immutable settings (readOnlyRootFilesystem: true)
   • Verify the remediated pod is running

---

Question 20: Comprehensive Security Monitoring Setup

Difficulty: Hard

Scenario: You need to set up a comprehensive security monitoring configuration for a new cluster.

Task:

1. Create an audit policy at /etc/kubernetes/audit/comprehensive-policy.yaml that:
   • Does not log get/list/watch on endpoints and services
   • Does not log kube-proxy watch requests
   • Does not log requests to /healthz*, /readyz*, /livez*
   • Logs Secrets at Metadata level
   • Logs RBAC changes at RequestResponse level
   • Logs pods/exec and pods/portforward at Request level
   • Logs all other events at Metadata level
2. Create a Falco rules file at /etc/falco/rules.d/comprehensive.yaml with rules to detect:
   • Shell spawning in containers (WARNING)
   • Package manager execution in containers (ERROR)
   • Sensitive file reads (/etc/shadow, /etc/passwd) in containers (WARNING)
   • Write operations to /etc in containers (ERROR)
3. Configure the API server to use the audit policy
4. Restart Falco to load the new rules
5. Verify both audit logging and Falco are operational