System Call Monitoring: sysdig and strace

Overview

System call monitoring is the foundation of runtime security. Every action a process takes -- reading a file, opening a network connection, executing a binary -- involves a system call (syscall) to the Linux kernel. Tools like sysdig and strace allow you to observe these syscalls in real time, providing deep visibility into container and process behavior.

CKS Exam Relevance

While sysdig and strace are less commonly tested than Falco and audit logging, you may encounter questions that require:

Using sysdig to trace container activity
Investigating a process with strace
Exploring the /proc filesystem for forensic evidence
Identifying suspicious syscall patterns

System Call Flow

Understanding how system calls flow from a container through the kernel is essential for runtime monitoring.

Key System Calls for Security Monitoring

System Call	Purpose	Security Relevance
`execve` / `execveat`	Execute a new program	Detects new process execution (shells, malware)
`open` / `openat`	Open a file	Detects file access to sensitive paths
`read` / `write`	Read/write data	Detects data exfiltration or modification
`connect`	Establish network connection	Detects outbound connections (C2, mining)
`accept`	Accept incoming connection	Detects reverse shells, unauthorized services
`bind`	Bind to a port	Detects unexpected network services
`clone` / `fork`	Create a new process	Detects process spawning
`setuid` / `setgid`	Change user/group ID	Detects privilege escalation
`mount`	Mount a filesystem	Detects container escape attempts
`ptrace`	Trace a process	Detects debugging or process injection
`socket`	Create a network socket	Detects network activity
`unlink`	Delete a file	Detects evidence destruction

sysdig Overview

sysdig is a system exploration and troubleshooting tool with deep container support. It captures system calls and events from the Linux kernel and provides powerful filtering and analysis capabilities. Falco is built on top of sysdig's libraries.

Basic sysdig Usage

bash

# Capture all events (very verbose -- use filters)
sysdig

# Capture events for a specific container
sysdig container.name=nginx

# Capture events for a specific process
sysdig proc.name=bash

# Capture to a file for later analysis
sysdig -w capture.scap

# Read from a capture file
sysdig -r capture.scap

# Display output in JSON format
sysdig -j container.name=nginx

sysdig Event Format

Each sysdig event follows this format:

<timestamp> <cpu> <process> <thread> <direction> <event_type> <arguments>

Example:

10:30:00.123456 0 nginx (1234) < open fd=5 name=/etc/nginx/nginx.conf flags=1(O_RDONLY)

Field	Description
`10:30:00.123456`	Timestamp
`0`	CPU number
`nginx`	Process name
`(1234)`	Thread ID
`<`	Direction: `>` = enter syscall, `<` = exit syscall
`open`	System call type
`fd=5 name=...`	Arguments

Common sysdig Filters for Security

sysdig filters use a class.field syntax. Here are the most useful security-related filters:

Process Filters

bash

# Filter by process name
sysdig proc.name=bash

# Filter by process ID
sysdig proc.pid=1234

# Filter by parent process name
sysdig proc.pname=nginx

# Filter by user name
sysdig user.name=root

# Filter by user ID
sysdig user.uid=0

# Filter by command line
sysdig proc.cmdline contains "curl"

Container Filters

bash

# Filter by container name
sysdig container.name=web-server

# Filter by container ID
sysdig container.id=abc123def

# Show only container events (exclude host)
sysdig container.id != host

# Filter by container image
sysdig container.image contains nginx

# Filter by Kubernetes pod name
sysdig k8s.pod.name=web-pod

# Filter by Kubernetes namespace
sysdig k8s.ns.name=production

Event Type Filters

bash

# Filter by event type
sysdig evt.type=open

# Filter by multiple event types
sysdig "evt.type in (open, openat, read, write)"

# Filter by event direction (enter/exit)
sysdig evt.dir="<"

# Filter for failed events
sysdig evt.failed=true

File Filters

bash

# Filter by file name
sysdig fd.name=/etc/shadow

# Filter by file name pattern
sysdig fd.name contains /etc/

# Filter by file directory
sysdig fd.directory=/etc

# Filter for file operations
sysdig "evt.type in (open, openat, openat2) and fd.name contains /etc/"

Network Filters

bash

# Filter by connection IP
sysdig fd.ip=10.0.0.5

# Filter by connection port
sysdig fd.port=443

# Filter by server port
sysdig fd.sport=8080

# Filter for network events
sysdig "evt.type in (connect, accept, bind)"

# Filter for specific protocol
sysdig fd.l4proto=tcp

Combining Filters

bash

# Find file opens by root in a container
sysdig "container.name=web and user.uid=0 and evt.type=open"

# Find network connections from a specific container
sysdig "container.name=app and evt.type=connect"

# Find failed file access attempts
sysdig "container.id != host and evt.type=open and evt.failed=true"

# Find process execution in containers
sysdig "container.id != host and evt.type=execve"

# Find writes to /etc in any container
sysdig "container.id != host and evt.type in (open, openat) and evt.is_open_write=true and fd.name startswith /etc"

sysdig Chisels

Chisels are pre-built analysis scripts for common tasks.

bash

# List available chisels
sysdig -cl

# Top processes by CPU
sysdig -c topprocs_cpu

# Top file I/O by process
sysdig -c topfiles_bytes

# Top network connections
sysdig -c topconns

# Show interactive process execution (spy_users)
sysdig -c spy_users

# Show container activity
sysdig -c spy_users container.name=web

# Show file I/O for a container
sysdig -c topfiles_bytes container.name=web

# Network activity for a container
sysdig -c topconns container.name=web

# Show executed commands
sysdig -c spy_users "container.id != host"

Exam Tip

The spy_users chisel is extremely useful for seeing what commands are being executed inside containers in real time. Use it as a first step when investigating suspicious activity.

strace Basics

strace uses the ptrace system call to intercept and record system calls made by a specific process. Unlike sysdig (which works at the kernel level for all processes), strace attaches to a single process.

Basic strace Usage

bash

# Trace a command
strace ls /tmp

# Trace a running process by PID
strace -p 1234

# Trace a process and its children
strace -f -p 1234

# Trace specific system calls only
strace -e trace=open,read,write -p 1234

# Trace file-related system calls
strace -e trace=file -p 1234

# Trace network-related system calls
strace -e trace=network -p 1234

# Trace process-related system calls
strace -e trace=process -p 1234

# Count system calls (summary)
strace -c -p 1234

# Write output to a file
strace -o /tmp/trace.log -p 1234

# Show timestamps
strace -t -p 1234

# Show relative timestamps
strace -r -p 1234

strace Trace Categories

Category	System Calls Included
`file`	open, stat, chmod, unlink, rename, etc.
`process`	fork, clone, execve, wait, etc.
`network`	socket, connect, accept, bind, send, recv, etc.
`signal`	kill, sigaction, sigprocmask, etc.
`ipc`	shmget, semget, msgget, etc.
`memory`	mmap, mprotect, brk, etc.

Using strace with Containers

To trace a process inside a container, you need the PID on the host (not the PID inside the container).

bash

# Find the container's PID on the host
# Method 1: Using crictl
CONTAINER_ID=$(crictl ps --name web -q)
PID=$(crictl inspect $CONTAINER_ID | jq .info.pid)

# Method 2: Using docker
PID=$(docker inspect --format '{{.State.Pid}}' <container-id>)

# Method 3: Using /proc
# Find the process in /proc that matches your container

# Now trace the process
strace -f -p $PID

# Trace file access only
strace -e trace=file -f -p $PID

# Trace network activity only
strace -e trace=network -f -p $PID

WARNING

strace introduces significant overhead because it uses ptrace to intercept every system call. It should be used for short-term debugging, not continuous monitoring. For ongoing monitoring, use Falco or sysdig.

/proc Filesystem Investigation

The /proc filesystem is a virtual filesystem exposed by the Linux kernel that provides detailed information about running processes. It is invaluable for forensic investigation.

Key /proc Paths for Each Process

For a process with PID <pid>:

Path	Description
`/proc/<pid>/cmdline`	Full command line of the process
`/proc/<pid>/exe`	Symlink to the actual executable binary
`/proc/<pid>/environ`	Environment variables
`/proc/<pid>/fd/`	Directory of open file descriptors
`/proc/<pid>/maps`	Memory mappings
`/proc/<pid>/status`	Process status (UID, GID, state, etc.)
`/proc/<pid>/cgroup`	Cgroup membership (identifies container)
`/proc/<pid>/ns/`	Namespace information
`/proc/<pid>/net/tcp`	TCP connections for the process namespace
`/proc/<pid>/root/`	Root filesystem of the process
`/proc/<pid>/mountinfo`	Mount information

Practical /proc Investigation Commands

bash

# Find the PID of a container process
PID=<host_pid_of_container_process>

# View the process command line
cat /proc/$PID/cmdline | tr '\0' ' '; echo

# View the actual executable
ls -la /proc/$PID/exe

# View environment variables
cat /proc/$PID/environ | tr '\0' '\n'

# List open file descriptors
ls -la /proc/$PID/fd/

# Check which container the process belongs to
cat /proc/$PID/cgroup

# View the process status (UID, GID, etc.)
cat /proc/$PID/status

# View network connections from the process namespace
cat /proc/$PID/net/tcp
# Note: The addresses are in hex format

# View the process memory map
cat /proc/$PID/maps

# Access the container's root filesystem from the host
ls /proc/$PID/root/

# Check if a suspicious binary exists in the container
ls -la /proc/$PID/root/tmp/
ls -la /proc/$PID/root/dev/shm/

# View the process namespace IDs
ls -la /proc/$PID/ns/

Converting /proc/net/tcp Hex Addresses

Network connections in /proc/<pid>/net/tcp are stored in hexadecimal. Here is how to read them:

bash

# View raw TCP connections
cat /proc/$PID/net/tcp

# Output format:
# sl  local_address rem_address   st tx_queue rx_queue ...
#  0: 0100007F:1F90 00000000:0000 0A 00000000:00000000 ...

# Convert hex IP to decimal:
# 0100007F = 127.0.0.1 (read each byte in reverse)
# 01=1, 00=0, 00=0, 7F=127 -> 127.0.0.1

# Convert hex port to decimal:
# 1F90 = 8080

# Connection states:
# 0A = LISTEN
# 01 = ESTABLISHED
# 06 = TIME_WAIT

Exam Tip

You can use a one-liner to decode /proc/net/tcp:

bash

# Quick decode of listening ports
awk '{print $2}' /proc/$PID/net/tcp | tail -n +2 | while read line; do
  port=$((16#${line##*:}))
  echo "Port: $port"
done

Practical Investigation Scenarios

Scenario 1: Find What Files a Container Is Accessing

bash

# Using sysdig
sysdig -p "%evt.time %proc.name %fd.name" \
  "container.name=suspicious-app and evt.type in (open, openat) and evt.dir=<"

# Using strace
PID=$(crictl inspect $(crictl ps --name suspicious-app -q) | jq .info.pid)
strace -e trace=file -f -p $PID -o /tmp/file_trace.log

Scenario 2: Identify Network Connections from a Container

bash

# Using sysdig
sysdig -p "%evt.time %proc.name %fd.name" \
  "container.name=suspicious-app and evt.type=connect and evt.dir=<"

# Using strace
strace -e trace=network -f -p $PID

# Using /proc
cat /proc/$PID/net/tcp
cat /proc/$PID/net/tcp6

Scenario 3: Detect Unexpected Process Execution

bash

# Using sysdig -- watch for all execve calls in containers
sysdig -p "%evt.time %container.name %proc.name %proc.cmdline" \
  "container.id != host and evt.type=execve and evt.dir=<"

# Using spy_users chisel
sysdig -c spy_users "container.id != host"

Scenario 4: Investigate a Suspicious Binary

bash

# Find the binary
PID=<suspicious_process_pid>

# Check the executable
ls -la /proc/$PID/exe
file /proc/$PID/exe

# Check the command line
cat /proc/$PID/cmdline | tr '\0' ' '; echo

# Check where it is on the container filesystem
readlink /proc/$PID/exe

# Check if it was recently modified
stat /proc/$PID/root/$(readlink /proc/$PID/exe)

# Check the binary's hash
sha256sum /proc/$PID/exe

Scenario 5: Monitor Container for Evidence of Compromise

bash

# Run a comprehensive monitoring command
sysdig -p "%evt.time %evt.type %proc.name %proc.cmdline %fd.name %user.name" \
  "container.name=target-container and (
    evt.type=execve or
    (evt.type in (open, openat) and fd.name startswith /etc) or
    evt.type=connect or
    evt.type in (setuid, setgid)
  )" | tee /tmp/monitoring.log

Summary

Tool	Best For	Overhead	Scope
sysdig	Real-time syscall monitoring with container awareness	Low (kernel module)	System-wide
strace	Deep debugging of a specific process	High (ptrace)	Single process
/proc	Point-in-time process inspection	None (read-only)	Single process
Falco	Continuous rule-based threat detection	Low (kernel module/eBPF)	System-wide

Exam Strategy

For the CKS exam:

Use Falco alerts as your first indicator of trouble
Use sysdig to investigate real-time container behavior
Use /proc to examine specific processes and their state
Use strace only when you need detailed syscall-level debugging of a single process

System Call Monitoring: sysdig and strace ​

Overview ​

System Call Flow ​

Key System Calls for Security Monitoring ​

sysdig Overview ​

Basic sysdig Usage ​

sysdig Event Format ​

Common sysdig Filters for Security ​

Process Filters ​

Container Filters ​

Event Type Filters ​

File Filters ​

Network Filters ​

Combining Filters ​

sysdig Chisels ​

strace Basics ​

Basic strace Usage ​

strace Trace Categories ​

Using strace with Containers ​

/proc Filesystem Investigation ​

Key /proc Paths for Each Process ​

Practical /proc Investigation Commands ​

Converting /proc/net/tcp Hex Addresses ​

Practical Investigation Scenarios ​

Scenario 1: Find What Files a Container Is Accessing ​

Scenario 2: Identify Network Connections from a Container ​

Scenario 3: Detect Unexpected Process Execution ​

Scenario 4: Investigate a Suspicious Binary ​

Scenario 5: Monitor Container for Evidence of Compromise ​

Summary ​

System Call Monitoring: sysdig and strace

Overview

System Call Flow

Key System Calls for Security Monitoring

sysdig Overview

Basic sysdig Usage

sysdig Event Format

Common sysdig Filters for Security

Process Filters

Container Filters

Event Type Filters

File Filters

Network Filters

Combining Filters

sysdig Chisels

strace Basics

Basic strace Usage

strace Trace Categories

Using strace with Containers

/proc Filesystem Investigation

Key /proc Paths for Each Process

Practical /proc Investigation Commands

Converting /proc/net/tcp Hex Addresses

Practical Investigation Scenarios

Scenario 1: Find What Files a Container Is Accessing

Scenario 2: Identify Network Connections from a Container

Scenario 3: Detect Unexpected Process Execution

Scenario 4: Investigate a Suspicious Binary

Scenario 5: Monitor Container for Evidence of Compromise

Summary