Lesson 04 — Pod Investigation

Learning Objectives

By the end of this lesson, you will be able to:

Explain the purpose of Kubernetes Pod investigation
Identify suspicious or compromised Pods
Preserve Pod metadata and runtime evidence
Review container images, processes and logs
Investigate Pod security contexts and privileges
Analyse Service Account and workload identity usage
Review mounted Secrets, ConfigMaps and volumes
Examine Pod networking and active connections
Identify persistence and lateral-movement indicators
Correlate Pod activity with Kubernetes Audit Logs and AWS telemetry
Contain a compromised Pod without unnecessarily destroying evidence
Build an enterprise Pod-investigation workflow

Why This Matters

Pods are where most Kubernetes applications run.

A compromised Pod may provide an attacker with access to:

Application data
Kubernetes Service Account tokens
AWS workload credentials
Mounted Secrets
Persistent volumes
Internal services
Databases
Other Pods
Kubernetes APIs
Cloud services

Because Pods are temporary, valuable evidence may disappear when a Pod is:

Restarted
Rescheduled
Scaled down
Replaced by a Deployment
Evicted
Deleted
Terminated during node maintenance

Application Vulnerability

↓

Pod Compromise

↓

Credential Theft

↓

Internal Reconnaissance

↓

Lateral Movement

↓

Data Exfiltration

A Cloud Security Engineer must investigate quickly while balancing:

Evidence preservation
Threat containment
Application availability
Business impact
Recovery requirements

What is Pod Investigation?

Pod investigation is the structured process of collecting, preserving and analysing evidence related to a Kubernetes Pod suspected of malicious, unauthorised or abnormal activity.

The investigation aims to determine:

What happened inside the Pod?
Which container was affected?
How did the attacker gain access?
Which identity did the Pod use?
Which Secrets or volumes were accessible?
Which internal or external systems were contacted?
Did the activity spread beyond the Pod?
Was the worker node affected?
What should be contained, revoked or rebuilt?

Pod Investigation Scope

Pod Investigation

├── Pod Metadata
├── Container Images
├── Container Processes
├── Application Logs
├── Security Context
├── Service Account
├── Workload IAM Role
├── Secrets and ConfigMaps
├── Volumes
├── Network Connections
├── Kubernetes Events
├── Audit Logs
└── Runtime Security Alerts

When Should a Pod Be Investigated?

Begin a Pod investigation when security or operational monitoring identifies:

Interactive shell execution
Reverse-shell behaviour
Malware execution
Cryptomining
Unexpected process creation
Suspicious outbound connections
Secret access
Unapproved image deployment
Privileged container usage
HostPath access
Runtime socket access
Service Account token abuse
Unexpected CPU or memory usage
Security-agent alerts
Unauthorised configuration changes
Connections to suspicious domains or IP addresses
Pod creation by an unknown identity

Common Pod-Level Incidents

Incident	Example
Application compromise	Remote code execution in a web service
Credential theft	Service Account token copied from the Pod
Malware execution	Downloaded binary running from `/tmp`
Reverse shell	Shell connected to an external host
Cryptomining	High CPU usage and mining-pool traffic
Privilege escalation	Container gaining elevated Linux capabilities
Data exfiltration	Large outbound data transfer
Lateral movement	Pod scanning other cluster services
Malicious image	Workload running an unapproved image
Secret exposure	Mounted credentials read by an attacker

Pod Investigation Principles

Follow these principles during an investigation:

Preserve evidence before deleting the Pod.
Record every investigative action.
Collect volatile evidence first.
Avoid changing the container unnecessarily.
Use approved forensic tools.
Assume mounted credentials may be compromised.
Investigate all containers in the Pod.
Correlate Kubernetes and AWS evidence.
Escalate to node forensics when host compromise is suspected.
Rebuild from a trusted image rather than repairing a compromised container.

Pod Investigation Lifecycle

Alert Received

↓

Identify Pod

↓

Preserve Metadata

↓

Collect Runtime Evidence

↓

Review Identity and Access

↓

Analyse Network and Storage

↓

Determine Scope

↓

Contain Workload

↓

Eradicate Root Cause

↓

Redeploy Trusted Workload

↓

Document Findings

Initial Investigation Questions

Before interacting with the Pod, determine:

Which cluster is affected?
Which namespace contains the Pod?
What is the Pod name?
Which workload owns the Pod?
Which node is hosting it?
Which container triggered the alert?
What image is running?
Which Service Account is assigned?
Is the Pod still active?
Is the application business-critical?
Has the Pod restarted?
Is immediate isolation required?
Is node compromise suspected?

Identify the Suspicious Pod

List all Pods:

kubectl get pods -A -o wide

List Pods in the affected namespace:

kubectl get pods \
  -n <namespace> \
  -o wide

Inspect the suspected Pod:

kubectl describe pod <pod-name> \
  -n <namespace>

Record:

Namespace
Pod name
Pod UID
Node
Pod IP
Creation time
Restart count
Container names
Container IDs
Images
Image IDs
Service Account
Volumes
Events
Security context

Preserve the Pod Manifest

Export the Pod object immediately:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o yaml \
  > pod.yaml

The manifest may contain important evidence such as:

Labels
Annotations
Container images
Commands
Arguments
Environment variables
Security contexts
Service Account
Volumes
Volume mounts
Node assignment
Owner references
Status information

Preserve JSON Metadata

kubectl get pod <pod-name> \
  -n <namespace> \
  -o json \
  > pod.json

JSON format is useful for:

Automated analysis
Field extraction
Evidence comparison
Timeline reconstruction

Record the Pod UID

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.metadata.uid}'

The Pod UID helps correlate:

Node log directories
Runtime records
Kubernetes Audit Logs
Container logs
Volume paths

Identify the Owning Workload

Check owner references:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.metadata.ownerReferences}'

The Pod may be owned by:

Deployment
ReplicaSet
StatefulSet
DaemonSet
Job
CronJob

Investigate the Owner Resource

Example for a Deployment:

kubectl get deployment <deployment-name> \
  -n <namespace> \
  -o yaml \
  > deployment.yaml

Review:

Desired image
Replica count
Security context
Service Account
Update strategy
Environment variables
Volumes
Recent changes

Identify All Containers

A Pod may contain:

Main application containers
Sidecars
Init containers
Ephemeral containers

List normal containers:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.spec.containers[*].name}'

List init containers:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.spec.initContainers[*].name}'

List ephemeral containers:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.spec.ephemeralContainers[*].name}'

Every container should be reviewed.

Preserve Current Container Logs

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  > current-container.log

For a single-container Pod:

kubectl logs <pod-name> \
  -n <namespace> \
  > current-container.log

Preserve Previous Container Logs

If the container restarted:

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --previous \
  > previous-container.log

Previous logs may contain evidence of:

Exploitation
Crashes
Malware execution
Failed authentication
Application errors
Secret exposure
Suspicious requests

Preserve Logs with Timestamps

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --timestamps \
  > timestamped-container.log

Timestamps are essential for correlation with:

Audit logs
CloudTrail
Falco
GuardDuty
VPC Flow Logs
Application load balancer logs

Review Pod Events

kubectl get events \
  -n <namespace> \
  --field-selector involvedObject.name=<pod-name> \
  --sort-by='.metadata.creationTimestamp'

Events may reveal:

Image pulls
Pod scheduling
Container restarts
Probe failures
Volume mount errors
Admission denials
Node pressure
Sandbox creation failures

Review Restart Activity

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\t"}{.lastState}{"\n"}{end}'

Repeated restarts may indicate:

Application failure
Exploit attempts
Malware instability
Resource exhaustion
Probe misconfiguration
Deliberate evidence destruction

Identify Container Image References

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'

Identify Runtime Image IDs

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.imageID}{"\n"}{end}'

Compare:

Declared Image

↓

Runtime Image ID

↓

Approved Registry Digest

↓

Expected Deployment Record

Image Investigation Questions

Ask:

Was the image pulled from an approved registry?
Was the image referenced by immutable digest?
Was the image scanned?
Was the image signed?
Was the signature verified?
Does the runtime digest match the approved digest?
Was the image recently changed?
Is the image deployed in other clusters?
Does the image contain vulnerable packages?
Was the image introduced through an approved pipeline?

Review Image Pull Policy

imagePullPolicy: Always

or:

imagePullPolicy: IfNotPresent

Image pull behaviour may affect whether a mutable tag resolved to unexpected content.

Production workloads should preferably use immutable digests.

Investigate Container Commands

Review:

command:

args:

Export command and arguments:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\nCommand: "}{.command}{"\nArgs: "}{.args}{"\n\n"}{end}'

Look for:

Shell wrappers
Encoded commands
Download-and-execute patterns
Suspicious startup scripts
Unexpected interpreters
Commands running from writable directories

Suspicious Command Patterns

Examples include:

curl <address> | sh

wget <address> -O /tmp/file

bash -c <encoded-command>

python -c <payload>

nc <address> <port>

chmod +x /tmp/file

These commands require investigation when not expected.

Runtime Process Investigation

Where approved, inspect processes:

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- ps auxww

If ps is unavailable, use an approved ephemeral debugging method.

Look for:

Shell processes
Download tools
Unknown binaries
Cryptominers
Suspicious interpreters
Unexpected child processes
Processes running from /tmp
Processes running as root

Process Tree Analysis

A process tree may reveal the compromise path.

Application Process

↓

Shell

↓

curl or wget

↓

Downloaded Binary

↓

External Connection

Where available:

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- ps -ef --forest

Do Not Install Tools Inside the Compromised Container

Installing packages may:

Modify evidence
Change file timestamps
Add network activity
Overwrite artefacts
Trigger package-manager logs
Contaminate the investigation

Use approved forensic or ephemeral debugging containers when necessary.

Ephemeral Debug Containers

An ephemeral container may assist live investigation when the application image lacks tools.

Example:

kubectl debug \
  -n <namespace> \
  pod/<pod-name> \
  -it \
  --image=<approved-debug-image> \
  --target=<container-name>

Security considerations include:

Who is authorised to create it?
Which debug image is approved?
Will it alter the Pod state?
Is the activity audited?
Could it expose sensitive process information?
Is evidence preservation more important than live inspection?

Document every use.

Inspect Network Connections

Where tools are available:

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- ss -plant

or:

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- netstat -antup

Look for:

Unknown external addresses
Reverse-shell connections
Unexpected listening ports
Mining pools
Internal scanning
Database access outside normal patterns
Connections to metadata services

Inspect DNS Activity

Review:

CoreDNS logs
Route 53 Resolver logs
Runtime alerts
Application logs
DNS security tools

Look for:

Newly observed domains
Long encoded subdomains
Repeated failed queries
Known malicious domains
Direct use of unauthorised DNS resolvers

Review Pod IP and Node

kubectl get pod <pod-name> \
  -n <namespace> \
  -o wide

Record:

Pod IP
Node IP
Node name
Namespace
Start time

These details support network-log correlation.

Review VPC Flow Logs

Use the Pod IP and event time to investigate:

External destinations
Connection direction
Ports
Accepted or rejected traffic
Data volume
Lateral movement
Unusual scanning behaviour

Review Network Policies

kubectl get networkpolicy \
  -n <namespace> \
  -o yaml

Determine:

Was the Pod isolated?
Was default-deny applied?
Which Pods could connect to it?
Which destinations could it access?
Could it reach DNS?
Could it reach the internet?
Were policies modified during the incident?

Review Security Groups for Pods

Where used, identify whether the Pod had a dedicated Security Group.

Review:

Allowed destinations
Database access
Cross-VPC access
Internet access
Recent rule changes
SecurityGroupPolicy selection

Review the Pod Security Context

Export relevant settings:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.spec.securityContext}'

Review each container:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{.securityContext}{"\n\n"}{end}'

High-Risk Pod Security Settings

Investigate:

privileged: true

allowPrivilegeEscalation: true

runAsUser: 0

hostNetwork: true

hostPID: true

hostIPC: true

Also review:

Added Linux capabilities
Missing seccomp
Writable root filesystem
HostPath volumes
Device access

Linux Capabilities

List added capabilities from the manifest.

Dangerous capabilities may include:

SYS_ADMIN
SYS_PTRACE
NET_ADMIN
SYS_MODULE
DAC_READ_SEARCH
SYS_RAWIO
BPF
PERFMON

Determine whether each capability had an approved business requirement.

Review Seccomp Configuration

Expected secure configuration:

seccompProfile:
  type: RuntimeDefault

Missing or unconfined seccomp increases the available system-call surface.

Review Root Execution

Determine whether the container runs as root.

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- id

Root inside a container does not automatically mean host root, but it increases the impact of exploitation.

Review the Root Filesystem

Expected secure setting:

readOnlyRootFilesystem: true

A writable filesystem may allow attackers to:

Download tools
Modify application files
Install persistence
Replace binaries
Stage exfiltration data

Inspect Suspicious Files

Common locations include:

/tmp

/var/tmp

/dev/shm

/app

/home

/root

Look for:

Recently created files
Hidden files
Executables
Scripts
Archives
Downloaded tools
Web shells
Encoded payloads

File Hashing

For suspicious artefacts:

sha256sum <file>

Record:

Evidence ID:

Pod:

Container:

Original Path:

SHA-256:

Collection Time:

Collector:

Evidence Location:

Review Environment Variables

Export only through approved procedures.

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- env

Environment variables may contain:

Database credentials
API keys
Service endpoints
Cloud settings
Feature flags
Tokens

Do not copy secret values into ordinary investigation reports.

Review Environment References in the Manifest

Check:

env:

envFrom:

Determine whether values come from:

Secret
ConfigMap
Plaintext manifest
Downward API
Static values

Mounted Secret Investigation

Review volume definitions:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.spec.volumes}'

Review volume mounts:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{.volumeMounts}{"\n\n"}{end}'

Secret Exposure Questions

Ask:

Which Secrets were mounted?
Were they mounted as files or environment variables?
Could the compromised process read them?
Were they application-specific?
Were they shared with other workloads?
Have they been rotated?
Was access logged?
Could the attacker use them outside the cluster?

Service Account Token Location

A projected Service Account token may be available under a path similar to:

/var/run/secrets/kubernetes.io/serviceaccount/

Review:

Whether token mounting was required
Whether automountServiceAccountToken was disabled
Token audience
Token permissions
Token use in audit logs

Do not display token contents unnecessarily.

Investigate the Service Account

kubectl get serviceaccount <service-account-name> \
  -n <namespace> \
  -o yaml

Review:

Name
Namespace
Labels
Annotations
Image pull secrets
Workload identity association
Token-mounting behaviour

Review Kubernetes RBAC

Check permissions available to the Service Account:

kubectl auth can-i --list \
  --as=system:serviceaccount:<namespace>:<service-account-name>

Test sensitive permissions:

kubectl auth can-i get secrets \
  -n <namespace> \
  --as=system:serviceaccount:<namespace>:<service-account-name>

kubectl auth can-i create pods \
  -n <namespace> \
  --as=system:serviceaccount:<namespace>:<service-account-name>

kubectl auth can-i create rolebindings \
  -n <namespace> \
  --as=system:serviceaccount:<namespace>:<service-account-name>

High-Risk Service Account Permissions

Investigate whether the Service Account can:

Read Secrets
List Secrets
Create Pods
Execute into Pods
Create Jobs
Create RoleBindings
Create ClusterRoleBindings
Impersonate identities
Create Service Account tokens
Modify admission webhooks
Access resources across namespaces

Workload AWS Identity

Determine whether the Pod uses:

EKS Pod Identity
IAM Roles for Service Accounts
Worker-node IAM role
Static AWS credentials

Investigate IRSA

Check Service Account annotations:

kubectl get serviceaccount <service-account-name> \
  -n <namespace> \
  -o yaml

Review associated IAM role permissions and trust policy.

Investigate EKS Pod Identity

Review the Pod Identity association through approved AWS tooling.

Determine:

IAM role
Namespace
Service Account
Allowed AWS actions
Allowed resources
CloudTrail activity
Whether the role was used unexpectedly

AWS Credential Exposure

If the Pod may have accessed AWS credentials, review:

CloudTrail
Role sessions
STS activity
Secrets Manager access
S3 activity
KMS activity
Database access
Source IPs
Event timestamps

Assume the credentials may need to be revoked or rotated.

Review ConfigMaps

kubectl get configmap \
  -n <namespace>

Export relevant ConfigMaps:

kubectl get configmap <configmap-name> \
  -n <namespace> \
  -o yaml \
  > configmap.yaml

Look for:

Suspicious commands
Modified startup scripts
Malicious URLs
Unapproved configuration
Plaintext secrets
Persistence mechanisms

Review Volumes

Pod volumes may include:

Secret
ConfigMap
EmptyDir
PersistentVolumeClaim
HostPath
CSI volume
Projected volume

Each volume should be reviewed.

EmptyDir Evidence

emptyDir data may disappear when the Pod is removed from the node.

It may contain:

Downloaded malware
Temporary credentials
Staged archives
Process output
Exfiltration data

Preserve relevant data before Pod deletion.

PersistentVolumeClaim Investigation

List claims:

kubectl get pvc \
  -n <namespace>

Describe the relevant claim:

kubectl describe pvc <pvc-name> \
  -n <namespace>

Determine:

StorageClass
Bound volume
Underlying storage
Access mode
Other Pods using it
Snapshot availability
Evidence-preservation requirements

HostPath Investigation

Review any HostPath volume carefully.

hostPath:
  path: /var/lib

HostPath may expose:

Node filesystem
Runtime sockets
Kubernetes files
Credentials
Logs
Devices

HostPath exposure may require escalation to node forensics.

Runtime Socket Investigation

Look for mounts such as:

/run/containerd/containerd.sock

/var/run/docker.sock

Runtime socket access may allow:

Starting new containers
Controlling other containers
Accessing host resources
Escaping the intended Pod boundary

Init Container Investigation

Init containers run before application containers and may:

Modify shared volumes
Download files
Retrieve secrets
Generate configuration
Change permissions

Review:

Image
Commands
Logs
Volume mounts
Security context
Network activity

Sidecar Investigation

Sidecars may have access to:

Application logs
Network traffic
Shared files
Secrets
Service mesh certificates
Monitoring data

Do not assume the main application container is the only affected component.

Ephemeral Container Investigation

An unexpected ephemeral container may indicate:

Legitimate troubleshooting
Unauthorised debugging
Credential access
Process inspection
Policy bypass

Review:

Who created it
When it was created
Which image was used
Which process namespace it targeted
Whether there was an approved incident or change ticket

Kubernetes Audit Log Analysis

Audit logs should be reviewed for:

Pod creation
Pod updates
Pod deletion
pods/exec
pods/attach
pods/portforward
Ephemeral container creation
Secret access
Service Account token creation
RoleBinding changes
Network Policy changes
Admission decisions

Pod Execution Investigation

pods/exec activity is especially important.

Determine:

User
Source IP
User agent
Pod
Container
Namespace
Time
Whether access was approved
What occurred immediately afterward

Port Forward Investigation

Port forwarding may bypass normal ingress paths.

Investigate:

Who initiated it
Which Pod and port
Source address
Duration
Whether it was approved
Whether sensitive services were accessed

Admission Controller Evidence

Review:

Pod Security Admission warnings
Kyverno PolicyReports
Gatekeeper audit findings
Admission webhook logs
Policy exceptions
Mutations applied to the Pod

Questions include:

Was the Pod compliant?
Was a security policy bypassed?
Was an exception active?
Was the policy engine unavailable?
Was the Pod mutated before storage?

Runtime Security Evidence

Review findings from:

Falco
GuardDuty Runtime Monitoring
Tetragon
Cilium Hubble
Commercial runtime tools
SIEM correlation rules

Runtime alerts may identify:

Shell execution
Sensitive file access
Malware
Privilege escalation
Container escape attempts
Unexpected networking
Security-agent tampering

Application Load Balancer Evidence

For externally exposed applications, review:

Request path
Client IP
User agent
Response code
Request volume
Timestamp
WAF findings
Suspicious payload patterns

This may reveal the initial exploitation request.

Timeline Reconstruction

Combine all evidence into one timeline.

10:02 — Malicious Request Reached Ingress

10:03 — Application Spawned Shell

10:04 — External Tool Downloaded

10:05 — Service Account Token Read

10:06 — Kubernetes API Queried

10:08 — Secret Retrieved

10:10 — External Connection Established

10:12 — Alert Generated

Timeline Evidence Sources

Use:

Application logs
Ingress logs
WAF logs
Kubernetes Audit Logs
Runtime alerts
Pod events
CloudTrail
VPC Flow Logs
DNS logs
File timestamps
Container logs

Determine the Initial Access Vector

Potential access vectors include:

Application vulnerability
Stolen developer credentials
Compromised CI/CD pipeline
Malicious container image
Exposed management endpoint
Unauthorised kubectl exec
Compromised Service Account
Vulnerable sidecar
Insecure admission exception

Determine the Blast Radius

Investigate:

Compromised Container

↓

Other Containers in the Pod

↓

Mounted Volumes

↓

Service Account Permissions

↓

Workload IAM Role

↓

Other Pods in the Namespace

↓

Other Namespaces

↓

Worker Node

↓

AWS Services

Indicators of Node Compromise

Escalate to node forensics when you identify:

Runtime socket access
HostPath access to sensitive paths
Host process creation
Host namespace access
Kernel exploitation
Unexpected mount activity
Access to node credentials
Security-agent disablement
Unknown host-level processes

Pod Containment Options

Containment actions may include:

Apply emergency Network Policies
Remove the Pod from its Service
Remove external ingress
Scale the owning Deployment to zero
Suspend a Job or CronJob
Revoke workload IAM access
Disable the Service Account
Block malicious destinations
Quarantine the image digest
Cordon the hosting node

Isolate with Network Policy

A temporary quarantine policy may deny ingress and egress for selected Pods.

Conceptual example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine-suspicious-pod
  namespace: payments

spec:
  podSelector:
    matchLabels:
      incident-status: quarantined

  policyTypes:
    - Ingress
    - Egress

Apply labels and policies only through approved incident procedures.

Remove the Pod from Service Traffic

A Pod can be removed from normal traffic by changing:

Service selection
Readiness
Workload replicas
Ingress routing
Load balancer registration

Avoid deleting the Pod until required evidence has been collected.

Scale the Workload to Zero

After evidence preservation:

kubectl scale deployment <deployment-name> \
  -n <namespace> \
  --replicas=0

This action affects application availability and should be authorised.

Credential Revocation

Rotate or revoke credentials accessible from the compromised Pod.

Examples include:

Kubernetes Service Account tokens
EKS Pod Identity role access
IRSA role access
Secrets Manager credentials
Database passwords
API keys
TLS private keys
Third-party tokens

Preserve the Malicious Image

Do not immediately delete a suspicious image.

Instead:

Record the digest.
Restrict further deployment.
Preserve an approved copy for analysis.
Review the registry audit trail.
Scan the image.
Compare it with known-good versions.
Identify every environment using it.

Eradication

Eradication may include:

Patching the vulnerable application
Rebuilding the image
Removing malicious code
Removing unauthorised configuration
Revoking compromised credentials
Removing excessive RBAC
Removing privileged settings
Updating Network Policies
Fixing the CI/CD pipeline
Removing unsafe policy exceptions

Recovery

Recovery should use:

Trusted source code
Approved dependencies
A clean build pipeline
Scanned and signed images
Immutable image digests
Secure manifests
Least-privilege identities
Validated security policies

Compromised Workload

↓

Root Cause Fixed

↓

Trusted Image Rebuilt

↓

Security Scan

↓

Image Signed

↓

Admission Validation

↓

Controlled Redeployment

Recovery Validation

Verify:

Correct image digest
Security context
Service Account
Workload IAM permissions
Secrets access
Network Policies
Resource limits
Runtime monitoring
Application health
No suspicious outbound traffic
No repeated indicators

Evidence Collection Checklist

Pod Metadata

Container Evidence

Identity Evidence

Storage Evidence

Security Evidence

Chain of Custody

Every evidence item should include:

Evidence ID:

Incident ID:

Cluster:

Namespace:

Pod:

Container:

Source:

Collector:

Collection Time:

Collection Method:

SHA-256 Hash:

Storage Location:

Access Restrictions:

Evidence Storage

Store evidence in a protected repository with:

Encryption
Restricted access
Versioning
Retention controls
Audit logging
Integrity verification
Cross-account separation where required
Legal hold where applicable

Investigation Documentation

Maintain a responder log.

Time	Responder	Action	Tool or Command	Result
10:20 UTC	Analyst A	Exported Pod YAML	`kubectl get pod`	Successful
10:24 UTC	Analyst A	Collected container logs	`kubectl logs`	Evidence saved
10:31 UTC	Analyst B	Reviewed Pod IAM role	AWS CLI	Broad access found
10:40 UTC	Incident Commander	Approved isolation	Incident workflow	Pod quarantined

Common Pod Investigation Mistakes

Deleting the Pod Immediately

Risk: Logs, processes and temporary files are lost.

Response: Preserve evidence and isolate before deletion where practical.

Reviewing Only the Main Container

Risk: Malicious activity in sidecars, init containers or ephemeral containers is missed.

Response: Investigate every container.

Ignoring Service Account Permissions

Risk: Kubernetes privilege escalation remains undiscovered.

Response: Review RBAC and token usage.

Ignoring Workload IAM Activity

Risk: AWS service compromise is missed.

Response: Review CloudTrail for the workload role.

Installing Tools Inside the Container

Risk: Evidence is contaminated.

Response: Use approved external or ephemeral tooling.

Exposing Secret Values in Reports

Risk: Investigation documentation creates another security incident.

Response: Record metadata and rotate credentials without reproducing secret values.

Ignoring Previous Logs

Risk: Evidence from a restarted container is missed.

Response: Collect --previous logs immediately.

Focusing Only on Runtime Alerts

Risk: Initial access and identity activity remain unclear.

Response: Correlate runtime, audit, application and AWS logs.

Reusing the Compromised Image

Risk: Malicious or vulnerable code is redeployed.

Response: Rebuild and verify a trusted image.

Failing to Check the Node

Risk: Container escape or host compromise is overlooked.

Response: Escalate when host-level indicators are present.

Enterprise Pod Investigation Workflow

Security Alert

↓

Identify Pod and Container

↓

Export Pod and Workload Metadata

↓

Collect Current and Previous Logs

↓

Capture Runtime Processes and Connections

↓

Review Security Context

↓

Review Service Account and AWS Identity

↓

Review Secrets and Volumes

↓

Analyse Audit and Network Evidence

↓

Determine Blast Radius

↓

Contain Pod

↓

Rotate Credentials

↓

Rebuild from Trusted Image

↓

Validate Recovery

↓

Document Lessons Learned

Enterprise Best Practices

As a Cloud Security Engineer:

Maintain Pod-investigation runbooks before incidents occur.
Enable Kubernetes Audit Logs and runtime monitoring.
Collect Pod metadata immediately.
Preserve current and previous container logs.
Investigate every container in the Pod.
Compare runtime image IDs with approved digests.
Review process trees and network connections.
Avoid installing tools in the compromised container.
Review security contexts and Linux capabilities.
Investigate Service Account and workload IAM permissions.
Assume mounted credentials may have been exposed.
Review Secrets, ConfigMaps and all volume types.
Preserve EmptyDir data before deleting the Pod.
Investigate HostPath and runtime socket access immediately.
Correlate Pod evidence with CloudTrail and VPC Flow Logs.
Use network isolation before destructive containment where appropriate.
Preserve suspicious images for analysis.
Rebuild workloads from trusted, signed images.
Rotate all potentially exposed credentials.
Escalate to node forensics when host compromise is suspected.
Record every action and maintain chain of custody.
Update preventive and detective controls after the investigation.

Real-World Scenario

A financial services company operates a payment API on Amazon EKS.

Falco generates a high-severity alert indicating that a shell was started inside a production Pod.

At the same time:

The Pod begins communicating with an unknown external IP.
Kubernetes Audit Logs show Secret access using the Pod’s Service Account.
CloudTrail records calls to AWS Secrets Manager from the workload IAM role.
The application container shows unusually high CPU usage.

The incident-response team begins a Pod investigation.

They:

Identify the Pod, container, namespace and worker node.
Export the Pod YAML, JSON and owner Deployment.
Preserve current and previous container logs.
Record the runtime image digest.
Capture the running process tree.
Identify a shell that downloaded a binary into /tmp.
Calculate the binary’s SHA-256 hash.
Review active network connections and identify communication with a malicious host.
Review the Pod security context and confirm it runs as root with a writable filesystem.
Review the Service Account and discover permission to read multiple namespace Secrets.
Review the workload IAM role and identify broad Secrets Manager access.
Preserve relevant Kubernetes Audit Logs, CloudTrail events and VPC Flow Logs.
Apply an emergency Network Policy to isolate the Pod.
Remove the Pod from Service traffic.
Revoke the workload IAM role permissions.
Rotate the affected application and database credentials.
Preserve the suspicious image and binary for malware analysis.
Rebuild the application image with patched dependencies.
Enforce non-root execution and a read-only root filesystem.
Restrict the Service Account and workload IAM role.
Redeploy the application using a signed, digest-pinned image.
Update Falco rules and admission policies.

The investigation determines that an application vulnerability allowed remote code execution, but the response prevented broader cluster compromise.

Key Takeaways

Pods are ephemeral, so evidence must be collected quickly.
Pod investigation includes metadata, containers, images, identities, storage and networking.
Current and previous logs are both important.
Every container, including init, sidecar and ephemeral containers, must be reviewed.
Service Account and workload IAM permissions determine the potential blast radius.
Mounted Secrets and volumes may contain important evidence and exposed credentials.
HostPath and runtime socket access may indicate node compromise.
Kubernetes Audit Logs, CloudTrail and network logs provide complementary evidence.
Containment should preserve evidence where business risk permits.
Compromised workloads should be rebuilt from trusted images.
Potentially exposed credentials must be rotated.
Investigation findings should improve RBAC, admission policies, workload hardening and runtime detection.

Knowledge Check

1. Why should a suspicious Pod not be deleted immediately?

Answer: Deleting the Pod may destroy volatile evidence such as running processes, active connections, temporary files, current logs and EmptyDir data.

2. Why should both current and previous container logs be collected?

Answer: Previous logs may contain evidence from a container that crashed or restarted, while current logs show activity from the presently running container.

3. Why is the Pod’s Service Account important during an investigation?

Answer: The Service Account determines which Kubernetes API actions the Pod can perform and may reveal whether the attacker could access Secrets, create workloads or escalate privileges.

4. When should Pod investigation escalate to node forensics?

Answer: Escalation is required when there are indicators of host compromise, such as runtime socket access, HostPath access to sensitive directories, host process creation, kernel exploitation or node credential theft.

5. Why should a compromised workload be rebuilt rather than repaired?

Answer: Rebuilding from trusted source code and a verified image provides greater assurance that malware, unauthorised changes and persistence mechanisms have been removed.

What’s Next?

In the next lesson, we will explore Kubernetes Audit Log Analysis, including how to interpret audit events, identify suspicious API operations, investigate identity activity, reconstruct attack timelines and build enterprise detection queries.

➡️ Next Lesson: Lesson 05 — Audit Log Analysis