Lab 02 — Runtime Threat Investigation

Mission Information

Item	Details
Lab ID	K8S-IR-LAB-02
Difficulty	Advanced
Estimated Time	2 Hours
Platform	Kubernetes
Environment	Kind / Minikube / Amazon EKS
Type	Runtime Security Investigation
Focus Area	Threat Detection & Incident Response
Tools	kubectl, Falco, Trivy, jq, Linux Commands
Prerequisites	Kubernetes Fundamentals, Container Security, Logging & Monitoring

---

Mission Brief

CloudNova Technologies’ Security Operations Centre (SOC) has detected several runtime alerts from the Kubernetes production environment.

Security monitoring systems report:

• Shell execution inside containers
• Suspicious outbound connections
• Unauthorized file modifications
• Privilege escalation attempts
• Unexpected Kubernetes API activity
• High CPU usage from multiple Pods

The SOC suspects an attacker has gained access to one or more running containers.

As the Cloud Security Engineer, your objective is to investigate the runtime behavior, determine the scope of the compromise, identify indicators of compromise (IOCs), and recommend containment actions while preserving forensic evidence.

---

Learning Objectives

By completing this lab you will learn how to:

• Investigate runtime security alerts
• Analyze Falco detections
• Review Kubernetes Events
• Inspect container processes
• Monitor network activity
• Detect privilege escalation
• Investigate suspicious file changes
• Review Kubernetes Audit Logs
• Identify attacker techniques
• Recommend containment actions

---

Enterprise Scenario

SOC Alert

      │

      ▼

Falco Runtime Alert

      │

      ▼

Container Investigation

      │

      ▼

Process Analysis

      │

      ▼

Network Analysis

      │

      ▼

Kubernetes Events

      │

      ▼

Threat Assessment

      │

      ▼

Incident Response

---

Lab Objectives

Investigate:

• Runtime alerts
• Running processes
• Active network connections
• File system activity
• Kubernetes Events
• Service Account activity
• Kubernetes Audit Logs
• Indicators of compromise
• Threat severity

---

Lab Environment

Resources available:

• Kubernetes Cluster
• Falco
• kubectl
• Worker Nodes
• CloudWatch (Amazon EKS)
• Kubernetes Audit Logs
• Sample vulnerable application

---

Task 1 — Review Runtime Alerts

Display Falco alerts:

kubectl logs \
-n falco \
deployment/falco

Look for:

• Terminal shell detected
• Sensitive file access
• Privileged container
• Reverse shell
• Unexpected process
• Host filesystem access

Questions:

• Which alert triggered first?
• Which namespace is affected?

---

Task 2 — Identify Affected Pods

List running Pods:

kubectl get pods -A

Show detailed information:

kubectl get pods -A -o wide

Review:

• Namespace
• Node
• Restart count
• IP address
• Status

---

Task 3 — Review Container Processes

Inspect running processes:

kubectl exec \
-it suspicious-pod \
-n production \
-- ps aux

Look for:

• Bash
• sh
• nc
• curl
• wget
• Python
• Perl
• Crypto miners
• Unknown executables

Questions:

• Are there unexpected processes?
• Do any indicate attacker activity?

---

Task 4 — Investigate Active Network Connections

View connections:

kubectl exec \
-it suspicious-pod \
-n production \
-- ss -tunap

Or

kubectl exec \
-it suspicious-pod \
-n production \
-- netstat -tunap

Review:

• External IPs
• Listening ports
• Established sessions
• Suspicious destinations

---

Task 5 — Review Kubernetes Events

Display Events:

kubectl get events \
-A \
--sort-by=.metadata.creationTimestamp

Look for:

• Pod restarts
• Failed authentication
• Secret access
• Container crashes
• Privilege escalation

---

Task 6 — Review Security Context

Inspect Pod configuration:

kubectl get pod suspicious-pod \
-o yaml \
-n production

Review:

• privileged
• allowPrivilegeEscalation
• hostPID
• hostIPC
• hostNetwork
• capabilities
• runAsUser
• readOnlyRootFilesystem

Questions:

• Is privilege escalation possible?
• Does the Pod violate security best practices?

---

Task 7 — Review Service Account Activity

Identify Service Account:

kubectl get pod suspicious-pod \
-o=jsonpath='{.spec.serviceAccountName}'

Review permissions:

kubectl auth can-i \
--list \
--as=system:serviceaccount:production:default

Determine:

• Can it access Secrets?
• Can it create Pods?
• Can it list resources?

---

Task 8 — Investigate File System Changes

List recently modified files:

kubectl exec \
-it suspicious-pod \
-n production \
-- find / -type f -mtime -1

Look for:

• Scripts
• Downloaded binaries
• SSH keys
• Cron jobs
• Hidden files

---

Task 9 — Review Kubernetes Audit Logs

If Audit Logging is enabled:

Review:

• exec requests
• Secret access
• RBAC changes
• Pod creation
• RoleBindings
• ClusterRoleBindings

Questions:

• Who initiated the activity?
• Which Service Account was used?

---

Task 10 — Review Container Logs

Display logs:

kubectl logs suspicious-pod \
-n production

Previous logs:

kubectl logs suspicious-pod \
--previous \
-n production

Review:

• Reverse shell commands
• Curl requests
• Authentication failures
• Suspicious scripts
• Malware downloads

---

Task 11 — Identify Indicators of Compromise

Record findings including:

• Suspicious IP addresses
• Unknown domains
• Reverse shell activity
• Unauthorized binaries
• Privileged containers
• Excessive API requests
• Secret access
• Shell execution
• Host filesystem access

---

Task 12 — Classify the Threat

Assign severity:

Severity	Description
Critical	Active compromise with privilege escalation
High	Confirmed malicious runtime activity
Medium	Suspicious behavior requiring investigation
Low	Benign anomaly

Justify your decision using the collected evidence.

---

Task 13 — Recommend Containment

Recommend actions:

• Isolate Pod
• Apply NetworkPolicy
• Rotate Secrets
• Disable Service Account
• Scale Deployment to zero
• Capture forensic evidence
• Notify Incident Response team
• Preserve logs

Avoid deleting evidence before collection.

---

Validation Checklist

Verify that you successfully:

• Reviewed runtime alerts
• Identified affected Pods
• Inspected running processes
• Investigated network activity
• Reviewed Kubernetes Events
• Evaluated Security Context
• Reviewed Service Account permissions
• Investigated file changes
• Reviewed Audit Logs
• Identified IOCs
• Assigned threat severity
• Recommended containment actions

---

Expected Findings

You should identify:

• Active runtime alerts
• Suspicious shell execution
• External network communication
• Privilege escalation attempts
• High-risk Security Context configuration
• Unauthorized file modifications
• Service Account misuse
• Indicators of compromise

---

Best Practices

• Enable runtime threat detection with Falco or similar tools.
• Correlate runtime alerts with Kubernetes Audit Logs.
• Investigate before taking destructive actions.
• Preserve logs and evidence.
• Monitor network activity continuously.
• Enforce least privilege for Pods and Service Accounts.
• Rotate compromised credentials immediately.
• Integrate runtime monitoring with SIEM solutions.

---

Challenge Exercise

Extend the investigation by:

• Reviewing GuardDuty findings (Amazon EKS)
• Correlating CloudTrail events
• Scanning the running container with Trivy
• Comparing container image hashes
• Mapping activity to the MITRE ATT&CK framework
• Producing an executive incident report

---

Key Takeaways

After completing this lab, you can:

• Investigate runtime threats in Kubernetes
• Analyze Falco security alerts
• Monitor container behavior
• Review runtime telemetry
• Detect privilege escalation
• Investigate suspicious network activity
• Identify indicators of compromise
• Classify runtime incidents
• Recommend effective containment strategies
• Perform enterprise Kubernetes runtime investigations

---

Next Steps

Next Lab: Lab 03 — Container Escape Investigation

In the next lab, you will investigate a suspected container escape by analyzing privileged containers, Linux namespaces, host access, kernel interactions, mounted volumes, and evidence of compromise to determine whether an attacker successfully escaped from a Kubernetes container to the underlying worker node.