Lesson 07 — Kubernetes Malware Analysis
Learning Objectives
Section titled “Learning Objectives”By the end of this lesson, you will be able to:
- Explain the purpose of malware analysis in Kubernetes investigations
- Identify common malware types affecting containers and worker nodes
- Safely collect suspicious files from compromised workloads
- Preserve malware evidence and maintain chain of custody
- Calculate and validate cryptographic file hashes
- Perform basic static malware analysis
- Perform controlled dynamic malware analysis
- Inspect malicious container images and image layers
- Analyse scripts, binaries and encoded payloads
- Extract Indicators of Compromise
- Correlate malware findings with Kubernetes and AWS telemetry
- Determine whether malware affected a Pod, node or wider environment
- Build an enterprise malware-analysis workflow for Amazon EKS
Why This Matters
Section titled “Why This Matters”Kubernetes workloads can be compromised through:
- Vulnerable applications
- Malicious dependencies
- Compromised container images
- Exposed management interfaces
- Stolen credentials
- Container escape vulnerabilities
- Supply-chain attacks
- Unauthorised administrative access
After gaining access, attackers may deploy malware to:
- Establish persistence
- Mine cryptocurrency
- Steal credentials
- Create reverse shells
- Exfiltrate data
- Disable security tooling
- Move laterally
- Compromise worker nodes
- Communicate with command-and-control infrastructure
Application Exploit
↓
Malware Downloaded
↓
Malware Executed
↓
Credentials Stolen
↓
Lateral Movement
↓
Data ExfiltrationMalware analysis helps investigators understand:
- What the suspicious file does
- How it entered the environment
- Which systems it contacted
- Which credentials it attempted to access
- Whether it created persistence
- Whether it escaped the container
- Which other systems may be affected
What is Malware Analysis?
Section titled “What is Malware Analysis?”Malware analysis is the structured examination of suspicious files, scripts, binaries, container images and runtime behaviour to determine their purpose, capabilities and impact.
The investigation attempts to answer:
What Is the File?
↓
Is It Malicious?
↓
How Does It Execute?
↓
What Does It Change?
↓
Which Systems Does It Contact?
↓
What Credentials or Data Does It Access?
↓
How Can It Be Detected and Removed?Kubernetes Malware Analysis Scope
Section titled “Kubernetes Malware Analysis Scope”Kubernetes Malware Analysis
├── Suspicious Files├── Shell Scripts├── Compiled Binaries├── Container Images├── Image Layers├── Running Processes├── Network Connections├── DNS Activity├── Persistence Mechanisms├── Service Account Tokens├── AWS Credentials├── Runtime Alerts└── Worker-Node ArtefactsCommon Malware in Kubernetes
Section titled “Common Malware in Kubernetes”Malware affecting Kubernetes environments may include:
| Malware Type | Purpose |
|---|---|
| Cryptominer | Abuses CPU or GPU resources |
| Reverse shell | Provides remote command access |
| Downloader | Retrieves additional malicious payloads |
| Credential stealer | Collects tokens, passwords or cloud credentials |
| Backdoor | Maintains unauthorised access |
| Rootkit | Hides processes or files |
| Botnet agent | Connects the workload to a botnet |
| Ransomware | Encrypts or destroys data |
| Web shell | Provides command execution through a web application |
| Network scanner | Searches for internal systems and services |
| Data exfiltration tool | Transfers sensitive information externally |
| Container escape exploit | Attempts to reach the worker node |
Common Kubernetes Malware Locations
Section titled “Common Kubernetes Malware Locations”Attackers frequently place files in writable locations such as:
/tmp
/var/tmp
/dev/shm
/app
/home
/root
var/runThey may also use:
- Mounted persistent volumes
- Writable application directories
- ConfigMaps
- Init containers
- Sidecars
- Container image layers
- HostPath mounts
- Runtime socket mounts
Malware Investigation Lifecycle
Section titled “Malware Investigation Lifecycle”Suspicious Activity Detected
↓
Identify Malware Artefact
↓
Preserve Evidence
↓
Calculate Hashes
↓
Perform Static Analysis
↓
Perform Dynamic Analysis
↓
Extract Indicators
↓
Determine Scope
↓
Contain and Eradicate
↓
Improve DetectionSafety Principles
Section titled “Safety Principles”Malware analysis must be performed carefully.
Never analyse suspicious malware directly on:
- A production workstation
- A production Kubernetes cluster
- A shared administrative system
- A corporate laptop
- An internet-connected personal device
- The original evidence volume
Use:
- Isolated forensic environments
- Dedicated analysis virtual machines
- Restricted network access
- Disposable sandboxes
- Read-only evidence copies
- Approved malware-analysis tools
- Documented handling procedures
Malware Sample
↓
Protected Evidence Repository
↓
Isolated Analysis Environment
↓
Controlled InvestigationDo Not Execute Malware in Production
Section titled “Do Not Execute Malware in Production”Do not execute suspicious binaries or scripts inside the affected Pod merely to see what they do.
This may:
- Trigger destructive activity
- Destroy evidence
- Exfiltrate data
- Spread to other systems
- Steal additional credentials
- Compromise the worker node
- Disable security tooling
Malware Analysis Types
Section titled “Malware Analysis Types”Malware analysis normally includes:
Static Analysis
Section titled “Static Analysis”Examines the file without executing it.
Dynamic Analysis
Section titled “Dynamic Analysis”Executes the sample in a controlled environment and observes behaviour.
Behavioural Analysis
Section titled “Behavioural Analysis”Studies process, file, network and system activity generated by the malware.
Code Analysis
Section titled “Code Analysis”Examines scripts, bytecode, disassembly or decompiled code.
Static Versus Dynamic Analysis
Section titled “Static Versus Dynamic Analysis”| Static Analysis | Dynamic Analysis |
|---|---|
| Does not execute the file | Executes the file in isolation |
| Lower operational risk | Higher operational risk |
| Extracts metadata and strings | Observes real behaviour |
| Identifies indicators quickly | Reveals runtime actions |
| May be defeated by obfuscation | May trigger anti-analysis behaviour |
| Suitable for initial triage | Suitable for deeper investigation |
Initial Malware Triage
Section titled “Initial Malware Triage”Before collecting the sample, record:
- Cluster
- Namespace
- Pod
- Container
- Worker node
- Original file path
- Process using the file
- File owner
- File permissions
- Creation time
- Modification time
- Detection source
- Related network connections
- Related runtime alert
Identify the Suspicious File
Section titled “Identify the Suspicious File”A runtime alert may identify:
File:
/tmp/update
Process:
/tmp/update
Parent:
/bin/sh
Pod:
payment-api
Namespace:
paymentsPreserve this original context.
Collect Pod Metadata
Section titled “Collect Pod Metadata”Export the Pod manifest:
kubectl get pod <pod-name> \ -n <namespace> \ -o yaml \ > pod.yamlExport JSON:
kubectl get pod <pod-name> \ -n <namespace> \ -o json \ > pod.jsonRecord:
- Pod UID
- Node
- Image
- Image ID
- Service Account
- Volumes
- Security context
- Commands
- Environment references
Preserve Container Logs
Section titled “Preserve Container Logs”kubectl logs <pod-name> \ -n <namespace> \ -c <container-name> \ --timestamps \ > container.logPrevious logs:
kubectl logs <pod-name> \ -n <namespace> \ -c <container-name> \ --previous \ --timestamps \ > container-previous.logCollect the Suspicious File
Section titled “Collect the Suspicious File”Use an approved forensic procedure.
Conceptual example:
kubectl cp \ <namespace>/<pod-name>:<suspicious-file-path> \ ./evidence/<evidence-file-name> \ -c <container-name>Important considerations:
- Confirm authorisation before collection.
- Document the exact command.
- Do not rename the original evidence without recording the original name.
- Store the file in a protected directory.
- Calculate a cryptographic hash immediately.
- Restrict access to the evidence.
Limitations of kubectl cp
Section titled “Limitations of kubectl cp”kubectl cp may depend on tools available inside the container and may not preserve every filesystem attribute.
For formal forensic acquisition, an organisation may instead use:
- Node-level acquisition
- Container-runtime tooling
- Volume snapshots
- Forensic agents
- Approved evidence-collection containers
- Read-only mounted evidence copies
The selected method should be documented.
Collect File Metadata
Section titled “Collect File Metadata”Where available:
stat <suspicious-file>Record:
- File type
- Size
- Owner
- Group
- Permissions
- Inode
- Access time
- Modification time
- Change time
Do not rely solely on timestamps because attackers may modify them.
Calculate Cryptographic Hashes
Section titled “Calculate Cryptographic Hashes”Calculate a SHA-256 hash:
sha256sum <evidence-file>Optionally record additional hashes according to organisational standards:
sha1sum <evidence-file>md5sum <evidence-file>SHA-256 should normally be the primary integrity hash.
Evidence Hash Record
Section titled “Evidence Hash Record”Evidence ID:
Original File Name:
Original Path:
Cluster:
Namespace:
Pod:
Container:
File Size:
SHA-256:
Collection Time:
Collector:
Collection Method:
Evidence Location:Why Hashing Matters
Section titled “Why Hashing Matters”Cryptographic hashes help:
- Identify duplicate samples
- Prove evidence integrity
- Compare files across clusters
- Search threat-intelligence systems
- Validate that a file has not changed
- Link related incidents
Malware File
↓
SHA-256 Hash
↓
Threat Intelligence Search
↓
Related Campaign or DetectionPreserve the Original Sample
Section titled “Preserve the Original Sample”Use separate copies for:
- Original evidence
- Static analysis
- Dynamic analysis
- Malware sharing under approved procedures
The original evidence should remain unchanged.
Original Evidence
↓
Read-Only Preservation
↓
Forensic Working Copy
↓
AnalysisChain of Custody
Section titled “Chain of Custody”Every transfer or access should be recorded.
Evidence ID:
Transferred From:
Transferred To:
Date and Time:
Purpose:
Hash Before Transfer:
Hash After Transfer:
Authorised By:Static Analysis Overview
Section titled “Static Analysis Overview”Static analysis examines the file without running it.
It may identify:
- File type
- Architecture
- Compiler information
- Embedded strings
- URLs
- IP addresses
- Domain names
- Commands
- Encryption keys
- Configuration
- Imported libraries
- Suspicious functions
- Packers or obfuscation
Identify File Type
Section titled “Identify File Type”Use:
file <sample>Example output may show:
ELF 64-bit executable
POSIX shell script
Python script
gzip compressed archive
ASCII textReview File Size
Section titled “Review File Size”ls -lh <sample>Very small binaries may be:
- Downloaders
- Loaders
- Shellcode wrappers
- Compressed payloads
Large files may contain:
- Embedded libraries
- Cryptomining software
- Statically linked tools
- Packed content
- Exfiltrated data
Review Printable Strings
Section titled “Review Printable Strings”strings <sample>Search for indicators:
strings <sample> | grep -Ei \'http|https|token|password|secret|wallet|pool|curl|wget|bash|sh|aws'Strings may reveal:
- Command-and-control domains
- URLs
- File paths
- Shell commands
- Wallet addresses
- Error messages
- User-agent strings
- API endpoints
- Credentials
Limitations of strings
Section titled “Limitations of strings”Malware may:
- Encrypt strings
- Compress configuration
- Generate values dynamically
- Use obfuscation
- Store data as encoded text
- Download configuration at runtime
An absence of readable strings does not mean the file is safe.
Identify Script Content
Section titled “Identify Script Content”For scripts, review a copy using:
sed -n '1,200p' <sample>or an approved text editor in the isolated analysis environment.
Look for:
- Downloads
- Encoded payloads
- Credential access
- Network connections
- Persistence commands
- File deletion
- Process killing
- Security-tool disablement
- Kubernetes API calls
- AWS metadata requests
Shell Script Indicators
Section titled “Shell Script Indicators”Suspicious commands may include:
curl
wget
base64 -d
chmod +x
nohup
crontab
pkill
killall
rm -rf
nc
socat
bash -iEncoded Payloads
Section titled “Encoded Payloads”Attackers may use:
- Base64
- Hex encoding
- URL encoding
- Gzip compression
- XOR
- Custom encoding
Example pattern:
echo <encoded-data> | base64 -d | shDo not decode and execute the output.
Decode only within an isolated analysis environment and save the result as evidence.
Review ELF Binaries
Section titled “Review ELF Binaries”Linux malware commonly uses the ELF format.
Static ELF analysis may include:
readelf -h <sample>readelf -S <sample>readelf -s <sample>objdump -x <sample>These may reveal:
- CPU architecture
- Entry point
- Sections
- Symbols
- Linking information
- Compiler artefacts
Architecture Identification
Section titled “Architecture Identification”Common architectures include:
- x86-64
- ARM64
- ARM
- MIPS
A malware sample compiled for several architectures may indicate a botnet or automated campaign targeting many systems.
Shared Library Analysis
Section titled “Shared Library Analysis”ldd <sample>Do not use ldd on untrusted files outside a controlled analysis environment because some malicious files may behave unexpectedly.
Safer static alternatives should be used according to the approved toolset.
Imported Function Analysis
Section titled “Imported Function Analysis”Suspicious capabilities may be inferred from imported functions related to:
- Network communication
- Process execution
- File manipulation
- Encryption
- Privilege changes
- Kernel operations
- Process injection
- Persistence
Packing and Obfuscation
Section titled “Packing and Obfuscation”Malware authors may pack binaries to make analysis difficult.
Indicators include:
- Very few readable strings
- High entropy
- Unusual executable sections
- Large compressed data sections
- Known packer signatures
- Runtime unpacking behaviour
Archive Analysis
Section titled “Archive Analysis”Suspicious archives may contain several components.
Examples:
.tar
.gz
.zip
.xzList contents without executing files:
tar -tf <archive>unzip -l <archive>Extract only in an isolated analysis directory.
Cryptominer Analysis
Section titled “Cryptominer Analysis”Cryptomining malware may contain:
- Mining-pool domains
- Wallet addresses
- CPU optimisation flags
- XMRig references
- Stratum protocol strings
- Configuration files
- Thread-count settings
- Process-killing routines
Reverse Shell Analysis
Section titled “Reverse Shell Analysis”A reverse-shell script or binary may contain:
- External IP address
- Port number
- Shell path
- Socket calls
/dev/tcpncsocat- Python socket code
- Perl socket code
Credential-Stealer Analysis
Section titled “Credential-Stealer Analysis”Credential stealers may search for:
/var/run/secrets/kubernetes.io/serviceaccount/
/root/.aws/
/home/*/.aws/
/etc/kubernetes/
/var/lib/kubelet/
/run/secrets/
/proc/*/environThey may also access:
169.254.169.254to attempt EC2 instance metadata credential theft.
Persistence Analysis
Section titled “Persistence Analysis”Inside a container, malware may attempt persistence through:
- Startup scripts
- Entrypoint modification
- Shared persistent volumes
- Cron jobs
- Kubernetes resource creation
- DaemonSets
- Init containers
- Sidecars
- Modified ConfigMaps
- Host filesystem modification
Because containers are ephemeral, attackers often seek persistence outside the original container.
Kubernetes Persistence Techniques
Section titled “Kubernetes Persistence Techniques”Investigate unexpected creation or modification of:
- Deployments
- DaemonSets
- CronJobs
- Jobs
- Mutating webhooks
- Validating webhooks
- Service Accounts
- RoleBindings
- ClusterRoleBindings
- Secrets
- ConfigMaps
- Admission-policy exceptions
Container Image Malware Analysis
Section titled “Container Image Malware Analysis”Malware may already exist inside the container image.
The analysis should determine whether it was:
- Present at build time
- Added through a malicious image layer
- Downloaded at runtime
- Injected through a mounted volume
- Added by an init container
- Introduced through a compromised registry
Record the Runtime Image
Section titled “Record the Runtime Image”kubectl get pod <pod-name> \ -n <namespace> \ -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.imageID}{"\n"}{end}'Record:
- Registry
- Repository
- Tag
- Digest
- Image ID
- Pull time
- Deployment source
Preserve the Suspicious Image
Section titled “Preserve the Suspicious Image”Do not delete the image before preservation.
Actions may include:
- Restricting further deployment
- Recording the digest
- Copying the image to an isolated evidence registry
- Exporting the image under approved procedures
- Preserving registry logs
- Preserving CI/CD evidence
- Preserving image signatures and attestations
Export a Container Image
Section titled “Export a Container Image”In an isolated analysis environment, approved tools may export an image for analysis.
Conceptual workflow:
Approved Registry
↓
Pull Exact Image Digest
↓
Export Image Archive
↓
Hash Archive
↓
Analyse LayersUse the exact digest rather than a mutable tag.
Image Layer Analysis
Section titled “Image Layer Analysis”Container images are composed of layers.
Base Image Layer
↓
Dependency Layer
↓
Application Layer
↓
Potential Malicious LayerInspect:
- Layer creation history
- Added files
- Removed files
- Modified permissions
- Downloaded binaries
- Embedded credentials
- Startup commands
Review Image History
Section titled “Review Image History”Using an approved container-analysis system:
docker history <image-reference>or equivalent tooling.
Review:
RUNcommandsCOPYoperations- Package installations
- Download commands
- Unexpected shell commands
- Large unexplained layers
Deleted Files in Image Layers
Section titled “Deleted Files in Image Layers”Deleting a secret or malware file in a later image layer may not remove it from earlier layers.
Layer 1
Adds Sensitive File
↓
Layer 2
Deletes Sensitive File
↓
File May Still Exist in Layer 1Inspect all layers.
Compare with Known-Good Image
Section titled “Compare with Known-Good Image”Compare the suspicious image against:
- Previous approved image
- Golden base image
- CI/CD build artefacts
- SBOM
- Registry metadata
- Image signature
- Provenance attestation
Image Signature Verification
Section titled “Image Signature Verification”Determine:
- Was the image signed?
- Which identity signed it?
- Did verification occur during admission?
- Does the signature cover the exact digest?
- Was the signing identity compromised?
- Was the image built by an approved pipeline?
SBOM Analysis
Section titled “SBOM Analysis”Review the Software Bill of Materials for:
- Unexpected packages
- Unapproved dependencies
- Vulnerable libraries
- Added command-line tools
- Mining libraries
- Networking utilities
- Shell interpreters
Static Image Scanning
Section titled “Static Image Scanning”Image scanners may identify:
- Known vulnerabilities
- Malware signatures
- Secrets
- Suspicious packages
- Policy violations
- Unapproved software
- Licence concerns
Scanning results are evidence, but they do not replace manual analysis.
Dynamic Analysis Overview
Section titled “Dynamic Analysis Overview”Dynamic analysis executes the malware in a controlled environment.
It observes:
- Processes
- Files created or modified
- Network connections
- DNS queries
- Registry or configuration changes
- Persistence
- Credential access
- Child processes
- System calls
Dynamic Analysis Environment
Section titled “Dynamic Analysis Environment”Use:
- Disposable virtual machines
- Isolated Kubernetes clusters
- Restricted sandbox accounts
- No production credentials
- No access to corporate networks
- Controlled DNS
- Simulated internet services
- Full packet capture
- Snapshot and reset capability
Never Use Real Credentials
Section titled “Never Use Real Credentials”The sandbox must not contain:
- Production Service Account tokens
- Real IAM roles
- Production Secrets
- Real API keys
- Customer data
- Corporate certificates
- Production network access
Dynamic Analysis Architecture
Section titled “Dynamic Analysis Architecture”Malware Sample
↓
Disposable Analysis Container or VM
↓
Process Monitoring
+
Filesystem Monitoring
+
Network Capture
+
System Call Monitoring
↓
Behaviour ReportNetwork Isolation Options
Section titled “Network Isolation Options”Dynamic malware analysis may use:
- No network access
- Simulated network services
- Controlled proxy
- Sinkhole DNS
- Restricted allowlist
- Packet capture gateway
Allowing unrestricted internet access may enable malware to:
- Attack external systems
- Exfiltrate data
- Download additional payloads
- Join a botnet
Observe Process Behaviour
Section titled “Observe Process Behaviour”Record:
- Initial process
- Child processes
- Command lines
- Process user
- Process tree
- Exit codes
- Runtime duration
- CPU and memory usage
Observe Filesystem Behaviour
Section titled “Observe Filesystem Behaviour”Monitor:
- Files created
- Files modified
- Files deleted
- Permission changes
- Executables dropped
- Configuration files
- Persistence files
- Staging directories
Observe Network Behaviour
Section titled “Observe Network Behaviour”Record:
- DNS queries
- Destination IPs
- Destination ports
- Protocols
- HTTP requests
- TLS connections
- User-agent strings
- Data volume
- Retry behaviour
Observe Credential Access
Section titled “Observe Credential Access”Monitor access to:
- Kubernetes Service Account tokens
- AWS credential locations
- Environment variables
- Secrets-mounted paths
- SSH keys
- Cloud metadata endpoints
- Application configuration
Observe System Calls
Section titled “Observe System Calls”Runtime sensors may show:
execveopenconnectmountchmodsetuidptracecloneunshare
System-call sequences can reveal behaviour even when process names are obfuscated.
Anti-Analysis Techniques
Section titled “Anti-Analysis Techniques”Malware may attempt to detect:
- Virtual machines
- Sandboxes
- Debuggers
- Low system uptime
- Missing network access
- Unusual process names
- Limited CPU or memory
- Analysis tools
It may then:
- Exit
- Sleep
- Behave differently
- Delete itself
- Trigger only after a delay
Delayed Execution
Section titled “Delayed Execution”Malware may wait:
Minutes
Hours
Specific Dates
Specific Environment ConditionsDynamic analysis periods should account for delayed behaviour.
Malware Configuration Extraction
Section titled “Malware Configuration Extraction”Malware configuration may contain:
- Command-and-control servers
- Campaign identifiers
- Wallet addresses
- Authentication tokens
- Target paths
- Sleep intervals
- Encryption keys
- Update URLs
- Persistence settings
Indicators of Compromise
Section titled “Indicators of Compromise”Indicators of Compromise may include:
File Indicators
Section titled “File Indicators”- SHA-256 hashes
- File names
- File paths
- File sizes
- Permissions
Network Indicators
Section titled “Network Indicators”- IP addresses
- Domains
- URLs
- Ports
- TLS certificates
- User agents
Process Indicators
Section titled “Process Indicators”- Process names
- Command lines
- Parent-child relationships
- Executable paths
Kubernetes Indicators
Section titled “Kubernetes Indicators”- Image digests
- Namespace names
- Pod labels
- Service Accounts
- DaemonSets
- CronJobs
- RBAC changes
AWS Indicators
Section titled “AWS Indicators”- IAM roles
- STS sessions
- API calls
- S3 buckets
- Secrets Manager access
- Security Group changes
Indicator Quality
Section titled “Indicator Quality”Indicators should include:
- Source
- Confidence
- First observed time
- Last observed time
- Context
- Related incident
- Expiry or review date
- False-positive considerations
Behavioural Indicators
Section titled “Behavioural Indicators”Behavioural indicators are often more durable than simple file hashes.
Example:
Web Server Process
↓
Starts Shell
↓
Downloads Binary to /tmp
↓
Adds Execute Permission
↓
Connects to External Mining PoolA new malware hash may still follow the same behaviour.
Malware Analysis Report
Section titled “Malware Analysis Report”A malware-analysis report should include:
Incident ID:
Evidence ID:
File Name:
Original Path:
SHA-256:
File Type:
Architecture:
Size:
Static Findings:
Dynamic Findings:
Network Indicators:
File Indicators:
Process Indicators:
Credential Access:
Persistence:
Container Escape Indicators:
Affected Systems:
Risk Rating:
Containment Actions:
Detection Recommendations:Severity Assessment
Section titled “Severity Assessment”Assess severity based on:
- Malware capabilities
- Credential access
- Data access
- Persistence
- Container escape
- Worker-node impact
- Network reachability
- Business criticality
- Confirmed exfiltration
- Number of affected clusters
Malware Scope Analysis
Section titled “Malware Scope Analysis”Determine whether the malware affected:
Single Process
↓
Single Container
↓
Entire Pod
↓
Namespace
↓
Worker Node
↓
Multiple Nodes
↓
Multiple Clusters
↓
AWS AccountSearch for the Same Hash
Section titled “Search for the Same Hash”Search enterprise telemetry for:
- Same file hash
- Same image digest
- Same download URL
- Same domain
- Same IP address
- Same process command line
- Same Service Account
- Same container image
- Same CI/CD build
Search Other Clusters
Section titled “Search Other Clusters”Questions include:
- Is the same image deployed elsewhere?
- Does the same file exist in other Pods?
- Did the same external destination receive traffic?
- Did other clusters generate similar runtime alerts?
- Was the same vulnerable application version deployed?
Correlate with Kubernetes Audit Logs
Section titled “Correlate with Kubernetes Audit Logs”Audit logs may show:
- Who created the Pod
- Who modified the Deployment
- Who created an ephemeral container
- Who executed into the Pod
- Who created a malicious DaemonSet
- Who modified a ConfigMap
- Who accessed Secrets
- Who deleted evidence
Correlate with CloudTrail
Section titled “Correlate with CloudTrail”CloudTrail may show:
- Role assumptions
- EKS configuration changes
- ECR image activity
- Secrets Manager access
- S3 access
- IAM changes
- Snapshot activity
- Security-tool changes
Correlate with Runtime Alerts
Section titled “Correlate with Runtime Alerts”Use:
- Falco
- GuardDuty Runtime Monitoring
- eBPF tools
- Commercial runtime platforms
- SIEM detections
These may reveal:
- Initial execution
- File access
- Network connections
- Privilege escalation
- Agent tampering
Correlate with Network Telemetry
Section titled “Correlate with Network Telemetry”Review:
- VPC Flow Logs
- DNS logs
- Proxy logs
- Firewall logs
- Load balancer logs
- Service-mesh telemetry
Example Malware Timeline
Section titled “Example Malware Timeline”10:01 — Malicious HTTP Request Reached Application
10:02 — Application Spawned Shell
10:03 — Script Downloaded to /tmp/update.sh
10:04 — Script Downloaded ELF Binary
10:05 — Binary Executed
10:06 — Service Account Token Read
10:07 — Secrets Listed Through Kubernetes API
10:08 — Connection Opened to External Command Server
10:10 — Cryptomining Process Started
10:12 — Falco Alert Generated
10:14 — Pod QuarantinedContainment Actions
Section titled “Containment Actions”Depending on the incident:
- Quarantine the affected Pod
- Restrict network egress
- Remove the workload from Service traffic
- Scale the owning workload to zero
- Block malicious domains and IP addresses
- Block the image digest
- Revoke workload IAM access
- Rotate Secrets
- Cordon the worker node
- Preserve relevant volumes
- Disable malicious Kubernetes resources
Do Not Destroy Evidence Prematurely
Section titled “Do Not Destroy Evidence Prematurely”Before deleting:
- Pod
- Image
- Namespace
- Node
- Persistent volume
- Registry artefact
confirm that required evidence has been preserved.
Credential Response
Section titled “Credential Response”Rotate or revoke credentials that malware could access:
- Service Account tokens
- EKS Pod Identity credentials
- IRSA credentials
- Node IAM credentials
- Database passwords
- API keys
- Secrets Manager values
- TLS certificates
- Registry credentials
- CI/CD credentials
Image Response
Section titled “Image Response”If malware was embedded in an image:
- Quarantine the repository or digest.
- Block deployment through admission policy.
- Preserve the image.
- Identify the introducing pipeline.
- Rebuild from trusted source.
- Review the build runner.
- Rotate build credentials.
- Sign the replacement image.
- Verify it during admission.
Node Response
Section titled “Node Response”Escalate to node forensics if malware:
- Accessed runtime sockets
- Used HostPath
- Executed host processes
- Accessed node credentials
- Modified host files
- Loaded kernel modules
- Disabled node security agents
- Escaped the container namespace
Eradication
Section titled “Eradication”Eradication may include:
- Removing malicious Kubernetes resources
- Patching application vulnerabilities
- Rebuilding container images
- Removing compromised dependencies
- Cleaning CI/CD pipelines
- Revoking credentials
- Replacing worker nodes
- Restoring clean persistent volumes
- Removing persistence mechanisms
- Updating security policies
Recovery
Section titled “Recovery”Recovery should use:
- Trusted source repositories
- Clean build environments
- Approved dependencies
- Scanned images
- Signed images
- Immutable digests
- Least-privilege identities
- Restricted networking
- Active runtime monitoring
Recovery Validation
Section titled “Recovery Validation”Confirm:
- Malware hash no longer exists
- Malicious image digest is blocked
- No suspicious processes are running
- No command-and-control traffic remains
- Credentials are rotated
- Worker nodes are trustworthy
- Application vulnerability is patched
- Admission controls are active
- Runtime monitoring is reporting
- Related indicators are monitored
Detection Engineering
Section titled “Detection Engineering”Malware-analysis findings should be converted into detections.
Examples include:
- File-hash detections
- Process-tree detections
- Network-domain alerts
- Image-digest blocks
- Command-line detections
- Sensitive file-access alerts
- Runtime socket-access alerts
- Kubernetes resource detections
- AWS API anomaly detections
Example Behaviour Detection
Section titled “Example Behaviour Detection”Production Container
AND
Application Process Spawns Shell
AND
Shell Downloads Executable
AND
Executable Runs from /tmpThis behavioural detection may identify future malware variants.
Admission Policy Improvements
Section titled “Admission Policy Improvements”Use findings to enforce controls such as:
- Approved image registries
- Signed image verification
- Immutable image digests
- Non-root execution
- Read-only root filesystems
- No HostPath
- No runtime socket mounts
- No privileged containers
- Restricted Linux capabilities
- Required resource limits
Runtime Rule Improvements
Section titled “Runtime Rule Improvements”Update runtime detections for:
- Specific hashes
- Specific paths
- Shell execution
- Package-manager use
- Downloader execution
- Credential-file access
- Mining-pool communication
- Security-agent termination
- Host namespace activity
Network Security Improvements
Section titled “Network Security Improvements”Use malware indicators to update:
- DNS filters
- Network firewall rules
- Proxy policies
- Egress allowlists
- SIEM detections
- Threat-intelligence platforms
- Network Policies where applicable
Malware Evidence Checklist
Section titled “Malware Evidence Checklist”Source Evidence
Section titled “Source Evidence”- Pod YAML
- Pod JSON
- Workload manifest
- Image reference
- Runtime image ID
- Container logs
- Runtime alert
- Node information
File Evidence
Section titled “File Evidence”- Original path
- File metadata
- File size
- SHA-256 hash
- Preserved original
- Working copy
- Collection method
- Chain of custody
Static Analysis
Section titled “Static Analysis”- File type
- Architecture
- Strings
- URLs
- IP addresses
- Domains
- Commands
- Imported functions
- Packer indicators
Dynamic Analysis
Section titled “Dynamic Analysis”- Process tree
- Files created
- Files modified
- Network connections
- DNS queries
- Credential access
- Persistence
- Exit behaviour
Enterprise Correlation
Section titled “Enterprise Correlation”- Kubernetes Audit Logs
- CloudTrail
- VPC Flow Logs
- DNS logs
- GuardDuty
- Falco
- Registry logs
- CI/CD evidence
Enterprise Malware Analysis Workflow
Section titled “Enterprise Malware Analysis Workflow”Runtime Alert or Suspicious File
↓
Identify Pod, Container and Node
↓
Preserve Metadata and Logs
↓
Collect Sample Safely
↓
Calculate Hash
↓
Store Original Evidence
↓
Perform Static Analysis
↓
Perform Controlled Dynamic Analysis
↓
Extract Indicators
↓
Search Enterprise Environment
↓
Determine Scope
↓
Contain Workload and Credentials
↓
Eradicate Root Cause
↓
Rebuild Trusted Workload
↓
Update Detections and PoliciesEnterprise Roles
Section titled “Enterprise Roles”| Role | Responsibility |
|---|---|
| SOC Analyst | Triage alert and collect initial context |
| Cloud Security Engineer | Collect Kubernetes and AWS evidence |
| Malware Analyst | Perform static and dynamic analysis |
| Platform Engineer | Isolate workloads and nodes |
| Application Owner | Explain expected application behaviour |
| IAM Team | Revoke and rotate identities |
| Network Security Team | Block malicious destinations |
| Incident Commander | Coordinate response |
| Legal or Compliance | Manage evidence and notification obligations |
Common Malware Analysis Mistakes
Section titled “Common Malware Analysis Mistakes”Executing Malware on a Corporate Workstation
Section titled “Executing Malware on a Corporate Workstation”Risk: The workstation or corporate network becomes compromised.
Response: Use a dedicated isolated analysis environment.
Analysing the Original Evidence File
Section titled “Analysing the Original Evidence File”Risk: Evidence may be modified.
Response: Preserve the original and analyse a verified working copy.
Deleting the Pod Before Collection
Section titled “Deleting the Pod Before Collection”Risk: Malware files and runtime evidence are lost.
Response: Quarantine and preserve evidence first.
Uploading Malware to Unapproved Public Services
Section titled “Uploading Malware to Unapproved Public Services”Risk: Sensitive information or proprietary samples may be exposed.
Response: Follow approved threat-intelligence and malware-sharing procedures.
Trusting File Names
Section titled “Trusting File Names”Risk: Malware may use names that resemble legitimate processes.
Response: Review hashes, paths, behaviour and process ancestry.
Focusing Only on File Hashes
Section titled “Focusing Only on File Hashes”Risk: New malware variants may use different hashes.
Response: Extract behavioural and network indicators.
Ignoring the Container Image
Section titled “Ignoring the Container Image”Risk: Malware embedded in the image is redeployed.
Response: Analyse image history and layers.
Ignoring Persistent Volumes
Section titled “Ignoring Persistent Volumes”Risk: Malware or web shells survive Pod replacement.
Response: Inspect attached storage.
Ignoring Credentials
Section titled “Ignoring Credentials”Risk: Stolen credentials remain usable after workload recovery.
Response: Rotate all potentially exposed credentials.
Restoring the Same Vulnerable Image
Section titled “Restoring the Same Vulnerable Image”Risk: The workload is immediately compromised again.
Response: Rebuild, scan, sign and validate a patched image.
Enterprise Best Practices
Section titled “Enterprise Best Practices”As a Cloud Security Engineer:
- Maintain a documented malware-handling procedure.
- Use isolated analysis environments.
- Preserve original samples and analyse copies.
- Calculate SHA-256 hashes immediately.
- Maintain chain of custody.
- Record original file metadata and paths.
- Perform static analysis before dynamic execution.
- Never use production credentials in a sandbox.
- Restrict malware-analysis network access.
- Capture process, file, DNS and network behaviour.
- Analyse container image layers and history.
- Compare suspicious images with approved digests and SBOMs.
- Correlate malware findings with Audit Logs and CloudTrail.
- Search other clusters for the same indicators.
- Treat accessible credentials as compromised.
- Escalate to node forensics when escape indicators exist.
- Convert findings into SIEM, runtime and network detections.
- Block malicious image digests through admission control.
- Rebuild workloads from trusted source.
- Test recovery before returning applications to production.
- Review malware-analysis procedures through regular exercises.
Real-World Scenario
Section titled “Real-World Scenario”A financial organisation operates a payment-processing application on Amazon EKS.
Falco detects an unknown executable running from:
/tmp/system-updateThe process consumes high CPU and connects to an external domain.
The Cloud Security team begins a malware investigation.
They:
- Identify the affected cluster, namespace, Pod, container and worker node.
- Export the Pod manifest and runtime image ID.
- Preserve current and previous container logs.
- Record the process tree and network connection.
- Copy the suspicious file through an approved forensic procedure.
- Calculate the SHA-256 hash.
- Store the original sample in a protected evidence repository.
- Perform static analysis on a verified working copy.
- Identify strings referencing a cryptocurrency mining pool.
- Discover code that reads the Kubernetes Service Account token.
- Analyse the container image and confirm the file was not present in the approved image.
- Perform dynamic analysis in an isolated sandbox.
- Observe the sample creating worker threads and contacting the mining pool.
- Extract domains, IP addresses, file paths and command-line indicators.
- Search all EKS clusters for the same hash and network indicators.
- Identify two additional compromised Pods using the same vulnerable application version.
- Quarantine the affected workloads.
- Block the malicious destination at the enterprise egress layer.
- Revoke workload IAM access and rotate associated Secrets.
- Patch the vulnerable application dependency.
- Rebuild, scan and sign the application image.
- Redeploy using an immutable digest.
- Add runtime detections for the process behaviour.
- Add an admission policy requiring read-only root filesystems.
- Confirm that no worker-node escape occurred.
The analysis determines that the malware was downloaded after the application was exploited and was used for cryptomining and credential discovery.
Key Takeaways
Section titled “Key Takeaways”- Malware analysis identifies the purpose, capabilities and impact of suspicious files.
- Malware samples must be collected and analysed safely.
- Original evidence should be preserved and working copies should be used for analysis.
- Static analysis examines a sample without execution.
- Dynamic analysis observes behaviour in an isolated environment.
- Container image layers must be analysed to determine whether malware existed at build time.
- Hashes, domains, IP addresses, paths and behaviours become Indicators of Compromise.
- Behavioural detections are often more durable than file-hash detections.
- Kubernetes Audit Logs, CloudTrail and network telemetry provide important context.
- Malware that accesses host resources requires node-forensics escalation.
- Potentially exposed credentials must be rotated.
- Recovery should use rebuilt, scanned, signed and digest-pinned images.
- Malware-analysis findings should improve runtime rules, admission policies and network controls.
Knowledge Check
Section titled “Knowledge Check”1. What is the difference between static and dynamic malware analysis?
Section titled “1. What is the difference between static and dynamic malware analysis?”Answer: Static analysis examines a suspicious file without executing it, while dynamic analysis executes the sample in an isolated environment to observe its runtime behaviour.
2. Why should the original malware sample not be used directly for analysis?
Section titled “2. Why should the original malware sample not be used directly for analysis?”Answer: Analysis may modify file metadata or content. Preserving the original sample and analysing a verified copy protects evidence integrity.
3. Why is the SHA-256 hash important?
Section titled “3. Why is the SHA-256 hash important?”Answer: It uniquely identifies the sample, supports integrity verification, enables searches across environments and helps correlate related incidents.
4. Why should container image layers be inspected?
Section titled “4. Why should container image layers be inspected?”Answer: Malware, secrets or malicious commands may exist in earlier layers even if later layers delete or hide them.
5. When should malware analysis escalate to node forensics?
Section titled “5. When should malware analysis escalate to node forensics?”Answer: Escalation is required when the malware accesses runtime sockets, host filesystems, node credentials, kernel resources or executes host-level processes.
What’s Next?
Section titled “What’s Next?”In the next lesson, we will explore Kubernetes Memory Forensics, including volatile evidence collection, process-memory analysis, credential discovery, memory-dump handling and investigation of advanced in-memory threats.
➡️ Next Lesson: Lesson 08 — Memory Forensics