Lesson 09 — Root Cause Analysis

Learning Objectives

By the end of this lesson, you will be able to:

Explain the purpose of Root Cause Analysis in Kubernetes incident response
Differentiate root causes, contributing factors and incident symptoms
Reconstruct a complete Kubernetes incident timeline
Identify the initial access vector
Analyse identity, workload, network and runtime evidence
Determine the technical and business blast radius
Apply structured Root Cause Analysis techniques
Use the Five Whys and fault-tree analysis
Identify failures across preventive, detective and responsive controls
Develop evidence-based corrective and preventive actions
Assign remediation owners and deadlines
Validate that remediation addresses the actual root cause
Produce an enterprise Kubernetes Root Cause Analysis report

Why This Matters

Containment stops an active incident.

Recovery restores the affected service.

Root Cause Analysis determines why the incident was possible.

Without Root Cause Analysis, an organisation may restore the application but leave the original weakness unchanged.

Incident

↓

Containment

↓

Recovery

↓

No Root Cause Analysis

↓

Same Weakness Remains

↓

Incident Happens Again

A Kubernetes incident may appear to be caused by one compromised Pod, but the deeper cause may involve:

A vulnerable application dependency
An insecure container image
Excessive Service Account permissions
A broad workload IAM role
Missing Network Policies
An unreviewed policy exception
Compromised CI/CD credentials
Missing runtime monitoring
Inadequate change management
Poor ownership and governance

Root Cause Analysis moves the investigation beyond:

What happened?

to:

Why was it able to happen, why was it not prevented and why was it not detected sooner?

What is Root Cause Analysis?

Root Cause Analysis, or RCA, is a structured process used to identify the underlying technical, procedural and organisational conditions that allowed an incident to occur.

The objective is not merely to identify the final malicious action.

The objective is to determine:

How the attacker entered the environment
Which weaknesses enabled the attack
Why existing controls did not prevent it
Why detection was delayed
Why the impact expanded
Which actions will prevent recurrence

Observed Incident

↓

Immediate Cause

↓

Contributing Conditions

↓

Underlying Root Cause

↓

Corrective and Preventive Actions

Root Cause Analysis Outcomes

A successful RCA should produce:

A verified incident timeline
A confirmed or most likely initial access vector
Identified root causes
Identified contributing factors
An assessment of affected systems and data
An explanation of control failures
Corrective actions
Preventive actions
Assigned owners
Target completion dates
Validation criteria
Lessons for the wider environment

Root Cause, Contributing Factor and Symptom

These terms should not be treated as interchangeable.

Term	Meaning	Example
Symptom	Observable result of the incident	High CPU usage
Indicator	Evidence suggesting compromise	Mining-pool connection
Immediate cause	Direct action producing the event	Cryptominer process executed
Contributing factor	Condition that increased likelihood or impact	Unrestricted egress
Root cause	Fundamental condition that enabled the incident	Vulnerable public application deployed without required patch validation

Example Cause Chain

Symptom

Worker Node CPU Reached 100%

↓

Immediate Cause

Cryptomining Process Running

↓

Attack Action

Attacker Downloaded Mining Binary

↓

Initial Access

Remote Code Execution in Application

↓

Contributing Factor

Writable Root Filesystem and Unrestricted Egress

↓

Root Cause

Known Vulnerable Dependency Was Not Blocked by the Software Supply-Chain Process

Removing the miner treats the symptom.

Patching the application addresses the immediate weakness.

Correcting the dependency-governance process addresses the root cause.

Root Cause Analysis Principles

An enterprise RCA should follow these principles:

Base conclusions on evidence.
Separate facts from assumptions.
Avoid blame-focused language.
Analyse technical and process failures together.
Consider multiple contributing factors.
Identify control failures across the complete lifecycle.
Assign measurable remediation actions.
Validate remediation effectiveness.
Share relevant lessons across other clusters and teams.
Treat RCA as a security-improvement activity.

Blameless Root Cause Analysis

A blameless RCA does not mean that accountability is removed.

It means the investigation focuses on:

System design
Process weaknesses
Missing safeguards
Ambiguous responsibilities
Inadequate training
Tooling limitations
Governance gaps

Avoid conclusions such as:

The Engineer Made a Mistake

Prefer:

The deployment process allowed an unreviewed privileged configuration to reach production because no automated admission control or approval gate was present.

The second statement identifies conditions that can be improved.

Kubernetes RCA Scope

Kubernetes Root Cause Analysis

├── Initial Access
├── Application Security
├── Container Image
├── CI/CD Pipeline
├── Kubernetes API Activity
├── IAM and RBAC
├── Workload Identity
├── Pod Security
├── Network Controls
├── Secrets and Data
├── Runtime Behaviour
├── Worker Node
├── Logging and Detection
├── Incident Response
└── Governance

Root Cause Analysis Lifecycle

Incident Stabilised

↓

Evidence Validated

↓

Timeline Reconstructed

↓

Initial Access Identified

↓

Attack Path Analysed

↓

Blast Radius Determined

↓

Control Failures Identified

↓

Root Causes Confirmed

↓

Corrective Actions Defined

↓

Remediation Validated

↓

Lessons Shared

When Should RCA Begin?

RCA normally begins after:

Immediate threats have been contained
Critical services have been stabilised
Volatile evidence has been preserved
Relevant logs and forensic artefacts have been collected
Incident ownership has been established

RCA preparation can begin during active response, but investigators should avoid premature conclusions.

RCA Roles and Responsibilities

Role	Responsibility
Incident Commander	Coordinates the RCA process
Cloud Security Engineer	Analyses Kubernetes and AWS security evidence
Platform Engineer	Explains cluster, node and platform behaviour
Application Owner	Provides application and dependency context
SOC Analyst	Provides detection and alert chronology
Forensic Analyst	Validates host, container, malware and memory evidence
IAM Team	Analyses identity and credential activity
Network Security Team	Analyses ingress, egress and lateral movement
CI/CD Team	Reviews build and deployment pipelines
Risk and Compliance	Assesses reporting and control implications
Executive Sponsor	Ensures remediation receives appropriate priority

RCA Evidence Sources

A Kubernetes RCA may use evidence from:

Kubernetes Evidence

Kubernetes Audit Logs
Pod manifests
Deployment manifests
Events
RBAC
Service Accounts
Network Policies
Admission-controller logs
Policy reports
Namespace labels
Secret references

Runtime Evidence

Falco alerts
GuardDuty Runtime Monitoring findings
Process trees
File hashes
Network connections
Malware-analysis results
Memory-analysis results
Container-runtime artefacts

AWS Evidence

AWS CloudTrail
VPC Flow Logs
Route 53 Resolver logs
Security Hub
Amazon Inspector
Amazon ECR activity
IAM policy history
STS activity
Secrets Manager access
KMS events
EC2 and EBS evidence

Application Evidence

Application logs
Web server logs
Load balancer logs
AWS WAF logs
Authentication logs
Database logs
API gateway logs
Distributed traces

Operational Evidence

Change records
Pull requests
Pipeline logs
Approval records
Incident tickets
Exception records
Maintenance activity
On-call communications

Evidence Reliability

Not every evidence source has the same reliability.

Evidence Type	Consideration
Kubernetes Audit Log	Strong record of Kubernetes API requests
CloudTrail	Strong record of AWS API activity
Runtime alert	Useful detection evidence requiring validation
Application log	May be incomplete or attacker-controlled
File timestamp	May be modified by an attacker
Human recollection	Useful but should be corroborated
Screenshot	Helpful context but weaker than raw data
SIEM event	Validate against the original source where possible

Facts, Assumptions and Unknowns

Every RCA should classify findings.

Confirmed Fact

Supported by reliable evidence.

CloudTrail confirms that the workload IAM role accessed the production secret at 10:07 UTC.

Supported Inference

Evidence strongly suggests a conclusion, but direct proof is unavailable.

The attacker most likely obtained the workload token after exploiting the application because token access occurred immediately after shell execution.

Assumption

A working theory requiring validation.

The same actor may have attempted access to the second cluster.

Unknown

Evidence is unavailable or insufficient.

It is unknown whether the attacker viewed the decrypted database records because query-level logging was not enabled.

This distinction increases report credibility.

Phase 1 — Define the Incident

Create a concise incident statement.

Example:

On 15 August 2026, an unauthorised actor exploited a vulnerable payment API running in the production Amazon EKS cluster, executed a reverse shell, accessed application credentials and attempted to retrieve data from Amazon S3.

The statement should identify:

Date and time
Environment
Affected service
Nature of compromise
Confirmed or suspected impact

Incident Scope Statement

Document:

Incident ID:

Incident Title:

Severity:

Date Detected:

Date Contained:

Affected AWS Accounts:

Affected Regions:

Affected Clusters:

Affected Namespaces:

Affected Applications:

Business Services:

Data Classification:

Current Status:

Phase 2 — Build the Timeline

A reliable timeline is the foundation of Root Cause Analysis.

Pre-Incident Conditions

↓

Initial Access

↓

Execution

↓

Privilege Escalation

↓

Credential Access

↓

Discovery

↓

Lateral Movement

↓

Collection

↓

Exfiltration

↓

Detection

↓

Containment

↓

Recovery

Timeline Construction

Collect events from all relevant sources and normalise them to one timezone, preferably UTC.

Time	Source	Event	Identity	Resource	Confidence
09:58 UTC	ALB log	Malicious request received	External client	Payment API	Confirmed
09:59 UTC	Falco	Shell launched by application	Container process	Payment Pod	Confirmed
10:00 UTC	Runtime evidence	Downloader executed	Container process	`/tmp/update`	Confirmed
10:02 UTC	Audit log	Secrets listed	Payment Service Account	Payments namespace	Confirmed
10:04 UTC	CloudTrail	Secret value requested	Workload IAM role	Secrets Manager	Confirmed
10:05 UTC	VPC Flow Logs	External connection opened	Pod IP	Malicious address	Confirmed
10:07 UTC	SIEM	Runtime incident alert raised	Falco	SOC	Confirmed
10:14 UTC	IR log	Pod quarantined	Platform responder	Payment Pod	Confirmed

Timestamp Normalisation

Record:

Original timestamp
Original timezone
Normalised UTC timestamp
Source clock
Known clock drift
Log-ingestion delay

Do not assume SIEM ingestion time equals event occurrence time.

Pre-Incident Timeline

RCA should include relevant activity before the visible attack.

Review:

Recent application releases
Dependency updates
IAM changes
RBAC changes
Policy exceptions
Network changes
Node upgrades
Detection-rule changes
Logging changes
CI/CD changes
Vulnerability findings

The root cause may have been introduced days or weeks before exploitation.

Change Correlation

Ask:

What changed before the incident?
Was a vulnerable version deployed?
Was a policy disabled?
Was an exception approved?
Was a Service Account permission expanded?
Was a Network Policy removed?
Did a deployment bypass the normal pipeline?
Was runtime monitoring degraded?

Phase 3 — Identify Initial Access

Initial access is the first verified or most likely point where the attacker gained unauthorised capability.

Common Kubernetes initial access vectors include:

Public application vulnerability
Stolen AWS credentials
Compromised kubeconfig
Malicious container image
Compromised CI/CD pipeline
Exposed Kubernetes API endpoint
Insecure dashboard or management tool
Vulnerable admission webhook
Compromised third-party integration
Insider misuse
Exposed Service Account token

Application Exploitation

Evidence may include:

Malicious inbound request
Application error
Application process spawning a shell
Runtime alert
New files under /tmp
Outbound command-and-control connection
No corresponding pods/exec audit event

External Request

↓

Application Vulnerability

↓

Application Process Spawns Shell

↓

Attacker Gains Container Execution

Credential Compromise

Evidence may include:

IAM role assumed from an unfamiliar address
Kubernetes API access from an unusual source
Successful activity after multiple failures
No related application exploit
Unapproved kubectl or API user agent
MFA or identity-provider anomalies
Activity outside normal working patterns

CI/CD Compromise

Evidence may include:

Malicious code introduced during build
Unexpected image layer
Unapproved build runner
Image signed by a compromised identity
Pipeline role misuse
Direct registry push
Deployment outside approved change flow

Malicious Image

Determine whether malware was:

Present in Source Code

or

Added During Build

or

Added to Registry

or

Downloaded at Runtime

Compare:

Source commit
Pipeline logs
Build provenance
Image signature
Registry digest
SBOM
Runtime filesystem

Exposed Kubernetes API

Evidence may include:

Public endpoint access
Requests from unknown IP addresses
Stolen IAM credentials
New access entries
Suspicious RBAC changes
Automated API enumeration
Unapproved user agents

Phase 4 — Reconstruct the Attack Path

An attack path describes how the attacker progressed from initial access to final impact.

Initial Access

↓

Code Execution

↓

Credential Access

↓

Privilege Escalation

↓

Internal Discovery

↓

Lateral Movement

↓

Data Access

↓

Exfiltration or Impact

Example Kubernetes Attack Path

Vulnerable Application

↓

Remote Code Execution

↓

Shell Inside Pod

↓

Service Account Token Read

↓

Secrets Listed

↓

Database Credential Retrieved

↓

Internal Database Access

↓

Sensitive Data Collected

↓

External Exfiltration Attempt

Attack Path Table

Stage	Attacker Action	Evidence	Control Expected	Control Result
Initial access	Exploited application	ALB and app logs	Patch management	Failed
Execution	Spawned shell	Falco	Runtime monitoring	Detected
Credential access	Read Pod token	Runtime event	Token minimisation	Failed
Discovery	Listed Secrets	Audit logs	Least-privilege RBAC	Failed
Lateral movement	Connected to database	Flow logs	Network Policy	Failed
Exfiltration	Connected externally	Flow logs	Egress restriction	Failed
Detection	Runtime alert generated	SIEM	Runtime detection	Succeeded
Containment	Pod isolated	IR record	Response runbook	Succeeded

Phase 5 — Determine the Blast Radius

Blast radius is the total potential and confirmed impact of the incident.

Assess:

Affected process
Affected container
Affected Pod
Other containers in the Pod
Namespace
Worker node
Other namespaces
Other clusters
AWS resources
Data
Customers
Business services

Technical Blast Radius

Compromised Process

↓

Container

↓

Pod

↓

Namespace

↓

Node

↓

Cluster

↓

AWS Account

↓

Connected Enterprise Systems

Identity Blast Radius

Review all identities accessible to the attacker:

Kubernetes Service Account
EKS Pod Identity role
IRSA role
Node IAM role
Secrets Manager credentials
Database identity
API keys
CI/CD token
Registry credentials
User credentials

Network Blast Radius

Determine which systems the compromised workload could reach:

Other Pods
Other namespaces
Cluster services
Databases
Internal APIs
AWS service endpoints
On-premises systems
Internet destinations
Other VPCs
Other accounts

Data Blast Radius

Determine:

Which data stores were reachable?
Which credentials were available?
Which records were queried?
Was data copied or staged?
Was outbound transfer observed?
Was sensitive data logged?
Was encryption bypassed?
Is customer notification required?

Image Blast Radius

If the incident involves an image, identify:

Every cluster using the digest
Every account with the image replicated
Every workload using the same base image
Every environment built by the same pipeline
Every image signed by the affected signing identity

Cluster and Account Blast Radius

Search for shared dependencies:

Same IAM roles
Same CI/CD pipeline
Same policy exception
Same Terraform module
Same admission configuration
Same vulnerable add-on
Same node AMI
Same registry repository
Same secret

Confirmed, Potential and Excluded Impact

Classify scope clearly.

Scope Type	Meaning
Confirmed	Evidence proves the resource was affected
Potential	The attacker had the capability to affect it
Excluded	Evidence supports that it was not affected
Unknown	Available evidence is insufficient

Phase 6 — Identify Control Failures

Analyse why each expected security control did or did not work.

Preventive Controls

↓

Detective Controls

↓

Responsive Controls

↓

Recovery Controls

Preventive Control Review

Review:

Application patching
Dependency scanning
Image scanning
Image signing
Admission policies
Pod Security Admission
RBAC
Workload IAM
Network Policies
Egress controls
Secrets management
API endpoint protection

Detective Control Review

Review:

Kubernetes Audit Logs
CloudTrail
Runtime monitoring
GuardDuty
SIEM rules
Network monitoring
DNS monitoring
Application security logging
Vulnerability alerts

Responsive Control Review

Review:

Alert routing
On-call ownership
Escalation
Quarantine procedures
Credential revocation
Evidence collection
Node isolation
Communication

Recovery Control Review

Review:

Trusted image rebuild
Node replacement
Backup restoration
Secret rotation
GitOps recovery
Validation
Post-recovery monitoring

Control Status Classification

Status	Meaning
Effective	Control operated as intended
Partially Effective	Control reduced impact but had limitations
Ineffective	Control existed but failed
Missing	Required control was absent
Bypassed	Control was intentionally or maliciously circumvented
Not Applicable	Control was not relevant to the event

Example Control Analysis

Control	Expected Outcome	Actual Result	Status
Dependency scanning	Block critical vulnerability	Vulnerable version passed pipeline	Ineffective
Non-root policy	Reduce exploit impact	Container ran as root	Missing
Network Policy	Prevent lateral movement	Namespace had no egress policy	Missing
Runtime detection	Detect shell	Alert generated in one minute	Effective
SIEM routing	Notify SOC	Alert reached SOC after six minutes	Partially Effective
IR runbook	Isolate Pod	Quarantine completed successfully	Effective

Phase 7 — Apply Root Cause Analysis Methods

Several methods can help structure the investigation.

The Five Whys

The Five Whys method repeatedly asks why an event occurred.

Example

Problem

A production Pod executed cryptomining malware.

Why 1

Why did the Pod execute mining malware?

Because an attacker gained remote command execution and downloaded a miner.

Why 2

Why did the attacker gain remote command execution?

Because the application contained a known vulnerable dependency.

Why 3

Why was the vulnerable dependency deployed?

Because the build pipeline did not block critical exploitable vulnerabilities.

Why 4

Why did the pipeline not block the vulnerability?

Because the vulnerability scanner operated in report-only mode.

Why 5

Why was it configured in report-only mode?

Because no production vulnerability acceptance standard or accountable approval owner had been defined.

Root Cause

The organisation lacked an enforced production vulnerability-gating standard with clear ownership.

Five Whys Limitations

The Five Whys should not force every incident into a single linear cause.

Complex Kubernetes incidents often have several causes.

Application Vulnerability

+

Excessive RBAC

+

Unrestricted Egress

+

Delayed Alert Triage

=

Expanded Incident Impact

Use branching analysis when necessary.

Fault-Tree Analysis

Fault-tree analysis works backward from the incident outcome.

Sensitive Data Accessed

AND / OR

├── Attacker Obtained Application Execution
│   ├── Vulnerable Dependency
│   └── Publicly Reachable Service
│
├── Attacker Obtained Credentials
│   ├── Service Account Token Mounted
│   └── Excessive RBAC
│
└── Attacker Reached Data Store
    ├── No Network Policy
    └── Shared Database Credential

This approach helps identify several conditions that combined to create impact.

Fishbone Analysis

A fishbone or cause-and-effect analysis may group contributing factors into categories.

People

Process

Technology

Configuration

Governance

Third Parties

Environment

Example Factors

People

Limited Kubernetes security training
Unclear on-call ownership

Process

No production vulnerability gate
Exception reviews not scheduled

Technology

Missing runtime monitoring in one cluster
No egress filtering

Configuration

Broad Service Account permissions
Writable root filesystem

Governance

No namespace security baseline
Security owner not assigned

Barrier Analysis

Barrier analysis asks which safeguards should have stopped or limited the event.

Threat

↓

Barrier 1 — Secure Coding

↓

Barrier 2 — Dependency Scanning

↓

Barrier 3 — Admission Control

↓

Barrier 4 — Workload Hardening

↓

Barrier 5 — Network Segmentation

↓

Barrier 6 — Runtime Detection

↓

Incident Impact

For each barrier, determine whether it was:

Present
Correctly configured
Monitored
Tested
Bypassed
Effective

Causal Factor Charting

Causal factor charting connects timeline events to conditions.

Vulnerable Release Approved

↓

Application Deployed

↓

Malicious Request Received

↓

Shell Spawned

↓

Token Read

↓

Secrets Listed

↓

Database Accessed

Contributing conditions are attached to relevant events.

Shell Spawned

├── Container Ran as Root
└── Writable Filesystem

Token Read

├── Automatic Token Mounting Enabled
└── Token Not Required by Application

Secrets Listed

└── Service Account Had Broad Permissions

Phase 8 — Identify Root Causes

A root cause should be:

Supported by evidence
Specific
Actionable
Deeper than the visible symptom
Connected to the incident outcome
Capable of preventing recurrence if corrected

Weak Root Cause Statements

Avoid statements such as:

The Pod Was Compromised

The Developer Made a Mistake

Security Was Weak

The Attacker Was Sophisticated

These do not identify actionable causes.

Strong Root Cause Statements

Prefer:

The production deployment pipeline allowed applications with unresolved critical exploitable dependencies because vulnerability scanning was configured only to report findings and no mandatory release gate existed.

The compromised workload could enumerate namespace Secrets because the Service Account was bound to a broad reusable Role rather than an application-specific least-privilege Role.

The attacker established outbound command-and-control communication because the production namespace had no default-deny egress policy or central egress restriction.

Multiple Root Causes

An incident may have several root causes.

Example:

Root Cause 1 — Software Supply Chain

The release process did not prevent deployment of a known exploitable dependency.

Root Cause 2 — Identity

The application Service Account had permissions beyond its business requirement.

Root Cause 3 — Network Security

The namespace permitted unrestricted outbound communication.

Root Cause 4 — Governance

A security exception had no expiry date or periodic review.

Contributing Factors

Contributing factors may not independently cause the incident but increase its likelihood, duration or impact.

Examples include:

Container ran as root
Root filesystem was writable
Service Account token was automatically mounted
No read-only filesystem
Broad network reachability
Delayed SIEM ingestion
Incomplete ownership metadata
No tested Pod-quarantine procedure
Insufficient log retention
Shared credentials

Root Cause Categories

Classify causes to support enterprise trend analysis.

Category	Examples
Application Security	Vulnerable code or dependency
Identity	Excessive IAM or RBAC
Network	Missing segmentation or egress control
Platform	Insecure cluster or node configuration
Supply Chain	Compromised build or unsigned image
Secrets	Static or shared credentials
Detection	Missing or ineffective monitoring
Response	Delayed escalation or containment
Process	Missing review or approval
Governance	Undefined standard or ownership
Training	Insufficient role-specific knowledge
Third Party	Compromised vendor component

Phase 9 — Develop Corrective Actions

Corrective actions address conditions directly related to the incident.

Examples include:

Patch the vulnerable dependency
Rebuild the image
Rotate credentials
Remove excessive RBAC
Restrict IAM permissions
Apply Network Policies
Replace compromised nodes
Block malicious image digests
Remove unsafe policy exceptions

Corrective Versus Preventive Actions

Corrective Action	Preventive Action
Fixes the specific incident condition	Reduces recurrence across the environment
Patch affected application	Add enforced vulnerability gates
Rotate affected secret	Automate secret rotation
Remove one broad RoleBinding	Deploy RBAC policy validation
Block one malicious domain	Implement controlled egress
Replace one compromised node	Adopt immutable node replacement standards

Remediation Action Requirements

Every action should include:

Action ID:

Related Root Cause:

Description:

Priority:

Owner:

Target Date:

Affected Scope:

Implementation Plan:

Validation Method:

Evidence Required:

Status:

SMART Remediation Actions

Actions should be:

Specific
Measurable
Achievable
Relevant
Time-bound

Weak action:

Improve Kubernetes Security

Strong action:

Deploy and enforce a default-deny egress Network Policy in every production application namespace by 30 September 2026, with automated compliance validation in CI/CD and weekly reporting.

Prioritising Actions

Prioritise based on:

Risk severity
Exploitability
Internet exposure
Number of affected clusters
Data sensitivity
Ease of attacker reuse
Compliance implications
Remediation complexity
Availability of compensating controls

Remediation Priority

Priority	Expected Response
Critical	Immediate containment and urgent remediation
High	Time-bound remediation with executive visibility
Medium	Planned remediation within defined SLA
Low	Improvement tracked through normal backlog

Immediate Actions

Examples:

Revoke compromised credentials
Block malicious destinations
Isolate affected workloads
Disable unsafe deployment paths
Quarantine image digests
Patch exposed services
Increase monitoring

Short-Term Actions

Examples:

Reduce Service Account permissions
Apply Network Policies
Enable runtime monitoring
Add admission controls
Improve alert routing
Update incident runbooks

Long-Term Actions

Examples:

Build a secure EKS landing zone
Redesign the software supply chain
Implement just-in-time access
Establish central egress inspection
Deploy multi-cluster policy governance
Improve security training
Define enterprise Kubernetes ownership

Remediation Dependency Mapping

Some actions depend on others.

Define Security Standard

↓

Build Policy

↓

Test in Audit Mode

↓

Remediate Existing Workloads

↓

Enable Enforcement

↓

Monitor Compliance

Dependencies should be documented to avoid unrealistic deadlines.

Phase 10 — Validate Remediation

Closing a ticket does not prove that the root cause has been addressed.

Validation may include:

Security retesting
CI/CD pipeline testing
Admission-policy testing
RBAC review
Network-isolation testing
Credential-access testing
Runtime attack simulation
Compliance scan
Tabletop exercise
Disaster-recovery test

Validation Questions

Ask:

Does the control prevent the original attack path?
Does it apply to every affected cluster?
Can teams bypass it?
Is the control monitored?
Is there an owner?
Does it generate evidence?
Has it been tested under failure conditions?
Did remediation introduce availability risk?

Reproduce the Original Condition Safely

In an authorised test environment:

Original Attack Step

↓

Expected Security Control

↓

Request Blocked or Detected

↓

Evidence Generated

↓

Runbook Triggered

Do not reproduce malicious activity in production.

Regression Testing

Confirm that security remediation does not break:

Application startup
Service communication
Deployment pipelines
Autoscaling
Monitoring
Backup
Incident response
Business transactions

Root Cause Closure Criteria

A root cause may be closed only when:

Corrective action is implemented
Preventive action is implemented or formally accepted
Validation has succeeded
Required evidence is attached
Residual risk is documented
Control owner is assigned
Relevant documentation is updated
Similar environments have been assessed

The incident may reveal risks beyond the original cluster.

Share relevant findings with:

Cloud Security
Platform Engineering
Application teams
SOC
DevSecOps
IAM
Network Security
Risk and Compliance
Leadership

Enterprise-Wide Search

After identifying the root cause, search for:

Same vulnerable dependency
Same image digest
Same base image
Same RBAC template
Same IAM policy
Same policy exception
Same exposed endpoint
Same missing Network Policy
Same pipeline configuration
Same unsupported add-on

Update Security Standards

RCA findings may require updates to:

Cluster baselines
Namespace templates
Admission policies
RBAC standards
IAM modules
Network standards
CI/CD controls
Logging requirements
Runtime rules
Backup procedures
Incident runbooks

Detection Engineering Improvements

Convert findings into detections.

Examples include:

Application process spawning a shell
Service Account listing Secrets
Workload role accessing unexpected resources
New cluster-admin binding
Runtime socket mount
Public LoadBalancer creation
Egress to unapproved destinations
Logging disablement
Deployment of the affected image digest

Threat-Hunting Improvements

Develop hunt queries for:

Related file hashes
Similar process trees
Same external infrastructure
Same identity activity
Same API sequence
Same image lineage
Same vulnerable workload version

Knowledge-Base Updates

Update:

Investigation guides
Runbooks
Detection documentation
Architecture standards
Secure coding guidance
Platform onboarding
Incident examples
Training labs

RCA Report Structure

A professional Root Cause Analysis report may contain:

1. Executive Summary

2. Incident Overview

3. Business Impact

4. Technical Scope

5. Detection and Response Summary

6. Evidence Sources

7. Incident Timeline

8. Initial Access Vector

9. Attack Path

10. Blast Radius

11. Root Causes

12. Contributing Factors

13. Control Effectiveness

14. Corrective Actions

15. Preventive Actions

16. Remediation Owners and Deadlines

17. Validation Plan

18. Residual Risk

19. Lessons Learned

20. Appendices

Executive Summary

The executive summary should explain:

What happened
Which business service was affected
Confirmed impact
How the incident was contained
Primary root causes
Highest-priority actions

Avoid unnecessary technical detail in the executive section.

Technical Summary

The technical summary should include:

Affected cluster and workload
Initial access
Attacker actions
Identity abuse
Network activity
Data access
Node impact
Detection sources
Containment actions

Business Impact Assessment

Assess:

Service downtime
Customer impact
Data exposure
Financial loss
Regulatory implications
Reputation risk
Recovery cost
Operational disruption

Root Cause Statement Template

The incident occurred because:

[Underlying condition]

which allowed:

[Attacker capability]

and was not prevented or sufficiently limited because:

[Control failure]

The impact increased because:

[Contributing factors]

Example Root Cause Statement

The incident occurred because the production application contained a known remotely exploitable dependency that was permitted through a report-only vulnerability scanning process.

This allowed an external attacker to execute commands inside the application container.

The attack was not sufficiently limited because the workload used a broadly privileged Service Account and the namespace did not enforce egress restrictions.

The impact increased because the Pod automatically mounted a Kubernetes token and the associated IAM role could access several production secrets.

Control Effectiveness Matrix

Control Domain	Control	Result	Improvement
Application	Dependency scanning	Failed	Enforce critical vulnerability gate
Supply chain	Image signing	Effective	Retain
Workload	Non-root execution	Missing	Enforce Restricted policy
Identity	Service Account RBAC	Failed	Application-specific Role
Network	Default-deny egress	Missing	Apply namespace baseline
Detection	Falco shell detection	Effective	Add process-tree context
Response	Pod quarantine	Partially effective	Automate owner identification
Recovery	Trusted image rebuild	Effective	Retain

RCA Action Register

ID	Action	Owner	Priority	Due Date	Validation
RCA-01	Enforce critical vulnerability deployment gate	DevSecOps	Critical	Defined date	Pipeline test
RCA-02	Replace shared Service Account Role	Platform Team	High	Defined date	RBAC review
RCA-03	Deploy default-deny egress policies	Network Security	High	Defined date	Connectivity test
RCA-04	Disable unnecessary token mounting	Application Team	Medium	Defined date	Manifest validation
RCA-05	Update runtime rule with downloader detection	SOC	Medium	Defined date	Attack simulation

Residual Risk

Not every risk can be eliminated immediately.

Document:

Remaining weakness
Business justification
Compensating controls
Risk owner
Approval
Review date
Expiry date

Residual risk should not be hidden inside technical notes.

RCA Metrics

Organisations can measure RCA effectiveness using:

Metric	Purpose
Time to complete RCA	Measures investigation efficiency
Percentage of actions completed on time	Measures accountability
Repeated incidents from same root cause	Measures remediation quality
Number of enterprise-wide findings	Measures systemic risk
Time to validate remediation	Measures closure effectiveness
Percentage of actions with assigned owners	Measures governance
Detection improvements implemented	Measures security learning

Common Root Cause Analysis Mistakes

Stopping at the Immediate Cause

Problem: The report states that malware caused the incident.

Why It Fails: It does not explain how malware entered the environment.

Better Approach: Trace the cause back to the failed security and operational conditions.

Blaming an Individual

Problem: The RCA concludes that an engineer made an error.

Why It Fails: It ignores why the system allowed one action to create major risk.

Better Approach: Identify missing validation, approvals, automation or guardrails.

Assuming One Root Cause

Problem: The investigation selects one cause for a complex attack.

Why It Fails: Kubernetes incidents often involve several control failures.

Better Approach: Document root causes and contributing factors separately.

Writing Conclusions Before Reviewing Evidence

Problem: Early assumptions drive the investigation.

Why It Fails: Contradictory evidence may be ignored.

Better Approach: Maintain hypotheses and validate them systematically.

Ignoring Pre-Incident Changes

Problem: Analysis begins only when the alert was generated.

Why It Fails: The weakness may have been introduced during an earlier release or configuration change.

Better Approach: Include pre-incident activity in the timeline.

Ignoring Business Impact

Problem: The report contains only technical findings.

Why It Fails: Leadership cannot properly prioritise remediation.

Better Approach: Connect technical findings to business services and data.

Vague Remediation

Problem: Actions say “improve monitoring” or “review access.”

Why It Fails: Completion cannot be measured.

Better Approach: Define specific owners, deadlines and validation methods.

Closing Without Validation

Problem: Actions are marked complete when configurations are changed.

Why It Fails: The original attack path may still work.

Better Approach: Test the control against the original failure scenario.

Limiting Review to One Cluster

Problem: The affected cluster is fixed, but shared templates remain unchanged.

Why It Fails: The same issue may exist across the enterprise.

Better Approach: Search for the root cause across all clusters and accounts.

Treating RCA as a Compliance Document

Problem: The report is produced only to satisfy audit requirements.

Why It Fails: No meaningful security improvement occurs.

Better Approach: Track remediation until validated closure.

Enterprise Root Cause Analysis Workflow

Incident Contained

↓

RCA Owner Assigned

↓

Evidence Sources Confirmed

↓

Unified Timeline Created

↓

Initial Access Identified

↓

Attack Path Reconstructed

↓

Blast Radius Determined

↓

Control Failures Assessed

↓

Root Causes and Contributing Factors Documented

↓

Corrective and Preventive Actions Assigned

↓

Enterprise-Wide Exposure Searched

↓

Remediation Implemented

↓

Security Validation Completed

↓

RCA Formally Closed

Enterprise Best Practices

As a Cloud Security Engineer:

Begin RCA only after critical evidence has been preserved.
Build a unified UTC incident timeline.
Separate confirmed facts, inferences, assumptions and unknowns.
Identify the initial access vector.
Reconstruct the full attack path.
Assess technical and business blast radius.
Review identity, network, workload, runtime and supply-chain controls.
Distinguish symptoms, immediate causes, contributing factors and root causes.
Use structured techniques such as the Five Whys and barrier analysis.
Avoid blame-focused conclusions.
Document multiple root causes where appropriate.
Create specific, measurable remediation actions.
Assign an accountable owner to every action.
Set risk-based deadlines.
Define validation methods before closing actions.
Search all clusters for the same underlying weakness.
Convert findings into preventive controls and detection rules.
Update runbooks, policies and architecture standards.
Document residual risk explicitly.
Share relevant lessons across technical and leadership teams.
Track remediation until evidence-based closure.
Measure whether similar incidents recur.

Real-World Scenario

A financial organisation operates a customer payment platform on Amazon EKS.

Falco detects a shell launched inside the payment API container.

The investigation confirms:

A vulnerable application dependency allowed remote code execution.
The attacker downloaded malware into /tmp.
The Pod ran as root with a writable filesystem.
The Service Account could list several Secrets.
The workload IAM role could retrieve multiple Secrets Manager values.
The namespace had no default-deny egress policy.
The attacker connected to an external command-and-control address.
Runtime detection generated an alert within two minutes.
SOC triage began six minutes later.
The Pod was isolated before confirmed data exfiltration.

The RCA team creates a unified timeline using:

Load balancer logs
Application logs
Falco alerts
Kubernetes Audit Logs
CloudTrail
VPC Flow Logs
Container forensic evidence
CI/CD records

The Five Whys analysis determines that the vulnerable application reached production because:

The application contained an exploitable dependency.
The dependency scanner identified the vulnerability.
The pipeline did not block the release.
The scanner was configured in report-only mode.
No mandatory production vulnerability gate or accountable exception owner had been defined.

The RCA identifies four root causes:

No enforced production vulnerability gate.
A shared, excessively privileged Service Account Role.
No mandatory namespace egress baseline.
No governance process for time-limited security exceptions.

Contributing factors include:

Root container execution
Writable root filesystem
Automatic Service Account token mounting
Broad workload IAM access
Delayed alert enrichment

The organisation implements the following actions:

Enforces vulnerability gates for exploitable critical findings.
Requires documented, expiring risk exceptions.
Replaces the shared Service Account Role with application-specific RBAC.
Restricts the workload IAM role to one secret.
Applies Restricted Pod Security requirements.
Disables unnecessary Service Account token mounting.
Deploys default-deny ingress and egress policies.
Adds central egress filtering.
Improves Falco alert context.
Tests the original attack path in an isolated environment.
Searches every production cluster for the same vulnerable dependency and RBAC template.
Updates the enterprise EKS security baseline.

The RCA is closed only after the original attack sequence is blocked during controlled validation.

Key Takeaways

Root Cause Analysis determines why an incident was possible, not only what happened.
Symptoms, immediate causes, contributing factors and root causes are different.
A unified timeline is essential for reliable RCA.
Initial access must be distinguished from later attacker actions.
Kubernetes incidents often have multiple root causes.
Identity, networking, supply chain, workload security and governance should all be reviewed.
Root-cause statements must be evidence-based and actionable.
Corrective actions address the specific incident.
Preventive actions reduce recurrence across the enterprise.
Remediation requires owners, deadlines and validation criteria.
The affected cluster is not the only environment that should be reviewed.
RCA is complete only when remediation has been tested and the residual risk is understood.

Knowledge Check

1. What is the difference between an immediate cause and a root cause?

Answer: An immediate cause is the direct event that produced the incident, while a root cause is the underlying technical, procedural or organisational condition that allowed the event to occur.

2. Why should a Kubernetes RCA include pre-incident activity?

Answer: The weakness may have been introduced before the attack through a deployment, IAM change, policy exception, network change or pipeline modification.

3. What is the purpose of the Five Whys technique?

Answer: The Five Whys repeatedly asks why an event occurred to move beyond the visible symptom and identify deeper underlying causes.

4. Why should blast radius include both confirmed and potential impact?

Answer: Confirmed impact shows what evidence proves occurred, while potential impact identifies what the attacker had permission or connectivity to affect.

5. When should an RCA remediation action be considered complete?

Answer: It should be considered complete only after implementation, successful validation, evidence collection, ownership confirmation and documentation of any remaining residual risk.

What’s Next?

In the next lesson, we will explore Enterprise Incident Investigation, where you will combine Kubernetes, container, node, memory, malware, identity, AWS and network evidence into a coordinated enterprise investigation.

➡️ Next Lesson: Lesson 10 — Enterprise Incident Investigation