Runbook 03 — Kubernetes Software Supply Chain Investigation

Runbook Information

Item	Details
Runbook ID	K8S-IMAGE-RUNBOOK-03
Category	Software Supply Chain Security Investigation
Runbook Type	Incident Investigation & Threat Hunting
Severity	High / Critical
Primary Team	Kubernetes Security Operations
Supporting Teams	SOC, DevSecOps, Platform Engineering, Cloud Security, Digital Forensics, Incident Response
Estimated Duration	4–24 Hours
Review Frequency	After Every Supply Chain Incident
Compliance Alignment	NIST SP 800-61, NIST SP 800-190, NIST SSDF, SLSA, CIS Kubernetes Benchmark, MITRE ATT&CK

Executive Summary

Software supply chain attacks target the trust relationship between developers, build systems, registries and Kubernetes workloads.

Unlike traditional attacks that exploit running applications, supply chain attacks compromise software before deployment.

Examples include:

Compromised source code
Malicious CI/CD pipelines
Dependency poisoning
Container image tampering
Registry compromise
Malicious image signing
SBOM manipulation
Build provenance manipulation
Image replacement attacks
Image tag hijacking

This runbook provides an enterprise investigation methodology for identifying, containing, analysing and recovering from Kubernetes software supply chain compromises.

Enterprise Scenario

CloudNova Technologies receives an alert from its SIEM indicating that a production workload has started communicating with an external command-and-control server.

Initial investigation reveals:

The deployed image passed vulnerability scanning.
The image originated from the production registry.
The deployment used a valid image tag.
The running image digest differs from the approved release documentation.
Registry logs show an unexpected image push outside the approved deployment window.
The image signature cannot be verified using the trusted signing key.
A CI/CD service account recently authenticated from an unknown IP address.

The CISO activates the Supply Chain Incident Response Team to determine:

What was compromised?
How the malicious image entered production.
Whether other workloads are affected.
Whether signing keys or CI/CD credentials have been compromised.
Whether software provenance can still be trusted.

Investigation Objectives

Determine:

Initial compromise point
Scope of compromise
Affected images
Affected registries
Affected Kubernetes clusters
Image integrity
Registry integrity
Build pipeline integrity
Signing-key integrity
CI/CD compromise
Software provenance
Root cause
Business impact
Recovery strategy

Enterprise Supply Chain Architecture

Developer

      │

      ▼

Source Repository

      │

      ▼

CI/CD Pipeline

      │

      ▼

Container Build

      │

      ▼

Security Scan

      │

      ▼

SBOM

      │

      ▼

Image Signing

      │

      ▼

Private Registry

      │

      ▼

Admission Controller

      │

      ▼

Kubernetes Cluster

      │

      ▼

Runtime Monitoring

Investigation Workflow

Incident Detection

        │

        ▼

Incident Validation

        │

        ▼

Containment

        │

        ▼

Evidence Preservation

        │

        ▼

Supply Chain Analysis

        │

        ▼

Impact Analysis

        │

        ▼

Eradication

        │

        ▼

Recovery

        │

        ▼

Lessons Learned

Investigation Scope

Review:

Source Code Repository
Git Commit History
CI/CD Pipelines
Build Servers
Build Artifacts
Container Images
Registry
Image Signatures
SBOM
Attestations
Kubernetes Deployments
Running Pods
Admission Logs
Registry Logs
Cloud IAM
Network Activity

Required Tools

Tool	Purpose
kubectl	Kubernetes Investigation
Docker	Image Analysis
Trivy	Vulnerability Review
Cosign	Signature Verification
Syft	SBOM Analysis
jq	JSON Parsing
Git	Source History
Cloud CLI	Registry Review
SIEM	Event Correlation

Phase 1 — Incident Validation

Collect:

Detection source
Detection time
Alert ID
Reporting system
Severity
Business impact

Determine:

False positive
Confirmed incident
Ongoing attack

Phase 2 — Initial Containment

Immediately:

Pause deployments
Freeze production releases
Disable automated promotions
Preserve CI/CD pipelines
Preserve registries
Preserve logs

Do NOT:

Delete images
Restart clusters
Remove evidence
Rotate keys before preserving evidence

Phase 3 — Preserve Evidence

Collect:

Image digests
Deployment manifests
Registry metadata
Image signatures
Build logs
Admission logs
Audit logs
CI/CD logs
IAM logs

Record SHA256 hashes for exported evidence.

Phase 4 — Identify Affected Images

Inventory running images.

kubectl get pods -A \
-o=jsonpath="{..image}"

Identify:

Registry
Repository
Digest
Tag
Namespace

Compare against:

Approved release inventory
Registry inventory
CMDB

Phase 5 — Validate Image Integrity

Review.

docker image inspect

Compare:

Digest
Layers
Metadata
Labels

Determine whether:

Image changed
Image replaced
Image rebuilt

Phase 6 — Verify Image Signatures

Verify every production image.

cosign verify

Review.

Signature
Signing Identity
Signing Key
Certificate
Issuer

Investigate:

Failed verification
Unknown keys
Revoked certificates

Phase 7 — Review Build Provenance

Review.

Build ID
Pipeline
Build Time
Source Commit
Builder Identity

Validate.

Build matches release documentation.

Phase 8 — Review SBOM

Compare.

Approved SBOM

Running Image

Identify.

New Packages
Missing Packages
Version Changes
Unknown Components

Phase 9 — Review CI/CD Pipeline

Investigate.

Recent Changes
Pipeline Configuration
Secrets
Variables
Service Accounts
Build Agents

Determine.

Unauthorised modifications.

Phase 10 — Review Source Repository

Review.

Recent Commits
Branch Protection
Pull Requests
Reviews
Force Pushes

Investigate.

Suspicious commits.

Phase 11 — Dependency Investigation

Review.

Dependencies.

Python
NodeJS
Java
Go
Rust

Look for.

Dependency Confusion
Typosquatting
Malicious Packages

Phase 12 — Registry Investigation

Review.

Image Push
Image Delete
Tag Changes
Permission Changes
Login Events

Determine.

Unauthorised registry activity.

Phase 13 — IAM Investigation

Review.

Service Accounts
CI/CD Roles
Registry Roles
Kubernetes Pull Roles

Investigate.

New Permissions
Privilege Escalation
Credential Theft

Phase 14 — Admission Controller Investigation

Review.

Admission logs.

Determine.

Policy bypass
Audit mode
Disabled enforcement
Exceptions

Phase 15 — Runtime Investigation

Review.

Running Pods.

kubectl get pods -A

Investigate.

Image Digest
Runtime User
Network Connections
File Changes

Phase 16 — Network Investigation

Review.

Outbound Traffic.

Identify.

Unknown Domains
Command & Control
Data Exfiltration
Cryptocurrency Mining

Phase 17 — Timeline Reconstruction

Build timeline.

Include.

Code Commit
Pipeline Run
Image Build
Registry Push
Image Sign
Deployment
Alert
Investigation

Phase 18 — Scope Assessment

Determine.

Affected.

Applications
Clusters
Registries
Developers
Pipelines
Customers

Phase 19 — Root Cause Analysis

Determine.

Primary Cause.

Examples.

Credential Theft
Registry Compromise
CI/CD Attack
Insider Threat
Dependency Poisoning

Phase 20 — Eradication

Perform.

Remove malicious images
Remove compromised registry tags
Revoke compromised credentials
Rotate signing keys
Patch CI/CD
Remove malicious dependencies

Phase 21 — Recovery

Deploy.

Trusted Images
Verified Signatures
New SBOM
New Build
New Registry Digest

Validate.

Admission Policies.

Phase 22 — Post-Incident Review

Review.

Detection Time
Response Time
Recovery Time
Lessons Learned

Update.

Policies.

Enterprise Investigation Checklist

Control	Status
Evidence Preserved	☐
Images Identified	☐
Registry Reviewed	☐
CI/CD Reviewed	☐
Source Code Reviewed	☐
Dependencies Reviewed	☐
SBOM Validated	☐
Signatures Verified	☐
IAM Reviewed	☐
Admission Logs Reviewed	☐
Runtime Images Validated	☐
Timeline Created	☐
Root Cause Identified	☐
Recovery Completed	☐
Lessons Learned	☐

Indicators of Compromise (IOCs)

Investigate for:

Registry

Unexpected image pushes
Image deletions
Tag modifications
New repositories
Public repository exposure

CI/CD

Unknown pipeline executions
Modified build scripts
Unexpected build agents
Secret access
New service accounts

Container Images

Digest mismatch
Signature verification failure
Embedded malware
Unexpected binaries
Modified layers

Kubernetes

Deployment outside change window
New namespaces
Unknown workloads
Privileged Pods
Images from untrusted registries

Identity

Login from unusual locations
MFA failures
New API keys
Privilege escalation
Service account abuse

Risk Classification

Critical

Malicious production image
Compromised signing key
Registry compromise
CI/CD compromise
Software provenance lost

High

Image signature failure
Dependency poisoning
Image replacement
Admission bypass
Build compromise

Medium

Missing SBOM
Missing provenance
Weak monitoring

Low

Documentation gaps
Metadata inconsistencies

Enterprise Investigation Matrix

Area	Rating
Source Code Integrity
CI/CD Security
Registry Security
Image Integrity
Image Signing
SBOM
Provenance
Kubernetes Deployment
Runtime Validation
Detection Capability

Ratings:

Effective
Partially Effective
Ineffective

Evidence Collection

Collect:

Source Repository History
Pull Requests
Build Logs
Pipeline Configuration
Registry Logs
Image Digests
Image Signatures
SBOM Files
Admission Logs
Kubernetes Events
IAM Logs
Network Logs
SIEM Timeline
Timeline Reconstruction
Root Cause Analysis

Incident Report Template

Incident Name:

Kubernetes Software Supply Chain Investigation

Incident ID:

Severity:

Detection Time:

Reported By:

Affected Clusters:

Affected Applications:

Affected Images:

Affected Registries:

Compromised CI/CD:

Compromised Credentials:

Compromised Signing Keys:

Supply Chain Stage:

Root Cause:

Containment Actions:

Recovery Actions:

Evidence Collected:

Business Impact:

Customer Impact:

Lessons Learned:

Preventive Controls:

Residual Risk:

Executive Approval:

Closed By:

Success Criteria

The investigation is successful when:

The initial compromise point is identified.
Evidence integrity is maintained.
All affected images are identified.
Registry integrity is verified.
CI/CD compromise is confirmed or ruled out.
Image signatures are validated.
Root cause is determined.
Recovery is completed using trusted artifacts.
Supply chain controls are strengthened.
Executive report is completed.

Operational Best Practices

Daily

Monitor image-signature verification failures
Review unexpected registry activity
Monitor CI/CD pipeline executions
Review admission-policy violations

Weekly

Audit registry access
Review signing-key usage
Validate build provenance
Compare production images with approved release inventory

Monthly

Conduct a complete software supply chain assessment
Rotate CI/CD credentials
Review SBOM inventory
Test image-signature verification
Validate registry disaster recovery

Quarterly

Rotate image-signing keys
Conduct software supply chain tabletop exercises
Test compromised registry recovery
Validate incident response procedures
Review SLSA maturity

Runbook Summary

This runbook provides a comprehensive enterprise methodology for investigating Kubernetes software supply chain incidents from initial detection through containment, forensic analysis, recovery, and post-incident improvement.

It guides security teams through every stage of the software delivery lifecycle—including source-code repositories, CI/CD pipelines, dependency management, container image creation, vulnerability scanning, SBOM validation, image signing, registry governance, admission control, and Kubernetes runtime verification—to identify how malicious software entered the environment.

By following this runbook, organisations can rapidly determine the origin and scope of a supply chain compromise, preserve critical forensic evidence, restore trusted software artefacts, and strengthen preventative controls to reduce the likelihood of future software supply chain attacks.

What’s Next?

Next Module: Kubernetes Secrets Management & Data Protection

In the next module, you will learn how to securely manage Kubernetes Secrets, implement encryption at rest, integrate external secret-management platforms such as HashiCorp Vault and cloud-native secret stores, protect sensitive application data, and enforce enterprise data protection controls across Kubernetes workloads.