Lab 06 — Runtime Security

In the previous Kubernetes labs, you built several preventive security layers:

Lab 01 — Kubernetes Fundamentals
        ↓
Understand Resources

Lab 02 — Kubernetes RBAC
        ↓
Control Identity and Permissions

Lab 03 — Kyverno
        ↓
Enforce Kubernetes-Native Policy

Lab 04 — Network Policies
        ↓
Control Communication

Lab 05 — OPA Gatekeeper
        ↓
Enforce Admission Guardrails

These controls help prevent insecure configurations from reaching production.

But there is another critical question:

What happens after
the container starts running?

A workload can be securely configured and still become compromised through:

Application Vulnerability

Stolen Credentials

Malicious Dependency

Compromised Image

Insider Activity

Runtime Exploitation

This is where runtime security becomes essential.

Mission Information

Difficulty: Intermediate to Advanced

Estimated Time: 90–120 minutes

Primary Skills:

Kubernetes Runtime Security

Container Behavior Monitoring

Process Monitoring

Runtime Detection

Falco Concepts

Security Alert Triage

Threat Investigation

Evidence Collection

Containment

Incident Response

Lab Scenario

Your organization operates several production Kubernetes clusters.

Preventive controls are already implemented:

RBAC

NetworkPolicy

Admission Policies

Image Controls

Workload Security Standards

During monitoring, the security team receives an alert indicating that an unexpected shell was started inside an application container.

The application normally runs only:

Web Server Process

but runtime telemetry shows:

/bin/sh

executing inside the container.

Your mission is to investigate the activity and determine:

What Happened?

Which Workload Was Affected?

Which Process Executed?

Who Initiated It?

What Identity Does the Pod Use?

What Could the Workload Access?

Was There Additional Suspicious Activity?

How Should the Incident Be Contained?

Lab Objectives

By the end of this lab, you should be able to:

Explain Kubernetes runtime security
Understand preventive vs detective controls
Establish normal workload behavior
Identify unexpected process execution
Understand Falco runtime detection concepts
Review runtime security events
Investigate unexpected shell activity
Investigate suspicious file access
Investigate unexpected outbound connections
Correlate runtime activity with Kubernetes context
Review service-account exposure
Evaluate workload privileges
Determine potential blast radius
Collect incident evidence
Contain suspicious workloads
Document runtime-security findings
Build a Kubernetes runtime investigation workflow

Part 01 — Preventive vs Detective Security

Preventive controls attempt to stop insecure activity before it occurs.

Examples:

RBAC
      ↓
Restrict API Permissions

Kyverno / Gatekeeper
      ↓
Block Insecure Configuration

NetworkPolicy
      ↓
Restrict Communication

Runtime security provides another layer:

Workload Running
      ↓
Observe Behavior
      ↓
Detect Suspicious Activity
      ↓
Investigate
      ↓
Respond

Security Model

PREVENT
   +
DETECT
   +
RESPOND

A mature Kubernetes security architecture needs all three.

Part 02 — Why Runtime Security Matters

Imagine a correctly configured application:

Non-Privileged

Non-Root

Resource Limits

Approved Image

Restricted Network Access

The application itself contains a vulnerability.

An attacker exploits it.

Internet
   ↓
Application
   ↓
Application Vulnerability
   ↓
Container Compromise

Admission policy cannot stop an exploit occurring after deployment.

Runtime monitoring may detect the attacker’s behavior.

Example Runtime Sequence

Normal Application
      ↓
Application Exploited
      ↓
Unexpected Shell
      ↓
Discovery Commands
      ↓
Credential Access
      ↓
Network Connection
      ↓
Lateral Movement Attempt

Each step can generate observable signals.

Part 03 — Runtime Security Questions

When monitoring Kubernetes workloads, ask:

Which Processes Are Running?

Which Files Are Being Accessed?

Which Network Connections Are Created?

Which Commands Are Executed?

Which User Is Executing Them?

Which Container Is Involved?

Which Pod?

Which Namespace?

Which Service Account?

Which Node?

Context turns:

Suspicious Process

into:

Actionable Security Incident

Part 04 — Runtime Security Architecture

A simplified architecture is:

Kubernetes Workloads
        ↓
Runtime Activity
        ↓
Runtime Security Sensor
        ↓
Detection Rules
        ↓
Security Events
        ↓
Alert Pipeline
        ↓
SOC / SIEM
        ↓
Investigation

Part 05 — Runtime Signals

Important signals include:

Process Execution

File Access

Network Connections

Container Activity

Privilege Changes

Namespace Activity

System Calls

Kubernetes API Activity

Different runtime-security platforms collect these signals differently.

Part 06 — Understand Falco

Falco is commonly used in cloud-native environments for runtime threat detection.

Conceptually:

Runtime Activity
      ↓
Falco
      ↓
Detection Rules
      ↓
Security Event

Falco rules can identify behavior such as:

Shell Started in Container

Sensitive File Access

Unexpected Process Execution

Package Management Activity

Suspicious Network Tools

Privilege-Related Activity

Important

Falco is primarily:

Detection

rather than:

Admission Prevention

Compare:

Gatekeeper
      ↓
Should this workload be deployed?

with:

Falco
      ↓
What is this workload doing now?

Part 07 — Lab Environment

Use only:

Your Own Kubernetes Cluster

Dedicated Training Environment

Explicitly Authorized Lab

You need:

kubectl

Training Namespace

Runtime Security Visibility

Falco or Equivalent Lab Runtime Sensor

If Falco is not installed, the investigation sections can still be performed using Kubernetes evidence and provided runtime-event scenarios.

Part 08 — Verify Kubernetes Context

Run:

kubectl config current-context

Then:

kubectl cluster-info

Record:

Cluster:

Context:

Environment:

Authorization:

Always verify the target before conducting security testing.

Part 09 — Create the Training Namespace

Create:

kubectl create namespace ghc-runtime-lab

Set your working namespace:

kubectl config set-context --current --namespace=ghc-runtime-lab

Verify:

kubectl config view --minify

Part 10 — Deploy the Training Application

Create:

runtime-app.yaml

Add:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: runtime-app
  namespace: ghc-runtime-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: runtime-app
  template:
    metadata:
      labels:
        app: runtime-app
    spec:
      containers:
        - name: web
          image: nginx
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: "50m"
              memory: "32Mi"
            limits:
              cpu: "200m"
              memory: "128Mi"

Apply:

kubectl apply -f runtime-app.yaml

Verify:

kubectl get pods

Then:

kubectl get deployment runtime-app

Part 11 — Inspect the Workload

Run:

kubectl describe deployment runtime-app

Then inspect the Pod:

kubectl get pods -l app=runtime-app

Record:

Pod Name:

Namespace:

Container:

Image:

Node:

Service Account:

Pod IP:

Restart Count:

Why Context Matters

A runtime alert stating:

Shell Executed

is incomplete.

You need:

Shell Executed
      ↓
Which Container?
      ↓
Which Pod?
      ↓
Which Namespace?
      ↓
Which Application?
      ↓
Which Identity?

Part 12 — Establish the Normal Baseline

Before detecting abnormal behavior, understand normal behavior.

For the training web application, expected activity may include:

Web Server Process

Normal Configuration Reads

Expected Log Writes

Expected Network Connections

Unexpected activity may include:

Interactive Shell

Package Installation

Credential Discovery

Network Scanning

Unexpected Downloads

Sensitive File Modification

Baseline Template

Application:
runtime-app

Expected Primary Process:
Web Server

Expected User:

Expected Network Destinations:

Expected Writable Locations:

Expected Child Processes:

Expected Administrative Activity:
Minimal

Part 13 — Process Baseline

Inspect running processes inside your own lab container:

kubectl exec <runtime-app-pod> -- ps

Available process tools depend on the image.

Record the expected process tree.

Conceptually:

Container
   ↓
Main Application Process
   ↓
Expected Worker Processes

Runtime Principle

You cannot reliably identify abnormal behavior unless you understand:

Normal Behavior

Part 14 — Review Container Logs

Run:

kubectl logs <runtime-app-pod>

If multiple containers exist:

kubectl logs <pod-name> -c <container-name>

Record:

Normal Log Pattern

Normal Requests

Errors

Unexpected Events

Part 15 — Review Kubernetes Events

Run:

kubectl get events --sort-by=.metadata.creationTimestamp

Events may reveal:

Scheduling

Image Pulls

Container Starts

Restarts

Probe Failures

Resource Problems

Events are useful context but should not be treated as a complete runtime security log.

Part 16 — Verify Falco Availability

If Falco is installed in your lab environment, identify its namespace and workloads.

For example:

kubectl get pods -A | grep -i falco

Then inspect the appropriate namespace:

kubectl get pods -n <falco-namespace>

What You Are Looking For

Runtime Sensor Healthy

Detection Components Running

Events Being Generated

Exact deployment architecture varies.

Do not assume fixed component names.

Part 17 — Understand Falco Rules

A Falco-style detection rule conceptually contains:

Rule Name

Condition

Output

Priority

Tags

Example logic:

IF
Shell Process Starts

AND
Process Is Inside Container

THEN
Generate Alert

Detection Model

Runtime Event
      ↓
Condition Match
      ↓
Alert
      ↓
Context

Part 18 — Safe Shell Detection Exercise

In your own isolated lab workload, deliberately open a shell:

kubectl exec -it <runtime-app-pod> -- /bin/sh

Inside the container, perform only harmless inspection such as:

whoami

Then:

pwd

Then exit:

exit

Security Purpose

The goal is not exploitation.

You are generating a controlled runtime event:

Shell Started Inside Container

so you can observe and investigate it.

Part 19 — Why a Shell Can Be Suspicious

Many production containers are designed to run:

One Application

They do not normally require interactive shell activity.

Therefore:

Web Server Container
      ↓
Interactive Shell

can be a valuable detection signal.

Important

A shell does not automatically prove compromise.

Legitimate reasons may include:

Administrator Troubleshooting

SRE Investigation

Approved Maintenance

Therefore the correct response is:

Detect
   ↓
Validate
   ↓
Investigate Context

not:

Shell = Attacker

Part 20 — Review Runtime Alert

If your runtime sensor detects the event, capture:

Timestamp

Rule Name

Priority

Namespace

Pod

Container

Process

User

Parent Process

Node

Alert Triage Template

Alert:

Timestamp:

Severity:

Cluster:

Namespace:

Pod:

Container:

Process:

Parent Process:

User:

Node:

Initial Assessment:

Part 21 — Correlate the Pod

Once the alert identifies a Pod:

kubectl get pod <pod-name> -o wide

Then:

kubectl describe pod <pod-name>

Record:

Node

Pod IP

Image

Service Account

Container State

Restart Count

Labels

Investigation Question

Ask:

Is this the workload
we expected the alert from?

Part 22 — Review Pod Security Context

Inspect:

kubectl get pod <pod-name> -o yaml

Review:

securityContext

runAsUser

runAsNonRoot

privileged

allowPrivilegeEscalation

capabilities

readOnlyRootFilesystem

Security Questions

Is the Container Privileged?

Is It Running as Root?

Can It Escalate Privileges?

Does It Have Extra Capabilities?

Can It Modify Its Root Filesystem?

Part 23 — Review Service Account

Identify:

kubectl get pod <pod-name> -o jsonpath='{.spec.serviceAccountName}'

Then inspect the ServiceAccount:

kubectl get serviceaccount <service-account-name>

Why This Matters

A compromised workload may attempt to use:

Workload Identity

to interact with Kubernetes APIs.

Attack Path

Application Compromise
      ↓
Container Access
      ↓
Service Account Credential
      ↓
Kubernetes API
      ↓
Authorized Actions

Part 24 — Review Effective Permissions

Where authorized, use:

kubectl auth can-i --list --as=system:serviceaccount:ghc-runtime-lab:<service-account-name>

Review the output carefully.

Ask:

Can It Read Secrets?

Can It Create Pods?

Can It Create Jobs?

Can It Modify Deployments?

Can It Access Other Namespaces?

Security Principle

Runtime compromise severity depends partly on:

Permissions Available
After Compromise

Part 25 — Blast Radius Analysis

Think:

Compromised Pod
      ↓
Service Account
      ↓
RBAC Permissions
      ↓
Network Reachability
      ↓
Secrets
      ↓
Other Workloads

The investigation must evaluate all of these.

Blast Radius Template

Affected Pod:

Service Account:

RBAC Access:

Reachable Services:

Accessible Secrets:

Mounted Volumes:

Host Access:

Potential Impact:

Part 26 — Review Mounted Volumes

Inspect:

kubectl describe pod <pod-name>

Look for:

Volumes

Mounts

Secrets

ConfigMaps

Persistent Storage

HostPath

Why Volumes Matter

A compromised container may gain access to:

Application Data

Configuration

Credentials

Shared Storage

through mounted volumes.

Part 27 — Review Environment Configuration

Inspect the workload specification.

Look for:

Environment Variables

Secret References

ConfigMap References

Do not print or copy actual production secret values into investigation notes.

Record:

Secret Reference Exists

rather than unnecessarily exposing the secret itself.

Part 28 — Sensitive File Access

Runtime tools can detect suspicious access to sensitive files.

Examples may include:

Credential Files

System Account Files

Service Account Credentials

Shell Configuration

Sensitive Application Configuration

Detection Model

Unexpected Process
      ↓
Sensitive File Access
      ↓
Runtime Alert

Investigation Question

Ask:

Does this application
normally access this file?

Context determines severity.

Part 29 — Safe Sensitive-File Exercise

In your isolated training container, inspect a harmless system identity file:

kubectl exec <runtime-app-pod> -- cat /etc/passwd

This may trigger a runtime rule depending on your sensor configuration.

The purpose is to observe:

File Access Event

not to obtain credentials.

Part 30 — Package Management Detection

Production containers generally should not require software installation after deployment.

Unexpected package-manager execution can indicate:

Interactive Modification

Troubleshooting

Attacker Tool Installation

Runtime detection may monitor commands associated with package management.

Better Container Model

Build Required Software
      ↓
Create Image
      ↓
Scan
      ↓
Deploy Immutable Image

rather than:

Deploy Container
      ↓
Install Tools at Runtime

Part 31 — Unexpected Network Tools

Security monitoring may flag unexpected use of diagnostic or network utilities inside application containers.

The key signal is not simply:

Tool Exists

but:

Application Does Not Normally
Execute This Tool

Behavioral Security

Expected Behavior
      ↓
Baseline
      ↓
Deviation
      ↓
Investigation

Part 32 — Outbound Connection Investigation

Suppose runtime monitoring reports:

Unexpected Outbound Connection

Investigate:

Source Pod

Source Process

Destination

Port

Protocol

Timestamp

Investigation Flow

Network Alert
   ↓
Identify Pod
   ↓
Identify Process
   ↓
Identify Destination
   ↓
Compare With Baseline
   ↓
Determine Legitimacy

Part 33 — NetworkPolicy Correlation

Review policies:

kubectl get networkpolicy

Then:

kubectl describe networkpolicy <policy-name>

Ask:

Should this workload
have been able to make
that connection?

Security Finding

If unexpected outbound traffic was possible:

Runtime Detection
      ↓
Reveals Egress Gap
      ↓
NetworkPolicy Improvement

Runtime security can therefore improve preventive controls.

Part 34 — Unexpected Process Investigation

Suppose an alert shows:

/bin/sh

followed by several unexpected processes.

Create a timeline:

10:14:02
Application Request

10:14:05
Shell Started

10:14:08
System Discovery

10:14:15
Sensitive File Access

10:14:30
Unexpected Network Connection

Why Timelines Matter

Individual alerts may appear unrelated.

A timeline reveals:

Attack Sequence

Part 35 — Runtime Investigation Timeline

Use:

Timestamp | Source | Event | Resource | Assessment

Example:

Time	Source	Event	Assessment
10:14:05	Runtime	Shell started	Suspicious
10:14:08	Runtime	Unexpected process	Suspicious
10:14:15	Runtime	Sensitive file read	Investigate
10:14:30	Network	External connection	High priority

Part 36 — Correlate Kubernetes Audit Activity

If Kubernetes audit logs are available in your lab, search around the runtime-event timestamp.

Look for actions such as:

pods/exec

Pod Creation

Secret Access

RoleBinding Changes

Deployment Changes

Critical Correlation

A shell alert plus:

pods/exec

may indicate an administrator intentionally opened the shell.

A shell alert without corresponding legitimate administrative activity may deserve greater investigation.

Part 37 — kubectl exec as Security Context

When an administrator runs:

kubectl exec

there are two important evidence sources:

Kubernetes API Activity
      +
Container Runtime Activity

Correlation helps answer:

Who Initiated the Shell?

Part 38 — Alert Context Enrichment

A raw alert might say:

Shell in Container

Enrich it with:

Cluster

Namespace

Application

Pod

Container Image

Node

Service Account

Owner

Environment

Network Exposure

Now the alert becomes much more actionable.

Part 39 — Severity Assessment

Consider:

Process Risk

Workload Exposure

Privileges

Identity Permissions

Data Sensitivity

Network Reachability

Evidence of Follow-On Activity

Example

Shell in Development Test Pod
+
No Sensitive Access
+
Known Administrator

may have different severity from:

Shell in Internet-Facing Production Pod
+
Powerful Service Account
+
Secret Access
+
Unexpected External Connection

Part 40 — Runtime Risk Matrix

Signal	Context	Potential Risk
Interactive shell	Web container	Medium–High
Privilege change	Restricted workload	High
Sensitive credential access	Unexpected process	High
Unknown outbound connection	Internet-facing workload	High
Package installation	Immutable production container	Medium–High
Unexpected process	Unknown parent	Investigate

Severity should always be adjusted for environment context.

Part 41 — Container Restart Investigation

Check:

kubectl get pods

Review:

RESTARTS

Then:

kubectl describe pod <pod-name>

Look for:

Previous State

Exit Code

Reason

Events

Unexpected restarts may indicate:

Application Failure

Resource Exhaustion

Crash

Security Activity

They are not automatically malicious.

Part 42 — Node Context

Determine which node hosts the affected Pod:

kubectl get pod <pod-name> -o wide

Why?

Because potential incident scope can include:

Container
   ↓
Pod
   ↓
Node
   ↓
Cluster

If evidence suggests host-level impact, escalation becomes more urgent.

Part 43 — Container Escape Concept

A container escape occurs when activity breaks expected container isolation and gains access to underlying host resources.

Conceptually:

Container Compromise
      ↓
Isolation Boundary Bypassed
      ↓
Node Access

Indicators may include unusual:

Host File Access

Namespace Activity

Privilege Operations

Kernel Interaction

Host Process Interaction

This lab does not attempt a container escape.

The objective is to understand detection and response.

Part 44 — Privileged Workloads and Runtime Risk

A privileged container may dramatically increase potential incident impact.

Check:

privileged: true

If present:

Runtime Alert
      +
Privileged Workload
      ↓
Higher Investigation Priority

Part 45 — HostPath Risk

A workload with:

hostPath

may have access to host filesystem locations.

During investigation determine:

Which Host Path?

Read-Only or Writable?

Why Is It Required?

What Could an Attacker Access?

Part 46 — Host Namespace Risk

Review whether the workload uses:

hostNetwork

hostPID

hostIPC

These configurations can weaken isolation and increase blast radius.

Part 47 — Capabilities Review

Containers can receive Linux capabilities.

Review the security context for added capabilities.

Ask:

Which Capabilities Are Added?

Does the Application Need Them?

Could They Increase Post-Compromise Impact?

Better Model

Drop Unnecessary Capabilities
      ↓
Add Only Explicitly Required Capabilities

Part 48 — Read-Only Root Filesystem

A read-only root filesystem can make some runtime modifications harder.

Review:

readOnlyRootFilesystem

This does not prevent every attack.

It contributes to:

Defense in Depth

Part 49 — Runtime Detection and Admission Policy

Suppose runtime investigations repeatedly identify:

Privileged Workloads

You should not rely only on detection.

Improve preventive policy:

Runtime Finding
      ↓
Security Requirement
      ↓
Kyverno / Gatekeeper
      ↓
Prevent Future Deployment

Security Feedback Loop

Detect
   ↓
Investigate
   ↓
Learn
   ↓
Prevent

Part 50 — Runtime Detection and NetworkPolicy

Suppose investigations repeatedly show:

Unexpected External Connections

Improve:

Egress NetworkPolicy

where technically appropriate.

Feedback Loop

Runtime Alert
      ↓
Network Gap
      ↓
Policy Improvement
      ↓
Reduced Attack Surface

Part 51 — Runtime Detection and RBAC

Suppose a compromised workload uses its ServiceAccount to access unnecessary resources.

Remediation should include:

Reduce RBAC Permissions

not only:

Delete Compromised Pod

Root Cause Thinking

Incident
   ↓
What Allowed Impact?
   ↓
Fix Supporting Control

Part 52 — Runtime Alert Triage Workflow

Use:

01 Receive Alert

02 Validate Alert

03 Identify Cluster

04 Identify Namespace

05 Identify Pod

06 Identify Container

07 Identify Process

08 Review Workload Baseline

09 Review Security Context

10 Review Service Account

11 Review RBAC

12 Review Network Reachability

13 Review Related Events

14 Determine Blast Radius

15 Assign Severity

Part 53 — False Positives

Runtime alerts require context.

Example:

Shell Started

may result from:

Approved Troubleshooting

Therefore verify:

Who?

Why?

When?

Approved?

Expected?

Detection Engineering Principle

Avoid both extremes:

Alert = Incident

and:

Alert = Ignore

Instead:

Alert
  ↓
Context
  ↓
Decision

Part 54 — Alert Tuning

If a rule repeatedly generates legitimate alerts, consider controlled tuning.

Do not simply disable useful detections.

Evaluate:

Rule Condition

Known Workloads

Namespaces

Process Names

Approved Administrative Activity

Goal

Reduce:

Noise

without creating:

Blind Spots

Part 55 — Runtime Rule Design

A useful detection rule should answer:

What Behavior?

Why Suspicious?

Which Workloads?

What Severity?

What Context?

What Response?

Detection Documentation Template

Rule Name:

Behavior:

Data Source:

Condition:

Severity:

Expected False Positives:

Investigation Steps:

Response:

Part 56 — Runtime Security Event Categories

Build detection coverage across categories such as:

Execution

Persistence

Privilege

Credential Access

Discovery

Network Activity

Defense Evasion

Container Escape Indicators

This helps structure a runtime-detection program.

Part 57 — Execution Detection

Examples:

Unexpected Shell

Unexpected Binary

Interpreter Execution

Administrative Tool Execution

Ask:

Does the application normally
execute this process?

Part 58 — Credential Access Detection

Monitor suspicious interaction with:

Service Account Credentials

Application Secrets

Cloud Credentials

Configuration Files

Critical Principle

Never include exposed secret values unnecessarily in incident reports.

Record:

Credential May Have Been Accessed

and rotate it when appropriate.

Part 59 — Discovery Detection

After compromise, attackers often try to understand the environment.

Runtime security may observe unexpected:

Process Discovery

Network Discovery

Identity Discovery

Filesystem Inspection

The important signal is deviation from normal application behavior.

Part 60 — Persistence Thinking

Containers are often ephemeral, but attackers may attempt persistence through Kubernetes resources.

Examples conceptually include unauthorized changes to:

Deployments

Jobs

CronJobs

DaemonSets

RBAC Objects

Runtime telemetry alone may not reveal all of these.

Correlate with:

Kubernetes Audit Logs

Part 61 — Defense Evasion

Suspicious activity may attempt to:

Disable Monitoring

Delete Logs

Modify Security Agents

Alter Policies

Changes affecting:

Falco

Logging

Admission Policies

NetworkPolicy

should receive strong scrutiny.

Part 62 — Evidence Collection

For a runtime incident collect:

Alert

Timestamp

Pod Metadata

Deployment Manifest

Container Image

Container Logs

Kubernetes Events

Service Account

RBAC Information

NetworkPolicy

Runtime Events

Audit Logs

Relevant Node Information

Evidence Principle

Preserve evidence before destructive remediation when feasible and consistent with incident severity.

Part 63 — Evidence Timeline

Create:

T1 — Initial Activity

T2 — Runtime Alert

T3 — Follow-On Process

T4 — Network Activity

T5 — Security Triage

T6 — Containment

T7 — Remediation

Part 64 — Containment Strategy

Once malicious activity is sufficiently validated, containment options may include:

Isolate Workload

Restrict Network Access

Revoke Credentials

Reduce Permissions

Remove Malicious Workload

Scale Down Affected Application

Quarantine Node

The correct response depends on severity.

Important

Do not immediately destroy evidence without considering:

Incident Response Requirements

Forensic Requirements

Business Availability

Part 65 — Network Isolation

One containment option is restricting network communication.

Conceptually:

Compromised Pod
      ↓
Emergency Network Isolation
      ↓
No Unnecessary Communication

This may reduce:

Lateral Movement

Command-and-Control

Exfiltration

Part 66 — Credential Containment

If a credential may have been exposed:

Identify Credential
      ↓
Determine Usage
      ↓
Revoke / Rotate
      ↓
Update Workload
      ↓
Validate

Potential credentials include:

Application Secrets

Service Account Credentials

Cloud Credentials

Database Credentials

Part 67 — RBAC Containment

If the compromised workload has excessive Kubernetes permissions:

Identify RoleBinding
      ↓
Assess Requirement
      ↓
Remove Excessive Permission
      ↓
Validate Application

Part 68 — Workload Replacement

Containers should generally be treated as:

Disposable

rather than manually cleaned and trusted again.

A typical recovery model is:

Identify Root Cause
      ↓
Fix Source / Image / Configuration
      ↓
Build Trusted Image
      ↓
Redeploy

not:

Manually Clean Compromised Container
      ↓
Return It to Production

Part 69 — Image Investigation

Record:

Image Repository

Image Tag

Image Digest

Build Source

Registry

Scan Status

Ask:

Was the image compromised before deployment?

Or was the running application compromised later?

These represent different incident paths.

Part 70 — Supply-Chain Scenario

Possible sequence:

Compromised Source
      ↓
Malicious Build
      ↓
Container Image
      ↓
Registry
      ↓
Deployment
      ↓
Runtime Activity

Runtime detection may be the first place malicious behavior becomes visible.

Part 71 — Application Exploit Scenario

Another sequence:

Trusted Image
      ↓
Vulnerable Application
      ↓
External Request
      ↓
Runtime Exploitation
      ↓
Suspicious Process

This requires different remediation:

Patch Application

rather than only:

Replace Registry Image

Part 72 — Investigation Scenario

Assume the following runtime event:

Rule:
Shell Spawned in Container

Namespace:
ghc-runtime-lab

Pod:
runtime-app-xxxxx

Container:
web

Process:
/bin/sh

Severity:
Warning

Your Investigation

Determine:

1. Is the Pod expected?

2. Is shell activity normal?

3. Who initiated it?

4. Was pods/exec used?

5. What ServiceAccount is attached?

6. What permissions exist?

7. What network access exists?

8. What secrets are mounted?

9. What processes followed?

10. Is containment required?

Part 73 — Incident Decision

After investigation, classify:

Benign

Expected Administrative Activity

Policy Violation

Suspicious

Confirmed Incident

Example Benign Outcome

Shell initiated by authorized SRE

Change ticket exists

No follow-on suspicious activity

No credential exposure

Result:

Benign Administrative Activity

But you may still decide production shell access should be better governed.

Example Malicious Outcome

No approved administrator activity

Unexpected shell

Credential access

External connection

Excessive ServiceAccount permissions

Result:

Potential Container Compromise

Part 74 — Runtime Security Finding

Document:

Finding:
Unexpected Interactive Shell
Observed in Application Container

Affected Resource:
runtime-app

Namespace:
ghc-runtime-lab

Observation:
An interactive shell executed inside
a container whose normal behavior
does not require shell activity.

Security Impact:
Unexpected shell execution may indicate
unauthorized administrative activity
or post-exploitation behavior.

Risk:
High

Recommendation:
Investigate the initiating identity,
review Kubernetes audit activity,
validate ServiceAccount permissions,
review subsequent runtime events,
and restrict unnecessary administrative access.

Part 75 — Excessive Runtime Privilege Finding

Example:

Finding:
Application Container Executes
with Excessive Runtime Privileges

Observation:
The workload configuration grants
permissions beyond the application's
normal operational requirement.

Threat Scenario:
If the application is compromised,
excessive privileges may increase
post-compromise impact.

Recommendation:
Apply least privilege through
securityContext controls and admission policy.

Part 76 — Runtime Finding Template

Use:

Finding:

Alert:

Timestamp:

Affected Cluster:

Namespace:

Pod:

Container:

Process:

Observed Behavior:

Expected Behavior:

Evidence:

Service Account:

RBAC Exposure:

Network Exposure:

Data Exposure:

Threat Scenario:

Business Impact:

Risk:

Recommendation:

Containment:

Remediation:

Validation:

Part 77 — Root Cause Analysis

Do not stop at:

We Deleted the Pod

Ask:

How Did the Activity Begin?

Why Was It Possible?

Why Was It Not Prevented?

Why Was It Detected?

What Increased the Blast Radius?

Which Control Should Change?

Root Cause Categories

Possible categories include:

Application Vulnerability

Weak Authentication

Excessive RBAC

Weak Network Segmentation

Insecure Workload Configuration

Compromised Image

Credential Exposure

Administrative Misuse

Part 78 — Security Control Improvement

Map findings to controls.

Runtime Finding	Preventive Improvement
Privileged container	Admission policy
Excessive SA permissions	RBAC
Unnecessary outbound access	NetworkPolicy
Root execution	Workload security policy
Unknown image	Registry policy
Unexpected shell	Runtime detection + access governance
Credential exposure	Secrets management

Part 79 — Detection-to-Prevention Loop

This is one of the most important concepts in the lab:

DETECT
  ↓
INVESTIGATE
  ↓
UNDERSTAND ROOT CAUSE
  ↓
IMPROVE PREVENTION
  ↓
IMPROVE DETECTION

Part 80 — Detection Coverage Assessment

Ask:

Can We Detect Unexpected Shells?

Can We Detect Privilege Changes?

Can We Detect Sensitive File Access?

Can We Detect Unexpected Processes?

Can We Detect Suspicious Network Activity?

Can We Correlate With Kubernetes Identity?

Record gaps.

Part 81 — Detection Gap Finding

Example:

Finding:
Insufficient Kubernetes Runtime Monitoring

Observation:
The environment lacks reliable detection
for unexpected process execution
inside production containers.

Impact:
Post-exploitation activity may occur
without timely security visibility.

Risk:
High

Recommendation:
Implement runtime telemetry and detection
for high-risk container behaviors and
integrate alerts with centralized monitoring.

Part 82 — SIEM Integration

In enterprise environments:

Runtime Security
      ↓
Alert Pipeline
      ↓
SIEM
      ↓
SOC

This allows correlation with:

Cloud Logs

Identity Logs

Kubernetes Audit Logs

Network Logs

Application Logs

Example Correlation

Suspicious Login
      ↓
Kubernetes API Request
      ↓
Pod Exec
      ↓
Runtime Shell Alert

Together, these events tell a stronger story than any one signal.

Part 83 — Runtime Alert Priority

Prioritize alerts using:

Severity

Confidence

Workload Criticality

Internet Exposure

Privilege

Data Sensitivity

Identity Permissions

Follow-On Activity

Part 84 — High-Risk Combination

For example:

Internet-Facing Pod
      +
Unexpected Shell
      +
Powerful ServiceAccount
      +
No Egress Restrictions

should generally receive more attention than an isolated low-impact training workload.

Part 85 — Detection Rule Quality

A strong detection rule should produce useful context.

Instead of:

Shell Detected

prefer telemetry containing:

Shell Detected

Cluster

Namespace

Pod

Container

Process

User

Parent Process

Node

Timestamp

Part 86 — Investigation Runbook Thinking

Every high-value runtime alert should eventually have:

Alert Description

Triage Steps

Evidence Sources

Escalation Criteria

Containment Options

Recovery Steps

This converts detection engineering into repeatable operations.

Part 87 — Runtime Security Challenge 01

Scenario:

Unexpected Shell
in Web Application Pod

Determine:

Was kubectl exec used?

Which user initiated it?

Is it approved?

What happened afterward?

Document your conclusion.

Part 88 — Runtime Security Challenge 02

Scenario:

Unexpected Sensitive File Access

Determine:

Which Process?

Which File?

Normal Application Behavior?

Any Credential Exposure?

Rotation Required?

Part 89 — Runtime Security Challenge 03

Scenario:

Unexpected Outbound Connection

Determine:

Source Process

Destination

Port

NetworkPolicy

Business Requirement

Potential Exfiltration Risk

Part 90 — Runtime Security Challenge 04

Scenario:

Compromised Pod
uses powerful ServiceAccount

Determine:

Effective RBAC

Accessible Resources

Potential Cluster Impact

Required Credential/RBAC Containment

Part 91 — Runtime Security Challenge 05

Build a timeline from:

Application Request

Shell Alert

Sensitive File Access

Network Connection

Kubernetes API Activity

Then determine the likely sequence of events.

Part 92 — Evidence Collection Checklist

Capture:

Part 93 — Incident Report Template

Incident:
Kubernetes Runtime Security Event

Date:

Cluster:

Namespace:

Affected Workload:

Initial Alert:

Observed Behavior:

Expected Behavior:

Runtime Evidence:

Kubernetes Evidence:

Identity Evidence:

Network Evidence:

Potential Data Exposure:

Blast Radius:

Root Cause:

Severity:

Containment:

Remediation:

Recovery:

Preventive Improvements:

Detection Improvements:

Final Status:

Part 94 — Cleanup

Before cleanup, preserve the evidence required for your lab report.

Then remove the training namespace:

kubectl delete namespace ghc-runtime-lab

Verify:

kubectl get namespace ghc-runtime-lab

Restore your namespace context if necessary:

kubectl config set-context --current --namespace=default

Important

Do not remove:

Falco

Runtime Security Agents

Cluster Logging

Monitoring Components

unless they were installed specifically for your isolated lab and you are responsible for removing them.

Lab Completion Checklist

Runtime Fundamentals

Understood runtime security
Understood preventive vs detective controls
Understood behavioral monitoring
Established application baseline

Runtime Detection

Understood Falco concepts
Generated a safe shell event
Reviewed runtime-event context
Understood process monitoring
Understood sensitive-file monitoring
Understood network monitoring

Kubernetes Investigation

Incident Analysis

Security Improvement

Mapped runtime findings to RBAC
Mapped findings to NetworkPolicy
Mapped findings to admission policy
Identified detection improvements
Documented security finding

Skills You Practiced

You have now worked with:

Kubernetes Runtime Security

Container Monitoring

Behavioral Detection

Falco Concepts

Process Analysis

Alert Triage

Runtime Investigation

Kubernetes Context Enrichment

Blast Radius Analysis

Containment

Incident Response

Career Connection

These skills are highly relevant for:

Kubernetes Security Engineer

Cloud Security Engineer

SOC Analyst

Cloud SOC Analyst

Detection Engineer

DevSecOps Engineer

Platform Security Engineer

Incident Responder

Cloud Incident Responder

Runtime security becomes especially important for security professionals responsible for:

Production Kubernetes

Cloud-Native Applications

Containerized Workloads

Enterprise Detection and Response

Interview Questions

What is Kubernetes runtime security?
Why is runtime security necessary if admission policies already exist?
What is the difference between preventive and detective controls?
What is Falco?
What types of behavior can runtime security monitor?
Why can a shell inside a container be suspicious?
Does a shell automatically mean compromise?
What context should accompany a runtime alert?
Why is application baselining important?
What is behavioral detection?
Why can package installation inside a production container be suspicious?
Why is unexpected process execution important?
What is a container escape?
Why are privileged containers high risk?
How can hostPath increase incident impact?
Why are hostPID and hostNetwork security-sensitive?
Why should Linux capabilities be reviewed?
How can read-only root filesystems improve security?
Why should you investigate a Pod’s ServiceAccount?
How can RBAC affect runtime incident blast radius?
Why should NetworkPolicies be reviewed during runtime investigations?
What is lateral movement?
How can egress restrictions reduce runtime risk?
What evidence would you collect during a container incident?
Why are Kubernetes audit logs useful?
How can pods/exec activity help explain a shell alert?
What is alert enrichment?
What is alert triage?
What is a false positive?
How should runtime rules be tuned?
Why should detections not simply be disabled because of noise?
How do you determine incident severity?
What is blast-radius analysis?
How would you contain a compromised Kubernetes workload?
Why might credentials need to be rotated?
Why should compromised containers generally be replaced rather than manually cleaned?
How can runtime findings improve admission policies?
How can runtime findings improve NetworkPolicies?
How can runtime findings improve RBAC?
What is the detect-investigate-prevent feedback loop?
How can runtime alerts integrate with a SIEM?
Why is event correlation important?
What is runtime evidence?
Why is timeline reconstruction useful?
How would you investigate an unexpected outbound connection?
How would you investigate suspicious credential access?
What is runtime detection coverage?
What is a detection gap?
How would you build a runtime incident runbook?
What controls should complement runtime security?

Practical Readiness Milestone

You should now be able to receive an alert:

Shell Spawned in Container

and perform:

Alert
  ↓
Identify Cluster
  ↓
Identify Pod
  ↓
Identify Process
  ↓
Review Baseline
  ↓
Review Security Context
  ↓
Review Service Account
  ↓
Review RBAC
  ↓
Review Network Access
  ↓
Correlate Evidence
  ↓
Determine Blast Radius
  ↓
Contain

You are no longer looking only at:

The Alert

You are investigating:

The Entire Kubernetes Context
Around the Alert.

Security Readiness Milestone

You should now understand the relationship between:

RBAC
   ↓
Who Can Do What?

Admission Policy
   ↓
What Can Be Deployed?

NetworkPolicy
   ↓
Where Can It Communicate?

Runtime Security
   ↓
What Is It Actually Doing?

This creates:

IDENTITY
   +
CONFIGURATION
   +
NETWORK
   +
BEHAVIOR

Final Lab Mental Model

Remember:

WORKLOAD
   ↓
NORMAL BASELINE
   ↓
RUNTIME TELEMETRY
   ↓
DETECTION
   ↓
ALERT
   ↓
CONTEXT
   ↓
INVESTIGATION
   ↓
BLAST RADIUS
   ↓
CONTAINMENT
   ↓
REMEDIATION

The goal of runtime security is not:

Generate More Alerts

The goal is:

Detect Meaningful
Post-Deployment Threats
Early Enough to Respond.

Lab Outcome

Before this lab:

You focused mainly on
preventing insecure Kubernetes
configurations and communication.

After this lab:

You established workload baselines,

generated safe runtime events,

investigated process activity,

correlated Kubernetes context,

reviewed service-account permissions,

assessed network exposure,

built an incident timeline,

evaluated blast radius,

and designed containment actions.

You have moved from:

Kubernetes Prevention

to:

Kubernetes Detection
and Response.

What’s Next?

➡️ Lab 07 — Workload Security

In the final Kubernetes lab, you will bring the security controls together and harden the workload itself.

You will focus on:

Security Contexts

Non-Root Execution

Privilege Escalation

Linux Capabilities

Read-Only Filesystems

Resource Controls

Service Accounts

Secrets

Container Images

Pod Security Standards

Workload Hardening

The lab progression becomes:

Lab 01 — Kubernetes Fundamentals
        ↓
Understand Resources

Lab 02 — Kubernetes RBAC
        ↓
Control Identity

Lab 03 — Kyverno
        ↓
Control Configuration

Lab 04 — Network Policies
        ↓
Control Communication

Lab 05 — OPA Gatekeeper
        ↓
Enforce Governance

Lab 06 — Runtime Security
        ↓
Detect Suspicious Behavior

Lab 07 — Workload Security
        ↓
Harden Kubernetes Workloads

After Lab 07, you will be ready to move from individual security controls into repeatable Kubernetes security operations through the Kubernetes Runbooks.