Lesson 05 — Audit Log Analysis

Learning Objectives

By the end of this lesson, you will be able to:

Understand Kubernetes Audit Logging
Explain the Kubernetes API audit pipeline
Interpret Kubernetes Audit Log events
Identify suspicious API activity
Investigate user authentication and authorization events
Analyze Service Account activity
Correlate Audit Logs with CloudTrail and runtime detections
Build attack timelines
Identify Indicators of Compromise (IOCs)
Perform enterprise Kubernetes forensic investigations using Audit Logs

Why Audit Logs Matter

Every action inside Kubernetes is performed through the Kubernetes API.

Examples include:

Creating Pods
Deleting Namespaces
Reading Secrets
Updating Deployments
Executing commands inside Pods
Creating RBAC roles
Scaling workloads
Creating Service Accounts
Changing Network Policies

Without Audit Logs, these activities are difficult—or impossible—to reconstruct during an investigation.

User

↓

kubectl

↓

Kubernetes API Server

↓

Audit Log

↓

SIEM

↓

SOC Investigation

Audit Logs provide the authoritative record of who did what, when, where and how within a Kubernetes cluster.

What are Kubernetes Audit Logs?

Kubernetes Audit Logs record every request processed by the Kubernetes API Server.

Each audit event contains:

Timestamp
User identity
Source IP
API request
Namespace
Resource
Request verb
Response status
Authorization decision
User Agent

Enterprise Audit Logging Architecture

Users

↓

kubectl / API Clients

↓

API Server

↓

Audit Policy

↓

Audit Logs

↓

CloudWatch Logs

↓

Amazon Security Lake

↓

SIEM

↓

SOC Analysts

Audit Log Lifecycle

API Request

↓

Authentication

↓

Authorization

↓

Admission Controllers

↓

Resource Processing

↓

Response

↓

Audit Event

Typical Audit Event

{
  "kind": "Event",
  "verb": "create",
  "user": {
    "username": "admin"
  },
  "objectRef": {
    "resource": "pods",
    "namespace": "payments"
  },
  "sourceIPs": [
    "192.168.1.100"
  ]
}

Major Audit Event Fields

Field	Purpose
timestamp	When request occurred
verb	API action
user	Identity
groups	User groups
sourceIPs	Client IP
objectRef	Resource accessed
namespace	Kubernetes namespace
responseStatus	Success or failure
userAgent	Client application

Common API Verbs

Verb	Meaning
create	New resource
get	Read one resource
list	List resources
watch	Watch changes
update	Modify resource
patch	Partial modification
delete	Remove resource
deletecollection	Remove multiple resources
connect	Exec, Attach, Port Forward

High-Value Resources

Monitor access to:

Secrets
Pods
Deployments
Nodes
Namespaces
Service Accounts
ClusterRoles
ClusterRoleBindings
Roles
RoleBindings
ConfigMaps
Network Policies
Admission Controllers

Enterprise Audit Flow

kubectl Apply

↓

Authentication

↓

RBAC

↓

Admission

↓

API Server

↓

Audit Event

↓

CloudWatch

↓

Security Lake

↓

SIEM

↓

Alert

Authentication Investigation

Review:

Username
Groups
Authentication method
Service Account
OIDC identity
IAM identity
Anonymous requests

Questions:

Who performed the action?
Was MFA used (AWS side)?
Was the identity expected?

Authorization Investigation

Determine:

Was access allowed?
Was access denied?
Which RBAC rule granted permission?
Which ClusterRole was used?
Was least privilege followed?

Service Account Activity

Service Accounts should be monitored carefully.

Look for:

Secret access
Pod creation
Exec commands
Namespace access
RBAC changes
Token usage

Example:

system:serviceaccount:payments:payment-api

Sensitive API Operations

High-risk actions include:

Reading Secrets
Creating Pods
Creating Privileged Pods
Creating ClusterRoles
Creating ClusterRoleBindings
Creating Service Accounts
Creating Ephemeral Containers
Executing inside Pods
Port Forwarding
Accessing Nodes

Secret Access Investigation

Example event:

verb: get

resource: secrets

namespace: production

Questions:

Who accessed the Secret?
Was it expected?
Which workload requested it?
Was it followed by suspicious activity?

Exec Investigation

Pod execution events:

resource: pods

subresource: exec

Review:

User
Source IP
Namespace
Pod
Container
Time
User Agent

Unexpected kubectl exec commands often indicate manual investigation—or attacker activity.

Port Forward Investigation

Monitor:

subresource: portforward

Questions:

Who initiated it?
Why?
Was there a change ticket?
Which service became accessible?

Pod Creation Investigation

Review:

verb: create

resource: pods

Determine:

Image used
Namespace
Service Account
Security Context
Labels
Owner

Namespace Activity

Review:

resource: namespaces

Unexpected namespace creation may indicate:

Malware deployment
Hidden workloads
Insider activity
Testing by attackers

RBAC Changes

Critical resources:

roles

rolebindings

clusterroles

clusterrolebindings

Questions:

Who modified permissions?
Why?
Were admin rights granted?

ClusterRoleBinding Investigation

Example:

cluster-admin

Investigate immediately if assigned to:

Developers
Applications
Unknown Service Accounts

Admission Controller Events

Review:

Pod Security Admission
Gatekeeper
Kyverno
Webhook approvals
Policy violations

Determine:

Was a policy bypassed?
Was an exception granted?

Failed Authentication

Monitor:

responseStatus:
401

May indicate:

Credential guessing
Expired tokens
Invalid authentication

Authorization Failures

Monitor:

403 Forbidden

Repeated failures may indicate reconnaissance.

Source IP Investigation

Review:

sourceIPs

Questions:

Internal?
VPN?
Bastion?
Corporate Network?
Public Internet?

Unexpected IP addresses deserve investigation.

User Agent Investigation

Example:

kubectl/v1.31

terraform

helm

argo

curl

python

Unexpected User Agents:

curl
python requests
custom scripts

may indicate automation or malicious tooling.

Timeline Reconstruction

Example:

09:00 Login

↓

09:02 Create Pod

↓

09:04 Exec

↓

09:05 Secret Read

↓

09:07 Port Forward

↓

09:09 Data Exfiltration

Correlation Sources

Combine Audit Logs with:

CloudTrail
Falco
GuardDuty
CloudWatch Logs
VPC Flow Logs
Security Hub
Prometheus Alerts
Application Logs

Example Investigation

Incident:

Unknown Pod appears.

Audit Logs reveal:

create pod

↓

Service Account

↓

Secret Read

↓

Exec

↓

Delete Pod

CloudTrail shows:

IAM Role assumption

Falco detects:

Reverse shell

Timeline confirms attacker activity.

Indicators of Compromise

Look for:

Secret access
Multiple failed logins
Unexpected exec
Unknown namespaces
ClusterRoleBinding creation
Privileged Pod creation
Pod deletion after compromise
Anonymous authentication
Port Forward sessions
Service Account abuse

Enterprise Detection Rules

Alert when:

Secrets are accessed
ClusterRoleBindings created
Privileged Pods created
HostPath mounted
Exec used in production
Anonymous requests occur
Multiple authorization failures
Pods created from unknown registries

Audit Log Retention

Recommended:

Centralized storage
Immutable storage
Long-term retention
Encryption
Access logging
Backup
SIEM integration

Enterprise Investigation Workflow

Alert

↓

Review Audit Logs

↓

Identify User

↓

Identify Resource

↓

Review RBAC

↓

Review Service Account

↓

Review CloudTrail

↓

Correlate Runtime Alerts

↓

Build Timeline

↓

Determine Root Cause

↓

Document Findings

Best Practices

As a Cloud Security Engineer:

Enable Kubernetes Audit Logging on every production cluster.
Forward Audit Logs to CloudWatch and a centralized SIEM.
Protect Audit Logs from unauthorized modification.
Alert on high-risk API operations.
Monitor Service Account activity continuously.
Review RBAC changes regularly.
Correlate Audit Logs with CloudTrail and GuardDuty.
Build automated detections for privileged activity.
Use Audit Logs during every incident investigation.
Retain logs according to organizational compliance requirements.

Real-World Scenario

A retail organization notices a privileged Pod running in the production namespace.

The SOC begins an investigation.

Audit Logs reveal:

A developer’s IAM identity authenticated to the cluster.
A ClusterRoleBinding granting cluster-admin privileges was created.
A new Service Account was created.
A Pod using the new Service Account was deployed.
The Pod executed kubectl exec commands against another workload.
Multiple Secrets were retrieved.
The attacker deleted the Pod minutes later.

CloudTrail confirms that the IAM identity assumed an administrative role from an unfamiliar IP address.

Falco simultaneously reports shell execution inside the Pod.

The investigation concludes that compromised developer credentials were used to escalate privileges and access sensitive data.

The response team revokes credentials, removes unauthorized RBAC permissions, rotates affected Secrets and strengthens monitoring rules.

Key Takeaways

Kubernetes Audit Logs provide a complete record of Kubernetes API activity.
Audit Logs are essential for reconstructing attack timelines.
Monitor high-risk operations such as Secret access, kubectl exec, RBAC changes and privileged Pod creation.
Correlating Audit Logs with CloudTrail and runtime security tools provides comprehensive visibility.
Proper log retention and centralized analysis improve detection, incident response and compliance.

Knowledge Check

1. What information is contained in a Kubernetes Audit Log?

Answer: Audit Logs include details such as timestamp, user identity, source IP, API verb, resource, namespace, response status, authorization result and user agent.

2. Which Kubernetes API operations should always be monitored?

Answer: High-risk operations include Secret access, Pod creation, kubectl exec, Port Forward, RBAC changes, Service Account creation, privileged workload deployment and admission policy changes.

3. Why are Audit Logs valuable during an incident investigation?

Answer: They provide a reliable record of API activity that helps investigators identify who performed an action, what resources were affected and when events occurred.

4. Why should Audit Logs be correlated with AWS CloudTrail?

Answer: Audit Logs record Kubernetes API activity, while CloudTrail records AWS API activity. Correlating both provides end-to-end visibility across Kubernetes and AWS.

5. What are common Indicators of Compromise visible in Audit Logs?

Answer: Indicators include unauthorized Secret access, repeated failed authentication attempts, unexpected kubectl exec sessions, privileged Pod creation, RBAC modifications, anonymous requests and suspicious Service Account activity.

What’s Next?

In the next lesson, we will explore Lesson 06 — Runtime Investigation, where you’ll learn how to investigate live container behavior using runtime security tools such as Falco, Amazon GuardDuty Runtime Monitoring and eBPF-based telemetry to detect malware, container escapes and other advanced threats.