Lesson 11 — Kubernetes Security Best Practices

Learning Objectives

By the end of this lesson, you will be able to:

Explain the core principles of Kubernetes security
Apply defence-in-depth across Amazon EKS environments
Design secure Kubernetes identity, workload and network controls
Implement secure software supply chain practices
Apply governance and compliance controls at enterprise scale
Protect the Kubernetes API and control plane
Improve logging, monitoring and incident readiness
Manage vulnerabilities, patches and platform upgrades
Build secure multi-account and multi-cluster EKS environments
Develop a practical enterprise Kubernetes security roadmap

Why This Matters

Kubernetes security is not achieved through one configuration, one tool or one policy.

A cluster may use:

Kubernetes RBAC
Network Policies
Pod Security Admission
Kyverno
OPA Gatekeeper
Container image scanning
Runtime monitoring
Amazon EKS control plane logging

Yet still remain vulnerable if these controls are poorly configured, inconsistently deployed or not monitored.

Enterprise Kubernetes security requires a coordinated approach across:

Identity
Workloads
Networks
Data
Images
Nodes
APIs
Policies
Monitoring
Governance
Incident response

Secure Kubernetes

=

Secure Platform

+

Secure Workloads

+

Secure Operations

+

Continuous Governance

For a Cloud Security Engineer, the objective is not simply to protect one cluster.

The objective is to create a repeatable and measurable security baseline that protects every cluster, application and environment.

Kubernetes Security Principles

A strong Kubernetes security programme should follow several core principles.

Kubernetes Security Principles

├── Least Privilege
├── Defence in Depth
├── Zero Trust
├── Secure by Default
├── Automation
├── Continuous Verification
├── Separation of Duties
├── Immutable Infrastructure
└── Measurable Governance

These principles should guide architecture, configuration and operational decisions.

Defence in Depth

Defence in depth means using multiple complementary security controls.

Identity Controls

↓

Network Controls

↓

Admission Policies

↓

Workload Hardening

↓

Runtime Monitoring

↓

Logging and SIEM

↓

Incident Response

If one control fails, another layer can reduce the impact.

For example:

RBAC restricts who can create workloads.
Admission policies reject privileged workloads.
Runtime monitoring detects suspicious behaviour.
Audit logs record the activity.
The SIEM alerts the SOC.

Secure by Default

Secure-by-default platforms apply protective controls automatically.

Examples include:

Private EKS API endpoints where feasible
Restricted Pod Security Admission
Default-deny Network Policies
Non-root containers
Read-only root filesystems
Dedicated Service Accounts
Encrypted Secrets
Approved image registries
Central logging
Automated compliance checks

New Cluster

↓

Approved Secure Baseline

↓

Security Controls Automatically Enabled

↓

Application Teams Deploy Within Guardrails

Security should not depend on every developer remembering every requirement.

Shared Responsibility in Amazon EKS

Amazon EKS follows the AWS shared responsibility model.

AWS manages the security of:

Managed Kubernetes control plane infrastructure
Control plane availability
Underlying managed-service infrastructure

The customer remains responsible for:

Cluster configuration
IAM
EKS access
Kubernetes RBAC
Worker nodes
Workloads
Images
Network Policies
Secrets
Logging
Monitoring
Compliance
Governance

AWS

Manages the Managed Control Plane

+

Customer

Secures Cluster Access, Nodes, Workloads and Data

Using a managed service does not remove the need for Kubernetes security engineering.

Enterprise Kubernetes Security Architecture

Enterprise Identity Provider

↓

AWS IAM and Federation

↓

Amazon EKS Access Governance

↓

Kubernetes RBAC

↓

Admission Control

↓

Workload Security

↓

Network Segmentation

↓

Runtime Detection

↓

Central Logging and SIEM

↓

Incident Response and Governance

Each layer should be centrally designed and consistently implemented.

Best Practice 1 — Maintain an Accurate Cluster Inventory

Security begins with knowing what exists.

Maintain an inventory of:

AWS accounts
AWS Regions
EKS clusters
Kubernetes versions
Node groups
Namespaces
Applications
Cluster owners
Business owners
Data classifications
Compliance requirements
Internet exposure
Lifecycle status

Example Cluster Inventory

Cluster	Environment	Region	Owner	Criticality	Compliance
payments-prod	Production	Primary Region	Payments Platform	Critical	PCI DSS
customer-dev	Development	Primary Region	Digital Team	Medium	Internal
analytics-prod	Production	Secondary Region	Data Platform	High	ISO-aligned

Unknown clusters become unmanaged risk.

Detect Unmanaged Clusters

Enterprise teams should detect:

Clusters created outside approved pipelines
Clusters without ownership tags
Clusters without central logging
Unsupported Kubernetes versions
Clusters outside governance scope
Test clusters left running indefinitely

AWS Organizations Inventory

↓

EKS Cluster Discovery

↓

Compare Against Approved Inventory

↓

Identify Unmanaged Clusters

↓

Assign Owner or Decommission

Best Practice 2 — Standardise Cluster Provisioning

Clusters should be created from approved templates.

Use:

Infrastructure as Code
Standard VPC architectures
Approved EKS versions
Approved node configurations
Standard logging
Standard add-ons
Standard monitoring
Standard admission controls
Standard encryption

Standard EKS Provisioning Workflow

Cluster Request

↓

Architecture and Risk Classification

↓

Infrastructure as Code

↓

Automated Security Checks

↓

Approval

↓

Cluster Provisioning

↓

Baseline Validation

↓

Application Onboarding

Manual cluster creation should be restricted.

Best Practice 3 — Protect the Kubernetes API

The Kubernetes API Server is one of the most sensitive components in the environment.

Protect it through:

Private endpoint access where feasible
Restricted public endpoint CIDRs where public access is required
Strong AWS IAM authentication
MFA for human administrators
Short-lived credentials
Least-privilege access
EKS control plane logging
Monitoring failed and unusual API activity

EKS API Access Architecture

Administrator

↓

Enterprise Identity Provider

↓

AWS IAM Role

↓

Approved Network Path

↓

Amazon EKS API Endpoint

↓

Kubernetes RBAC

Avoid unrestricted public API access.

Public and Private Endpoints

Endpoint Model	Benefit	Consideration
Private only	Reduced external exposure	Requires private connectivity
Public and private	Operational flexibility	Public access must be restricted
Public only	Simpler access	Larger attack surface

The selected model should match business and operational requirements.

Best Practice 4 — Use Strong Human Authentication

Human access should use:

Enterprise identity federation
MFA
Short-lived sessions
Approved administrator roles
Conditional access controls where available
Central access reviews
Session logging

Avoid:

Long-lived IAM users
Shared administrator accounts
Static kubeconfig credentials
Permanent cluster-admin access

Human Access Flow

User

↓

Enterprise Identity Provider

↓

MFA

↓

Temporary AWS Role

↓

EKS Access Entry

↓

Kubernetes RBAC

Best Practice 5 — Apply Least-Privilege RBAC

Kubernetes RBAC should provide only the permissions required.

Avoid:

verbs:
  - "*"

resources:
  - "*"

Prefer explicit permissions:

rules:
  - apiGroups:
      - apps

    resources:
      - deployments

    verbs:
      - get
      - list
      - watch

High-Risk Kubernetes Permissions

Review access to:

Secrets
Pods
Pod execution
Pod creation
Service Account tokens
Roles
RoleBindings
ClusterRoles
ClusterRoleBindings
Webhook configurations
Nodes
Impersonation

Pod creation can itself be a privilege-escalation path.

RBAC Review Questions

Ask:

Who has cluster-admin?
Who can create RoleBindings?
Who can read Secrets?
Who can create privileged Pods?
Who can create or modify admission webhooks?
Who can impersonate users or groups?
Are wildcard permissions necessary?
Are old bindings still required?

Best Practice 6 — Separate Human and Workload Identities

Human users and workloads should not share identities.

Human Identity

Used for administration

Workload Identity

Used by applications

Workloads should use:

Dedicated Kubernetes Service Accounts
EKS Pod Identity or IRSA
Least-privilege IAM roles
Short-lived credentials

Avoid allowing workloads to inherit broad node IAM permissions.

Workload Identity Architecture

Application Pod

↓

Dedicated Service Account

↓

EKS Pod Identity or IRSA

↓

Least-Privilege IAM Role

↓

Required AWS Service

Best Practice 7 — Disable Unnecessary Service Account Tokens

A Pod that does not need Kubernetes API access should not receive a token.

spec:
  automountServiceAccountToken: false

This reduces the impact of a Pod compromise.

Use dedicated Service Accounts only when required.

Best Practice 8 — Protect Privileged Access

Privileged access should be:

Approved
Time-limited
Logged
Monitored
Reviewed
Revoked after use

Use a controlled emergency-access process.

Privileged Access Request

↓

Approval

↓

Temporary Role Assignment

↓

Logged Administrative Session

↓

Access Expiry

↓

Post-Access Review

Best Practice 9 — Apply Pod Security Admission

Use Pod Security Admission as a native baseline.

Recommended approach:

Use warn and audit during rollout.
Remediate violations.
Enable enforce.
Target the Restricted profile for production where possible.
Version policy labels to manage upgrades predictably.

Example:

kubectl label namespace production \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/audit=restricted \
pod-security.kubernetes.io/warn=restricted

Best Practice 10 — Deny Privileged Containers

Privileged containers should be prohibited by default.

Unsafe configuration:

securityContext:
  privileged: true

Privileged workloads may gain extensive worker-node access.

Exceptions may be required for:

Security agents
Storage drivers
Networking components

These workloads should use:

Dedicated namespaces
Dedicated node groups
Restricted Service Accounts
Additional monitoring
Formal exceptions

Best Practice 11 — Run Containers as Non-Root

Application containers should not run as root.

securityContext:
  runAsNonRoot: true
  runAsUser: 10001

Running as non-root limits the impact of:

Application vulnerabilities
Malicious packages
Command execution
Container escape attempts

The container image must also support non-root execution.

Best Practice 12 — Disable Privilege Escalation

securityContext:
  allowPrivilegeEscalation: false

This prevents processes from gaining additional privileges through mechanisms such as setuid binaries.

Best Practice 13 — Drop Linux Capabilities

Containers should drop all unnecessary capabilities.

securityContext:
  capabilities:
    drop:
      - ALL

Add back only the specific capabilities required.

capabilities:
  drop:
    - ALL

  add:
    - NET_BIND_SERVICE

Every added capability should have a documented justification.

Best Practice 14 — Use Read-Only Root Filesystems

securityContext:
  readOnlyRootFilesystem: true

Applications that need writable paths should use dedicated volumes.

Example:

volumeMounts:
  - name: temporary-data
    mountPath: /tmp

volumes:
  - name: temporary-data
    emptyDir: {}

This limits unauthorised changes to the container filesystem.

Best Practice 15 — Apply Seccomp

Use the default runtime seccomp profile.

securityContext:
  seccompProfile:
    type: RuntimeDefault

Seccomp restricts the Linux system calls available to the container.

Custom profiles may be appropriate for high-risk workloads but require additional operational maturity.

Secure Pod Example

apiVersion: v1
kind: Pod
metadata:
  name: secure-application
  namespace: production

spec:
  serviceAccountName: application-sa
  automountServiceAccountToken: false

  securityContext:
    runAsNonRoot: true

    seccompProfile:
      type: RuntimeDefault

  containers:
    - name: application
      image: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/application@sha256:exampledigest

      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true

        capabilities:
          drop:
            - ALL

      resources:
        requests:
          cpu: 100m
          memory: 128Mi

        limits:
          cpu: 500m
          memory: 512Mi

Best Practice 16 — Restrict Host Access

Avoid using:

hostNetwork: true
hostPID: true
hostIPC: true

Also restrict:

HostPath volumes
Device mounts
Host ports
Privileged DaemonSets

These settings weaken isolation between Pods and worker nodes.

Best Practice 17 — Enforce Resource Requests and Limits

Every workload should define resource requests and limits.

resources:
  requests:
    cpu: 100m
    memory: 128Mi

  limits:
    cpu: 500m
    memory: 512Mi

Benefits include:

Improved scheduling
Reduced resource exhaustion
Better capacity planning
More predictable performance
Reduced denial-of-service risk

Use LimitRanges and ResourceQuotas at namespace level.

Best Practice 18 — Use Dedicated Node Groups

Separate workloads with different trust or risk levels.

Amazon EKS Cluster

├── General Application Node Group
├── Sensitive Workload Node Group
├── System Workload Node Group
├── Security Tooling Node Group
└── Restricted Legacy Workload Node Group

Use:

Taints
Tolerations
Node selectors
Affinity
Separate IAM roles
Separate security controls

This limits lateral movement and reduces blast radius.

Best Practice 19 — Harden Worker Nodes

Worker-node security should include:

Approved EKS-optimised AMIs
Regular patching
Minimal installed software
Restricted SSH access
IMDSv2
Host-level monitoring
Encrypted storage
Restricted Security Groups
Automated replacement
No unnecessary local accounts

Treat nodes as replaceable infrastructure.

Immutable Node Strategy

Updated Node Image

↓

New Node Group

↓

Workloads Migrated

↓

Old Nodes Drained

↓

Old Node Group Removed

Replacing nodes is generally safer than maintaining long-lived manually modified servers.

Best Practice 20 — Restrict Instance Metadata Access

The worker-node instance metadata service may expose credentials.

Use:

IMDSv2
Restricted metadata hop limits
Pod-level AWS identities
Network controls
Least-privilege node roles

Pod

↓

Should Use Pod-Level Identity

Not

↓

Broad Node IAM Credentials

Best Practice 21 — Apply Default-Deny Network Policies

Namespaces should begin with default-deny behaviour.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production

spec:
  podSelector: {}

  policyTypes:
    - Ingress
    - Egress

Then allow only required traffic.

Network Policy Model

Default Deny

↓

Explicitly Allow:

- Frontend to API
- API to Database
- Workloads to DNS
- Approved Egress

Best Practice 22 — Control East-West Traffic

East-west traffic is communication within the Kubernetes environment.

Protect it using:

Namespace isolation
Network Policies
Security Groups
Service mesh controls where required
mTLS
Workload identity
Application-level authentication

Do not assume internal traffic is trusted.

Best Practice 23 — Secure Ingress

Ingress security should include:

Approved Ingress controllers
TLS
Approved certificates
Restricted hostnames
AWS WAF where appropriate
Load balancer logging
Rate limiting
Authentication
Security headers
No unnecessary public exposure

Internet

↓

AWS WAF

↓

Application Load Balancer

↓

Ingress Controller

↓

Kubernetes Service

↓

Application Pod

Best Practice 24 — Control Egress

Unrestricted egress may allow:

Data exfiltration
Command-and-control traffic
Malware downloads
Credential abuse
Unapproved external-service access

Use:

Egress Network Policies
NAT controls
Proxies
Firewalls
DNS monitoring
Approved endpoint lists
VPC endpoints

Best Practice 25 — Protect DNS

Kubernetes DNS is critical infrastructure.

Protect it by:

Restricting access to DNS services
Monitoring unusual queries
Applying resource limits
Protecting CoreDNS configuration
Restricting configuration changes
Patching CoreDNS
Monitoring resolution failures

DNS activity can reveal compromised workloads.

Best Practice 26 — Encrypt Data at Rest

Use encryption for:

Kubernetes Secrets
EBS volumes
EFS file systems
Databases
Backups
Logs
Evidence repositories

For EKS Secrets encryption, integrate an approved AWS KMS key where applicable.

Encryption Architecture

Kubernetes Secret

↓

EKS Encryption Configuration

↓

AWS KMS Key

↓

Encrypted Storage

Best Practice 27 — Use External Secret Management

For sensitive enterprise workloads, consider:

AWS Secrets Manager
AWS Systems Manager Parameter Store
Secrets Store CSI Driver
External Secrets integrations

Application Pod

↓

Dedicated Workload Identity

↓

External Secrets Provider

↓

Short-Lived Secret Access

Avoid storing plaintext secrets in:

Git repositories
Container images
ConfigMaps
CI/CD logs
Application manifests

Best Practice 28 — Restrict Secret Access

Secret access should be tightly controlled.

Review identities that can:

get Secrets
list Secrets
watch Secrets
Create Pods using Secret volumes
Create Service Account tokens
Read application environment variables

A user who can create a Pod may sometimes access Secrets indirectly.

Best Practice 29 — Use Trusted Container Images

Allow production workloads to use only:

Approved registries
Approved repositories
Approved base images
Immutable digests
Scanned images
Signed images

Example:

image: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/payment-api@sha256:exampledigest

Avoid:

image: payment-api:latest

Best Practice 30 — Scan Images Continuously

Image scanning should occur:

During development
During CI/CD
When pushed to Amazon ECR
Before deployment
After new vulnerability intelligence is published
Periodically while the image remains deployed

Source Code

↓

Build

↓

Dependency Scan

↓

Image Scan

↓

Approval

↓

Deployment

↓

Continuous Reassessment

Best Practice 31 — Sign and Verify Images

Image signing helps prove authenticity.

Approved CI/CD Pipeline

↓

Build Image

↓

Scan Image

↓

Sign Image

↓

Push to Amazon ECR

↓

Verify Signature During Admission

↓

Deploy to Amazon EKS

Verification may also require:

Build provenance
SBOM
Vulnerability attestations
Approved signer identity

Best Practice 32 — Generate and Retain SBOMs

A Software Bill of Materials identifies software components included in an image.

SBOMs support:

Vulnerability investigations
Supply chain assurance
Licence review
Incident response
Component inventory
Customer assurance

SBOM generation should be integrated into the build process.

Best Practice 33 — Protect CI/CD Pipelines

The deployment pipeline is a high-value target.

Protect:

Source repositories
Branch protection
Build runners
Pipeline credentials
Signing identities
Artifact repositories
Deployment roles
GitOps repositories

Use separation of duties between:

Code authors
Reviewers
Build systems
Deployment approvers
Production administrators

Secure Deployment Pipeline

Developer Commit

↓

Peer Review

↓

Security Tests

↓

Approved Build Runner

↓

Image Scan and Signing

↓

Amazon ECR

↓

GitOps Approval

↓

Admission Verification

↓

Amazon EKS

Best Practice 34 — Use Admission Control

Use admission control to prevent insecure configurations before deployment.

A layered approach may include:

Pod Security Admission

For standard Pod hardening

+

Kyverno or Gatekeeper

For custom enterprise requirements

+

Native Admission Policies

For suitable validation use cases

High-Value Admission Policies

Start with:

Block privileged containers
Require non-root execution
Disable privilege escalation
Restrict HostPath
Restrict host namespaces
Require resource limits
Require approved registries
Deny latest tags
Require ownership labels
Restrict LoadBalancer Services
Require dedicated Service Accounts

Best Practice 35 — Use Audit Mode Before Enforcement

New policies should follow a staged rollout.

Policy Developed

↓

Tested in CI/CD

↓

Audit Mode

↓

Violations Reviewed

↓

Workloads Remediated

↓

Exceptions Approved

↓

Enforce Mode

This reduces unexpected application failures.

Best Practice 36 — Manage Policy Exceptions

Exceptions should be:

Narrow
Approved
Documented
Time-limited
Monitored
Reviewed
Removed when no longer needed

Every exception should record:

Policy:

Cluster:

Namespace:

Workload:

Business Justification:

Risk:

Compensating Controls:

Owner:

Approver:

Expiry Date:

Remediation Plan:

Best Practice 37 — Store Policies in Git

Policy repositories should contain:

kubernetes-security-policies/

├── workload-security/
├── identity/
├── networking/
├── supply-chain/
├── governance/
├── tests/
├── exceptions/
└── documentation/

Benefits include:

Review
Traceability
Rollback
Automated testing
Multi-cluster consistency
Audit history

Best Practice 38 — Enable EKS Control Plane Logging

Enable appropriate EKS control plane logs.

These may include:

API Server logs
Audit logs
Authenticator logs
Controller Manager logs
Scheduler logs

Amazon EKS Control Plane Logs

↓

CloudWatch Logs

↓

Central Log Archive

↓

SIEM

↓

SOC Investigation

Logging must be configured according to organisational and regulatory requirements.

Best Practice 39 — Protect Kubernetes Audit Logs

Audit logs should be:

Centralised
Encrypted
Access-controlled
Retained
Monitored
Protected from modification
Available for investigations

Monitor events such as:

RBAC changes
Secret access
Privileged Pod creation
Policy changes
Webhook changes
Namespace deletion
Pod execution
Service Account token creation

Best Practice 40 — Monitor Runtime Behaviour

Admission policies validate configuration but do not detect all runtime threats.

Runtime monitoring should detect:

Shell execution
Unexpected processes
Sensitive file access
Privilege escalation attempts
Container escape behaviour
Cryptomining
Unexpected network connections
Package manager execution
Changes to critical files

Tools such as Falco can support runtime detection.

Best Practice 41 — Centralise Security Monitoring

EKS Audit Logs

+

Runtime Alerts

+

CloudTrail

+

GuardDuty

+

Admission Denials

+

Vulnerability Findings

↓

Enterprise SIEM

↓

SOC Investigation

Centralised monitoring improves correlation and incident detection.

Best Practice 42 — Monitor Policy Engine Health

Monitor:

Admission-controller availability
Webhook latency
Webhook errors
Policy evaluation errors
Audit scan failures
PolicyReport failures
Certificate expiry
Controller CPU and memory
Denied request volume

A failed policy engine may either block operations or create an enforcement gap.

Best Practice 43 — Protect Security Tooling

Attackers may attempt to disable:

Kyverno
Gatekeeper
Falco
Logging agents
Monitoring agents
Admission webhooks
Network Policies

Protect security tooling through:

Restricted RBAC
Dedicated namespaces
GitOps
Change alerts
Multiple replicas
PodDisruptionBudgets
Dedicated node groups where appropriate
Runtime monitoring

Best Practice 44 — Use Continuous Vulnerability Management

Vulnerability management should cover:

Container images
Worker nodes
Kubernetes versions
EKS add-ons
Helm charts
Application dependencies
CI/CD tooling
Runtime packages

Vulnerability Lifecycle

Discover

↓

Validate

↓

Prioritise

↓

Assign Owner

↓

Remediate

↓

Rescan

↓

Close

Prioritisation should consider:

Severity
Exploitability
Internet exposure
Runtime usage
Business criticality
Data sensitivity
Available compensating controls

Best Practice 45 — Keep Kubernetes Supported

Track:

EKS Kubernetes versions
Standard support timelines
Extended support implications
Add-on compatibility
Node AMI compatibility
API deprecations
Admission-policy compatibility

Unsupported or outdated versions increase:

Security risk
Upgrade difficulty
Compliance findings
Operational instability

Upgrade Strategy

Review Compatibility

↓

Test in Development

↓

Upgrade Control Plane

↓

Upgrade Add-ons

↓

Replace Worker Nodes

↓

Validate Workloads

↓

Monitor

↓

Complete Production Rollout

Best Practice 46 — Patch and Replace Worker Nodes

Use managed node groups or automated node lifecycle processes.

Patch through:

Updated AMIs
New launch-template versions
New node groups
Controlled draining
Automated workload rescheduling

Avoid manually patching nodes as long-lived servers where replacement is practical.

Best Practice 47 — Protect Backups

Backups should be:

Encrypted
Access-controlled
Monitored
Retained according to policy
Stored outside the primary failure domain
Protected from deletion
Regularly tested

Back up:

Application data
Persistent volumes
Configuration
GitOps repositories
Critical policy definitions
Recovery documentation

Best Practice 48 — Test Recovery

A backup is not useful unless it can be restored.

Test:

Application recovery
Data recovery
Cluster rebuild
GitOps restoration
Secret restoration
Network configuration
Identity configuration
Security control re-establishment

Backup Success

≠

Recovery Success

Best Practice 49 — Prepare for Incident Response

Create Kubernetes-specific response procedures.

Scenarios should include:

Compromised Pod
Stolen Service Account token
Malicious image
Exposed Secret
Privileged container
Compromised worker node
Suspicious cluster-admin activity
Admission policy bypass
Cryptomining
Data exfiltration

Kubernetes Incident Response Workflow

Alert

↓

Validate

↓

Identify Affected Cluster and Workload

↓

Contain

↓

Preserve Evidence

↓

Eradicate

↓

Recover

↓

Lessons Learned

Containment Actions

Possible containment measures include:

Isolate a namespace
Apply emergency Network Policies
Scale a workload to zero
Revoke IAM access
Remove an EKS access entry
Disable a Service Account
Quarantine an image
Drain and replace a node
Restrict egress
Rotate Secrets

Containment decisions should consider business impact.

Best Practice 50 — Preserve Forensic Evidence

During an incident, preserve:

Audit logs
CloudTrail events
Pod metadata
Container images
Runtime alerts
Node logs
Network telemetry
IAM role activity
Deployment records
Admission-policy results

Avoid destroying evidence before collection.

Best Practice 51 — Use Continuous Compliance

Continuously evaluate:

Cluster configuration
IAM and RBAC
Pod Security coverage
Admission policies
Network Policy coverage
Image security
Vulnerabilities
Logging
Backup testing
Exceptions
Kubernetes versions

Cluster State

↓

Automated Assessment

↓

Evidence

↓

Finding

↓

Remediation

↓

Retest

Best Practice 52 — Measure Control Effectiveness

Do not only verify that a control exists.

Verify:

It is correctly designed.
It covers every required cluster.
It is operating.
It cannot be easily bypassed.
Violations are investigated.
Evidence is available.
Exceptions are controlled.

Example:

Kyverno Policy Exists

But

Policy Is in Audit Mode

And

Production Namespace Is Excluded

Therefore

Control Is Not Effectively Enforced

Best Practice 53 — Use Risk-Based Security

Not every cluster requires identical controls.

Classify clusters by:

Business criticality
Data sensitivity
Compliance scope
Internet exposure
Availability requirements
Threat profile

Security Tiers

Tier	Environment	Example Controls
Tier 1	Sandbox	Audit policies and basic logging
Tier 2	Development	Baseline enforcement and scanning
Tier 3	Production	Restricted policies and runtime detection
Tier 4	Regulated	Enhanced isolation, signing and evidence

Every environment should still meet a minimum baseline.

Best Practice 54 — Apply Multi-Account Governance

Use AWS Organisations and landing-zone controls to standardise:

Account structure
Logging
Security services
Network connectivity
IAM
Encryption
Tagging
Evidence collection
Guardrails

AWS Organization

↓

Organisational Units

↓

Workload Accounts

↓

Amazon EKS Clusters

↓

Central Security and Logging Accounts

Best Practice 55 — Centralise Security Services

A central security account may aggregate:

CloudTrail
GuardDuty
Security Hub
Inspector
Compliance evidence
SIEM feeds
Security dashboards

Centralisation improves:

Visibility
Separation of duties
Evidence integrity
Cross-account correlation
Incident response

Best Practice 56 — Implement Separation of Duties

Separate responsibilities between:

Platform administrators
Security policy owners
Application developers
Deployment approvers
Compliance reviewers
SOC analysts
Business risk owners

One individual should not control every stage of a critical production deployment.

Best Practice 57 — Use GitOps for Production

Production configuration should be managed through approved repositories.

Developer Change

↓

Pull Request

↓

Automated Validation

↓

Peer Review

↓

Approval

↓

GitOps Controller

↓

Amazon EKS

Direct changes should be:

Restricted
Logged
Temporary
Reconciled back into Git
Reviewed after emergency use

Best Practice 58 — Detect Configuration Drift

Drift may occur through:

Manual changes
Failed GitOps reconciliation
Unapproved policy edits
Emergency actions
Controller failures
Cluster upgrades

Monitor differences between:

Approved State in Git

and

Actual State in Amazon EKS

Best Practice 59 — Apply Ownership Metadata

Resources should include labels such as:

metadata:
  labels:
    owner: payments-team
    application: payment-api
    environment: production
    data-classification: restricted
    criticality: high

Ownership metadata supports:

Incident response
Cost management
Compliance
Finding assignment
Operational support

Best Practice 60 — Review Security Regularly

Conduct periodic reviews of:

Cluster access
RBAC
Service Accounts
Node IAM roles
Admission policies
Exceptions
Network Policies
Vulnerabilities
Logging
Backup tests
Incident response
Compliance mappings

Reviews should also occur after:

Major incidents
Platform upgrades
New regulatory requirements
Architecture changes
New threat intelligence

Secure Amazon EKS Reference Architecture

Enterprise Identity Provider
        |
        v
AWS IAM Federation + MFA
        |
        v
Restricted EKS API Access
        |
        v
EKS Access Entries + Kubernetes RBAC
        |
        v
Pod Security Admission
        |
        v
Kyverno or Gatekeeper
        |
        v
Approved Signed Images from Amazon ECR
        |
        v
Hardened Workloads on Dedicated Node Groups
        |
        v
Default-Deny Network Policies
        |
        v
Encrypted Data and External Secrets
        |
        v
Runtime Detection and EKS Audit Logging
        |
        v
CloudWatch, Security Hub and Enterprise SIEM
        |
        v
SOC, Compliance and Incident Response

Enterprise Security Baseline

Every production EKS cluster should have a documented minimum baseline.

Security Area	Minimum Baseline
Identity	Federation, MFA and least privilege
API Access	Restricted endpoint access
RBAC	No unapproved cluster-admin
Workloads	Restricted Pod security
Admission	Enforced custom security policies
Networking	Default-deny and approved ingress
Secrets	Encryption and restricted access
Images	Approved, scanned and immutable
Nodes	Patched and hardened
Logging	Control plane and audit logging
Monitoring	Runtime and SIEM integration
Governance	Inventory, ownership and exceptions
Recovery	Protected backups and restore tests

Enterprise Monitoring Architecture

Amazon EKS

├── Control Plane Logs
├── Kubernetes Audit Logs
├── Admission Policy Events
├── Runtime Security Alerts
├── Workload Metrics
├── Network Telemetry
└── Vulnerability Findings

↓

CloudWatch, Prometheus and Security Services

↓

Central SIEM

↓

SOC

↓

Incident Response and Compliance Reporting

Enterprise Security Metrics

Metric	Target
Production clusters with approved baseline	100%
Production namespaces under policy enforcement	100%
Unapproved privileged workloads	0
Unapproved cluster-admin bindings	0
Critical vulnerabilities outside SLA	0
Clusters with audit logging	100%
Expired policy exceptions	0
Production images from approved registries	100%
Signed production images	Defined enterprise target
Successful restore tests	100%
Supported Kubernetes versions	100%
High-risk findings with assigned owners	100%

Common Security Failures

Relying Only on RBAC

RBAC controls actions but not workload configuration.

Risk: Authorised users deploy insecure workloads.

Control: Add admission policy and workload hardening.

Using One Security Tool as the Complete Solution

A single policy engine or scanner cannot cover all threats.

Risk: Important security gaps remain.

Control: Apply defence in depth.

Permanent Cluster-Admin Access

Administrators retain broad access indefinitely.

Risk: Credential compromise has severe impact.

Control: Use temporary privileged access and regular reviews.

Unrestricted Pod Egress

Compromised workloads can communicate externally.

Risk: Data exfiltration and command-and-control activity.

Control: Apply egress controls and monitoring.

Mutable Container Images

Image tags can change after approval.

Risk: Unreviewed code may reach production.

Control: Use immutable digests and signature verification.

Policies Without Enforcement

Policies remain in Audit mode indefinitely.

Risk: Violations are recorded but not prevented.

Control: Define enforcement dates and readiness criteria.

Broad Policy Exceptions

Entire namespaces or teams are excluded.

Risk: Policy coverage is significantly weakened.

Control: Use narrow, time-limited exceptions.

Logging Without Monitoring

Logs are collected but never reviewed.

Risk: Security incidents remain undetected.

Control: Create SIEM alerts and SOC procedures.

Backups Without Restore Testing

Backups complete successfully but cannot be restored.

Risk: Recovery fails during an incident.

Control: Conduct scheduled recovery exercises.

Unsupported Kubernetes Versions

Clusters remain on outdated versions.

Risk: Security vulnerabilities and compatibility issues increase.

Control: Maintain a planned upgrade programme.

Enterprise Implementation Strategy

Phase 1 — Discover and Classify

Inventory AWS accounts and EKS clusters.
Assign owners.
Classify business criticality.
Classify data sensitivity.
Identify compliance requirements.
Identify unmanaged clusters.

Phase 2 — Establish the Platform Baseline

Standardise EKS architecture.
Restrict API access.
Enable control plane logging.
Configure encryption.
Standardise managed add-ons.
Harden worker nodes.
Implement central monitoring.

Phase 3 — Secure Identity and Access

Implement federation and MFA.
Use EKS access entries.
Apply least-privilege RBAC.
Remove unnecessary cluster-admin.
Implement pod-level AWS identities.
Review access regularly.

Phase 4 — Secure Workloads

Apply Pod Security Admission.
Require non-root execution.
Disable privilege escalation.
Drop capabilities.
Require read-only filesystems.
Restrict host access.
Enforce resource controls.

Phase 5 — Secure Networks and Data

Apply default-deny Network Policies.
Restrict ingress and egress.
Protect DNS.
Encrypt data at rest and in transit.
Use external secret management.
Restrict Secret access.

Phase 6 — Secure the Software Supply Chain

Use approved registries.
Scan images and dependencies.
Generate SBOMs.
Sign images.
Verify signatures and attestations.
Protect CI/CD and GitOps systems.

Phase 7 — Enforce Policy

Deploy Kyverno, Gatekeeper or suitable native policies.
Begin in Audit mode.
Remediate violations.
Define exceptions.
Enable enforcement.
Monitor policy health.

Phase 8 — Implement Detection and Response

Centralise audit logs.
Deploy runtime monitoring.
Integrate AWS security services.
Send findings to the SIEM.
Create Kubernetes incident runbooks.
Conduct response exercises.

Phase 9 — Establish Continuous Compliance

Automate evidence collection.
Assess controls continuously.
Track findings.
Review exceptions.
Build technical and executive dashboards.
Retest remediation.

Phase 10 — Improve Continuously

Review security metrics.
Update baselines.
Incorporate new threat intelligence.
Improve policy coverage.
Reduce manual controls.
Test recovery.
Conduct architecture reviews.

Enterprise Best-Practice Checklist

Cluster and Control Plane

Maintain a complete EKS inventory.
Use approved Infrastructure as Code.
Restrict the Kubernetes API endpoint.
Enable control plane logging.
Use supported Kubernetes versions.
Encrypt Kubernetes Secrets.

Identity and Access

Use enterprise federation and MFA.
Use temporary credentials.
Apply least-privilege RBAC.
Review cluster-admin access.
Use dedicated Service Accounts.
Use EKS Pod Identity or IRSA.
Disable unnecessary token mounting.

Workload Security

Apply Restricted Pod Security standards.
Deny privileged containers.
Require non-root execution.
Disable privilege escalation.
Drop unnecessary capabilities.
Apply seccomp.
Use read-only root filesystems.
Restrict host namespaces and HostPath.
Require resource requests and limits.

Network Security

Supply Chain Security

Monitoring and Response

Governance and Compliance

Real-World Scenario

A global financial organisation operates over 400 Amazon EKS clusters across multiple AWS accounts and Regions.

The environment supports:

Online banking
Payment processing
Fraud detection
Customer identity services
Internal analytics

A security review identifies:

Permanent cluster-admin access
Public EKS API endpoints
Privileged workloads
Images using mutable tags
Missing Network Policies
Broad node IAM roles
Inconsistent audit logging
Untracked policy exceptions
Unsupported Kubernetes versions

The organisation launches an enterprise EKS security improvement programme.

The Cloud Security and Platform teams:

Create a central cluster inventory.
Classify clusters by business criticality and compliance scope.
Standardise cluster provisioning using Infrastructure as Code.
Restrict EKS API access.
Replace permanent administrator access with federated temporary roles.
Implement EKS access entries and least-privilege Kubernetes RBAC.
Adopt Pod Security Admission and Kyverno.
Enforce non-root, non-privileged and restricted workload settings.
Introduce dedicated Service Accounts and pod-level AWS identities.
Apply default-deny Network Policies.
Restrict production images to approved Amazon ECR repositories.
Require image scanning, signing and verification.
Enable EKS control plane logging in every production cluster.
Deploy runtime monitoring and SIEM correlation.
Establish continuous compliance reporting.
Create a time-limited exception workflow.
Implement a planned Kubernetes upgrade programme.
Conduct incident-response and recovery exercises.

The organisation achieves:

A consistent enterprise security baseline
Reduced privileged access
Improved workload isolation
Stronger supply chain assurance
Faster detection and response
Better audit readiness
Reduced configuration drift
Clear ownership and accountability

Key Takeaways

Kubernetes security requires defence in depth.
Amazon EKS reduces control-plane management but does not remove customer security responsibilities.
Identity, RBAC and admission policies must work together.
Production workloads should run with restricted privileges.
Default-deny networking reduces lateral movement.
Pod-level AWS identities are safer than broad node permissions.
Images should be approved, scanned, signed and immutable.
Admission control prevents insecure configurations before deployment.
Runtime detection identifies malicious behaviour after deployment.
Logs must be centralised, protected and actively monitored.
Policies, exceptions and evidence require formal governance.
Continuous compliance is more effective than periodic assessment.
Backups must be tested through restoration exercises.
Supported Kubernetes versions and patched nodes reduce platform risk.
Security baselines should be automated, measurable and continuously improved.

Knowledge Check

1. Why is defence in depth important for Kubernetes security?

Answer: No single control can prevent or detect every threat. Defence in depth combines identity, network, admission, workload, runtime and monitoring controls so that failure of one layer does not leave the entire environment unprotected.

2. Why should application Pods use pod-level AWS identities instead of node IAM roles?

Answer: Pod-level identities provide application-specific, least-privilege AWS permissions and reduce the risk that a compromised Pod can access the broader permissions assigned to the worker node.

3. What is the purpose of default-deny Network Policies?

Answer: Default-deny policies block network traffic unless it is explicitly allowed, reducing unauthorised communication and lateral movement.

4. Why should container images use immutable digests?

Answer: Immutable digests ensure that the exact approved image content is deployed and cannot be silently replaced while retaining the same tag.

5. Why must security controls be continuously monitored?

Answer: Controls may fail, drift, be bypassed or become ineffective after configuration and platform changes. Continuous monitoring confirms that controls remain operational and violations are detected.

What’s Next?

You have completed Module 07 — Kubernetes Governance, Compliance & Policy.

In the upcoming practical activities, you will apply the concepts from this module by assessing an Amazon EKS environment against the CIS Kubernetes Benchmark and implementing admission policies using Gatekeeper.

➡️ Next: Lab 01 — CIS Benchmark Assessment