Lab 04 — Harden Production Pods

Mission Information

Item	Details
Lab ID	K8S-WORKLOAD-LAB-04
Difficulty	Advanced
Estimated Time	4–5 Hours
Environment	Kubernetes Cluster
Platform	Amazon EKS / Azure AKS / Google GKE / kind / minikube
Cost	Free for Local Cluster / Cloud Charges May Apply
Primary Role	Kubernetes Security Engineer
Supporting Roles	Platform Engineer, DevSecOps Engineer, Site Reliability Engineer
Module	Kubernetes Workload and Pod Security
Previous Lab	Lab 03 — Configure Security Context
Next Lab	Lab 05 — Runtime Security Validation

Mission Scenario

CloudNova Technologies is preparing to release a new customer-facing application into its production Kubernetes environment.

The application has successfully passed functional testing, but the Production Readiness Review identified several security and operational concerns:

• Containers were not consistently running as non-root.
• Privilege escalation controls were missing.
• The root filesystem was writable.
• Linux capabilities were not explicitly restricted.
• Service Account tokens were mounted without a business requirement.
• Resource requests and limits were incomplete.
• Health probes were not configured.
• The application was exposed directly through an external Service.
• Network Policies were missing.
• Pod disruption controls were not implemented.
• Image versions were not pinned consistently.
• Security labels and ownership metadata were incomplete.
• No formal production workload assessment had been performed.

The CISO has directed the Cloud Security and Platform Engineering teams to establish a hardened production deployment before the application is approved for release.

Your mission is to build a secure, resilient, observable, and production-ready Kubernetes workload using defence-in-depth security controls.

Learning Objectives

By completing this lab, you will learn how to:

• Design a hardened production Pod specification
• Enforce the Restricted Pod Security Standard
• Configure secure Pod and container security contexts
• Run application containers as non-root
• Disable privileged mode and privilege escalation
• Drop unnecessary Linux capabilities
• Apply RuntimeDefault seccomp
• Configure a read-only root filesystem
• Provide controlled writable storage
• Disable unnecessary Service Account token mounting
• Apply resource requests and limits
• Configure liveness, readiness, and startup probes
• Apply network isolation
• Configure secure Service exposure
• Implement Pod disruption protection
• Apply application metadata and governance labels
• Validate workload availability and security
• Perform an enterprise production readiness assessment

Enterprise Production Architecture

                         Internet Users
                                │
                           HTTPS Only
                                │
                    Enterprise Ingress / WAF
                                │
                         ClusterIP Service
                                │
             ┌──────────────────┴──────────────────┐
             │                                     │
      Hardened Application Pod              Hardened Application Pod
             │                                     │
      Non-Root Container                    Non-Root Container
      Read-Only Filesystem                  Read-Only Filesystem
      Capabilities Dropped                  Capabilities Dropped
      Seccomp Enabled                       Seccomp Enabled
      Resource Limits                       Resource Limits
             │                                     │
             └──────────────────┬──────────────────┘
                                │
                       Network Policy Controls
                                │
                       Approved Dependencies

Defence-in-Depth Model

Container Image Security

        │

        ▼

Admission Control

        │

        ▼

Pod Security Context

        │

        ▼

Container Security Context

        │

        ▼

Network Isolation

        │

        ▼

Runtime Monitoring

        │

        ▼

Production Governance

Lab Outcomes

By the end of this lab, you will have:

• Created a protected production namespace
• Enforced the Restricted Pod Security Standard
• Deployed a hardened multi-replica application
• Configured secure Pod and container security contexts
• Disabled automatic Service Account token mounting
• Implemented a read-only root filesystem
• Configured dedicated writable directories
• Added CPU and memory governance
• Added startup, readiness, and liveness probes
• Created an internal ClusterIP Service
• Implemented default-deny and application-specific Network Policies
• Configured a PodDisruptionBudget
• Validated secure application behaviour
• Simulated common attack actions
• Produced an enterprise production readiness report

Production Security Principles

Apply the following principles throughout the lab:

• Deny by default.
• Run workloads with the minimum required privilege.
• Treat container filesystems as immutable.
• Grant access only when a documented requirement exists.
• Prevent direct public exposure of backend workloads.
• Protect availability through resource and health controls.
• Validate security controls at runtime.
• Record evidence before production approval.

Prerequisites

Before starting, ensure that you have:

• Completed Labs 01–03
• A running Kubernetes cluster
• kubectl installed and configured
• A CNI plugin that supports Network Policies
• Permission to create namespaces, Deployments, Services, Network Policies, and PodDisruptionBudgets
• Visual Studio Code or another YAML editor
• Git Bash or PowerShell
• Basic knowledge of Kubernetes Deployments, Services, probes, and Network Policies

Tools Used

Tool	Purpose
kubectl	Deploy, inspect, and validate Kubernetes resources
Kubernetes	Run the production workload
nginx-unprivileged	Provide a non-root web application
BusyBox	Perform connectivity and runtime testing
Visual Studio Code	Create and edit manifests
Git Bash / PowerShell	Execute lab commands

Recommended Lab File Structure

lab-04-harden-production-pods/
├── 01-production-namespace.yaml
├── 02-configmap.yaml
├── 03-production-deployment.yaml
├── 04-production-service.yaml
├── 05-default-deny-policy.yaml
├── 06-allow-application-policy.yaml
├── 07-pod-disruption-budget.yaml
├── 08-security-test-pod.yaml
├── evidence/
└── production-readiness-report.md

Task 01 — Verify Cluster Health

Verify cluster connectivity.

kubectl cluster-info

Review the nodes.

kubectl get nodes -o wide

Record the Kubernetes version.

kubectl version

Confirm:

• The API Server is reachable.
• All required worker nodes report Ready.
• The CNI components are healthy.
• Network Policy support is available.

Review cluster networking components.

kubectl get pods -n kube-system

Task 02 — Create the Production Namespace

Create 01-production-namespace.yaml.

apiVersion: v1
kind: Namespace
metadata:
  name: production-app
  labels:
    environment: production
    owner: application-platform
    security-owner: cloud-security
    data-classification: internal
    compliance-scope: in-scope
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

Apply the namespace.

kubectl apply -f 01-production-namespace.yaml

Verify the labels.

kubectl get namespace production-app --show-labels

Task 03 — Review the Restricted Pod Security Standard

The Restricted Pod Security Standard is intended for security-critical and production workloads.

It restricts or requires controls related to:

• Privileged containers
• Host namespaces
• HostPath volumes
• Privilege escalation
• Root execution
• Linux capabilities
• Seccomp profiles
• Unsafe volume types

Validate the enforcement level.

kubectl get namespace production-app \
  -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}{"\n"}'

Expected:

restricted

Task 04 — Test Admission Control

Attempt to create an insecure Pod.

kubectl run insecure-production-test \
  -n production-app \
  --image=busybox:1.36 \
  --command -- sleep 3600

Expected:

• The Pod should be rejected or generate policy violations because required restricted settings are missing.

Check whether it exists.

kubectl get pod insecure-production-test -n production-app

Capture the admission response as evidence.

Task 05 — Create the Application Configuration

Create 02-configmap.yaml.

apiVersion: v1
kind: ConfigMap
metadata:
  name: production-web-content
  namespace: production-app
  labels:
    app: production-web
    environment: production
data:
  index.html: |
    <!DOCTYPE html>
    <html>
      <head>
        <title>CloudNova Production Application</title>
      </head>
      <body>
        <h1>CloudNova Production Application</h1>
        <p>Secure Kubernetes workload is running.</p>
      </body>
    </html>

Apply it.

kubectl apply -f 02-configmap.yaml

Verify it.

kubectl get configmap production-web-content \
  -n production-app

Task 06 — Create the Hardened Production Deployment

Create 03-production-deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: production-web
  namespace: production-app
  labels:
    app: production-web
    app.kubernetes.io/name: production-web
    app.kubernetes.io/component: frontend
    app.kubernetes.io/part-of: customer-platform
    app.kubernetes.io/managed-by: kubectl
    environment: production
    owner: application-platform
    security-tier: restricted
spec:
  replicas: 3
  revisionHistoryLimit: 5
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: production-web
  template:
    metadata:
      labels:
        app: production-web
        app.kubernetes.io/name: production-web
        app.kubernetes.io/component: frontend
        environment: production
        security-tier: restricted
    spec:
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        runAsUser: 101
        runAsGroup: 101
        fsGroup: 101
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: web
          image: nginxinc/nginx-unprivileged:1.27-alpine
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 300m
              memory: 128Mi
          startupProbe:
            httpGet:
              path: /
              port: http
              scheme: HTTP
            failureThreshold: 30
            periodSeconds: 2
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /
              port: http
              scheme: HTTP
            initialDelaySeconds: 15
            periodSeconds: 20
            timeoutSeconds: 2
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command:
                  - sh
                  - -c
                  - sleep 5
          volumeMounts:
            - name: web-content
              mountPath: /usr/share/nginx/html
              readOnly: true
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-run
              mountPath: /var/run
            - name: temporary-files
              mountPath: /tmp
      volumes:
        - name: web-content
          configMap:
            name: production-web-content
        - name: nginx-cache
          emptyDir:
            sizeLimit: 50Mi
        - name: nginx-run
          emptyDir:
            sizeLimit: 10Mi
        - name: temporary-files
          emptyDir:
            sizeLimit: 50Mi

Apply the Deployment.

kubectl apply -f 03-production-deployment.yaml

Monitor the rollout.

kubectl rollout status deployment/production-web \
  -n production-app

List the Pods.

kubectl get pods \
  -n production-app \
  -l app=production-web \
  -o wide

Expected:

• Three Pods are running.
• Each Pod is Ready.
• No Pod Security violations are reported.

Task 07 — Review the Deployment Strategy

Inspect the Deployment.

kubectl describe deployment production-web \
  -n production-app

Confirm:

• Replicas: 3
• RollingUpdate strategy enabled
• Maximum unavailable Pods: 1
• Maximum surge Pods: 1
• Minimum ready duration configured
• Revision history retained

Production Value

Rolling updates reduce deployment risk by replacing workloads gradually rather than stopping every replica simultaneously.

Task 08 — Create the Internal Service

Create 04-production-service.yaml.

apiVersion: v1
kind: Service
metadata:
  name: production-web
  namespace: production-app
  labels:
    app: production-web
    environment: production
spec:
  type: ClusterIP
  selector:
    app: production-web
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http

Apply it.

kubectl apply -f 04-production-service.yaml

Verify the Service.

kubectl get service production-web \
  -n production-app

Inspect its endpoints.

kubectl get endpoints production-web \
  -n production-app

Confirm:

• The Service type is ClusterIP.
• No direct NodePort is exposed.
• All Ready application replicas appear as endpoints.

Task 09 — Implement Default-Deny Network Policies

Create 05-default-deny-policy.yaml.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production-app
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: production-app
spec:
  podSelector: {}
  policyTypes:
    - Egress

Apply the policies.

kubectl apply -f 05-default-deny-policy.yaml

Verify them.

kubectl get networkpolicy -n production-app

Security Outcome

All ingress and egress communication is now denied unless an explicit allow policy exists.

Task 10 — Allow Approved Application Traffic

Create 06-allow-application-policy.yaml.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-ingress
  namespace: production-app
spec:
  podSelector:
    matchLabels:
      app: production-web
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              access: production-web
      ports:
        - protocol: TCP
          port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: production-app
spec:
  podSelector:
    matchLabels:
      app: production-web
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Apply the policies.

kubectl apply -f 06-allow-application-policy.yaml

Verify them.

kubectl describe networkpolicy \
  -n production-app

Task 11 — Create the PodDisruptionBudget

Create 07-pod-disruption-budget.yaml.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: production-web-pdb
  namespace: production-app
  labels:
    app: production-web
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: production-web

Apply it.

kubectl apply -f 07-pod-disruption-budget.yaml

Verify it.

kubectl get poddisruptionbudget \
  -n production-app

Describe it.

kubectl describe poddisruptionbudget production-web-pdb \
  -n production-app

Availability Outcome

During voluntary disruptions, Kubernetes should attempt to keep at least two application replicas available.

Task 12 — Create a Hardened Test Pod

Create 08-security-test-pod.yaml.

apiVersion: v1
kind: Pod
metadata:
  name: production-security-test
  namespace: production-app
  labels:
    access: production-web
spec:
  automountServiceAccountToken: false
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: tester
      image: busybox:1.36
      command:
        - sh
        - -c
        - sleep 3600
      securityContext:
        privileged: false
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
        limits:
          cpu: 50m
          memory: 32Mi
      volumeMounts:
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: tmp
      emptyDir:
        sizeLimit: 10Mi

Apply it.

kubectl apply -f 08-security-test-pod.yaml

Wait for readiness.

kubectl wait \
  --for=condition=Ready \
  pod/production-security-test \
  -n production-app \
  --timeout=120s

Task 13 — Validate Application Connectivity

Test access through the Service.

kubectl exec production-security-test \
  -n production-app \
  -- wget -qO- http://production-web

Expected output includes:

CloudNova Production Application

Confirm that the Service distributes traffic to Ready Pods.

Repeat the request several times.

kubectl exec production-security-test \
  -n production-app \
  -- sh -c "for i in 1 2 3 4 5; do wget -qO- http://production-web | grep CloudNova; done"

Task 14 — Validate Network Isolation

Create a second test Pod without the required access label.

kubectl run unauthorised-test \
  -n production-app \
  --image=busybox:1.36 \
  --labels=access=denied \
  --overrides='
{
  "spec": {
    "automountServiceAccountToken": false,
    "restartPolicy": "Never",
    "securityContext": {
      "runAsNonRoot": true,
      "runAsUser": 10002,
      "runAsGroup": 10002,
      "seccompProfile": {
        "type": "RuntimeDefault"
      }
    },
    "containers": [
      {
        "name": "unauthorised-test",
        "image": "busybox:1.36",
        "command": ["sh", "-c", "sleep 3600"],
        "securityContext": {
          "allowPrivilegeEscalation": false,
          "readOnlyRootFilesystem": true,
          "capabilities": {
            "drop": ["ALL"]
          }
        },
        "resources": {
          "requests": {
            "cpu": "10m",
            "memory": "16Mi"
          },
          "limits": {
            "cpu": "50m",
            "memory": "32Mi"
          }
        },
        "volumeMounts": [
          {
            "name": "tmp",
            "mountPath": "/tmp"
          }
        ]
      }
    ],
    "volumes": [
      {
        "name": "tmp",
        "emptyDir": {
          "sizeLimit": "10Mi"
        }
      }
    ]
  }
}'

Wait for the Pod.

kubectl wait \
  --for=condition=Ready \
  pod/unauthorised-test \
  -n production-app \
  --timeout=120s

Attempt to access the Service.

kubectl exec unauthorised-test \
  -n production-app \
  -- wget -T 5 -qO- http://production-web

Expected:

• The connection times out or is denied.
• The authorised test Pod can connect.
• The unauthorised test Pod cannot connect.

Task 15 — Store a Production Pod Name

Git Bash:

POD_NAME=$(kubectl get pod \
  -n production-app \
  -l app=production-web \
  -o jsonpath='{.items[0].metadata.name}')

PowerShell:

$POD_NAME = kubectl get pod `
  -n production-app `
  -l app=production-web `
  -o jsonpath='{.items[0].metadata.name}'

Confirm the value.

echo "$POD_NAME"

Task 16 — Validate Non-Root Execution

Check the runtime identity.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- id

Expected:

uid=101
gid=101

Confirm the process is not running as UID 0.

Inspect the configured values.

kubectl get pod "$POD_NAME" \
  -n production-app \
  -o jsonpath='{.spec.securityContext.runAsNonRoot}{"\n"}{.spec.securityContext.runAsUser}{"\n"}{.spec.securityContext.runAsGroup}{"\n"}'

Task 17 — Validate Privileged Mode

Inspect the setting.

kubectl get pod "$POD_NAME" \
  -n production-app \
  -o jsonpath='{.spec.containers[0].securityContext.privileged}{"\n"}'

Expected:

false

Confirm that no host namespaces are enabled.

kubectl get pod "$POD_NAME" \
  -n production-app \
  -o jsonpath='{.spec.hostPID}{"\n"}{.spec.hostIPC}{"\n"}{.spec.hostNetwork}{"\n"}'

Expected:

• Values are false or unset.

Task 18 — Validate Privilege Escalation Protection

Inspect the manifest setting.

kubectl get pod "$POD_NAME" \
  -n production-app \
  -o jsonpath='{.spec.containers[0].securityContext.allowPrivilegeEscalation}{"\n"}'

Expected:

false

Check runtime status.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "grep '^NoNewPrivs' /proc/1/status"

Expected:

NoNewPrivs: 1

Task 19 — Validate Dropped Linux Capabilities

Inspect the configured capability policy.

kubectl get pod "$POD_NAME" \
  -n production-app \
  -o jsonpath='{.spec.containers[0].securityContext.capabilities.drop}{"\n"}'

Expected:

["ALL"]

Check runtime capabilities.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "grep '^CapEff' /proc/1/status"

Expected:

CapEff: 0000000000000000

Task 20 — Validate the Read-Only Root Filesystem

Attempt to modify the system configuration.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "touch /etc/production-test"

Expected:

Read-only file system

Attempt to modify application content.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "echo compromised > /usr/share/nginx/html/index.html"

Expected:

• The write fails because the ConfigMap volume is mounted read-only.

Task 21 — Validate Approved Writable Storage

Write to the temporary directory.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "echo approved > /tmp/approved-write.txt"

Read the file.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- cat /tmp/approved-write.txt

Check the NGINX runtime directory.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "touch /var/run/approved-runtime-file"

Expected:

• Approved temporary writes succeed.
• Protected root filesystem writes fail.

Task 22 — Validate Seccomp Protection

Inspect the configured profile.

kubectl get pod "$POD_NAME" \
  -n production-app \
  -o jsonpath='{.spec.securityContext.seccompProfile.type}{"\n"}'

Expected:

RuntimeDefault

Review runtime seccomp status.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "grep '^Seccomp' /proc/1/status"

A typical protected process reports:

Seccomp: 2

Task 23 — Validate Service Account Token Protection

Inspect the setting.

kubectl get pod "$POD_NAME" \
  -n production-app \
  -o jsonpath='{.spec.automountServiceAccountToken}{"\n"}'

Expected:

false

Check the token path.

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "ls /var/run/secrets/kubernetes.io/serviceaccount"

Expected:

No such file or directory

Task 24 — Validate Resource Governance

Inspect the resource configuration.

kubectl get deployment production-web \
  -n production-app \
  -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'

Confirm:

Resource	Request	Limit
CPU	100m	300m
Memory	64Mi	128Mi

Describe a Pod.

kubectl describe pod "$POD_NAME" \
  -n production-app

Security and Reliability Value

Resource controls help prevent:

• CPU exhaustion
• Memory exhaustion
• Noisy-neighbour impact
• Uncontrolled application consumption
• Resource-based denial-of-service conditions

Task 25 — Validate Health Probes

Inspect the Deployment.

kubectl get deployment production-web \
  -n production-app \
  -o yaml

Confirm:

• Startup probe configured
• Readiness probe configured
• Liveness probe configured

Review Pod conditions.

kubectl get pod "$POD_NAME" \
  -n production-app \
  -o jsonpath='{.status.conditions}{"\n"}'

Review recent events.

kubectl describe pod "$POD_NAME" \
  -n production-app

Probe Responsibilities

Probe	Purpose
Startup	Allows slow applications to initialise safely
Readiness	Removes unhealthy Pods from Service endpoints
Liveness	Restarts containers that become unresponsive

Task 26 — Simulate a Failed Readiness Condition

Temporarily modify the readiness path in a copy of the manifest.

Example invalid path:

readinessProbe:
  httpGet:
    path: /not-found
    port: http

Apply only in the training environment.

Observe:

kubectl get pods -n production-app

kubectl get endpoints production-web -n production-app

Expected:

• Pods may continue running.
• Pods should not become Ready.
• Unready Pods should be removed from Service endpoints.

Restore the valid path:

path: /

Reapply the original Deployment.

kubectl apply -f 03-production-deployment.yaml

Verify recovery.

kubectl rollout status deployment/production-web \
  -n production-app

Task 27 — Validate Replica Resilience

List all replicas.

kubectl get pods \
  -n production-app \
  -l app=production-web

Delete one Pod.

kubectl delete pod "$POD_NAME" \
  -n production-app

Watch the Deployment recover.

kubectl get pods \
  -n production-app \
  -l app=production-web \
  --watch

Stop the watch after a replacement Pod becomes Ready.

Expected:

• The Deployment creates a replacement.
• Desired replica count returns to three.
• The Service continues to have available endpoints.

Task 28 — Validate the PodDisruptionBudget

Review PDB status.

kubectl get pdb production-web-pdb \
  -n production-app

Check:

• Desired healthy replicas
• Current healthy replicas
• Allowed disruptions

kubectl describe pdb production-web-pdb \
  -n production-app

Document how the PDB protects application availability during voluntary disruptions such as node maintenance.

Task 29 — Review Image Governance

Inspect the image reference.

kubectl get deployment production-web \
  -n production-app \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Confirm:

• The image does not use latest.
• An explicit version is configured.

For stronger production governance, organisations should use an immutable image digest.

Example:

image: registry.example.com/production-web@sha256:<approved-digest>

Production Recommendation

Use:

• Trusted private registries
• Image vulnerability scanning
• Signature verification
• Software bills of materials
• Immutable digests
• Deployment admission policies

Task 30 — Review Metadata and Ownership

Inspect resource labels.

kubectl get deployment production-web \
  -n production-app \
  --show-labels

kubectl get pods \
  -n production-app \
  --show-labels

Confirm that metadata identifies:

• Application
• Component
• Environment
• Owner
• Security tier
• Management method

Governance Value

Accurate metadata supports:

• Incident ownership
• Cost allocation
• Policy targeting
• Inventory management
• Compliance reporting
• Operational escalation

Task 31 — Simulate a Compromised Production Container

Assume an attacker has achieved command execution inside an application Pod.

Store a current Pod name again if required.

POD_NAME=$(kubectl get pod \
  -n production-app \
  -l app=production-web \
  -o jsonpath='{.items[0].metadata.name}')

Attempt to Operate as Root

kubectl exec "$POD_NAME" \
  -n production-app \
  -- id

Expected:

• Non-root user.

Attempt to Modify System Files

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "echo attacker >> /etc/passwd"

Expected:

Read-only file system

Attempt to Create a Device Node

kubectl exec "$POD_NAME" \
  -n production-app \
  -- mknod /tmp/test-device c 1 3

Expected:

Operation not permitted

Attempt to Locate Kubernetes Credentials

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "find /var/run/secrets -type f 2>/dev/null"

Expected:

• No automatically mounted Service Account token.

Attempt to Modify Application Content

kubectl exec "$POD_NAME" \
  -n production-app \
  -- sh -c "echo malicious > /usr/share/nginx/html/index.html"

Expected:

• The write is denied.

Task 32 — Analyse the Reduced Attack Surface

Attack Technique	Security Control	Expected Outcome
Operate as root	runAsNonRoot and explicit UID	Prevented
Gain additional privilege	allowPrivilegeEscalation: false	Prevented
Use kernel capabilities	Drop ALL	Restricted
Modify system files	Read-only root filesystem	Blocked
Modify application content	Read-only ConfigMap mount	Blocked
Abuse Kubernetes API token	Token automount disabled	Prevented
Use unrestricted syscalls	RuntimeDefault seccomp	Reduced
Consume excessive resources	CPU and memory limits	Constrained
Receive unauthorised traffic	Network Policies	Blocked
Cause full outage during maintenance	Multiple replicas and PDB	Reduced
Serve traffic while unhealthy	Readiness probe	Prevented
Remain permanently unresponsive	Liveness probe	Restarted

Task 33 — Review Production Workload Configuration

Export the final Deployment.

kubectl get deployment production-web \
  -n production-app \
  -o yaml

Validate:

automountServiceAccountToken: false

runAsNonRoot: true

runAsUser: 101

runAsGroup: 101

seccompProfile:
  type: RuntimeDefault

privileged: false

allowPrivilegeEscalation: false

readOnlyRootFilesystem: true

capabilities:
  drop:
    - ALL

Also confirm:

• Explicit image version
• Three replicas
• RollingUpdate strategy
• Startup probe
• Readiness probe
• Liveness probe
• Resource requests
• Resource limits
• Controlled writable volumes
• Internal ClusterIP Service
• Network Policies
• PodDisruptionBudget
• Ownership labels

Task 34 — Perform the Production Security Assessment

Complete the following scorecard.

Security Domain	Validation	Status
Pod Security Standard	Restricted enforcement
Non-root execution	UID 101 confirmed
Privileged mode	Disabled
Privilege escalation	Disabled
Linux capabilities	All dropped
Seccomp	RuntimeDefault active
Root filesystem	Read-only
Writable directories	Explicitly mounted
Service Account token	Not mounted
Host namespace access	Disabled
HostPath volumes	Not present
Image governance	Explicit version
Resource governance	Requests and limits set
Health monitoring	Three probes configured
Service exposure	ClusterIP
Network isolation	Default deny plus explicit allow
Availability	Three replicas plus PDB
Metadata governance	Ownership and environment labels
Runtime validation	Completed

Use one of the following classifications:

• Production Ready
• Conditionally Approved
• Not Production Ready

Task 35 — Identify Residual Risks

Even after hardening, review the following residual risks:

• Application vulnerabilities
• Vulnerable base image packages
• Supply-chain compromise
• Weak application authentication
• Missing TLS at the Ingress layer
• Unmonitored runtime behaviour
• Missing image signature verification
• Inadequate secrets management
• Excessively permissive application dependencies
• Lack of backup or recovery testing

Document compensating controls and ownership for each applicable risk.

Task 36 — Produce the Production Readiness Report

Create production-readiness-report.md.

Assessment Title:
Production Pod Hardening and Readiness Review

Cluster Name:

Namespace:
production-app

Assessment Date:

Assessor:

Application:
production-web

Business Owner:

Technical Owner:

Security Owner:

Container Image:

Image Version or Digest:

Replica Count:

Pod Security Standard:

Non-Root Validation:

Privilege Escalation Validation:

Capability Validation:

Seccomp Validation:

Read-Only Filesystem Validation:

Writable Storage Review:

Service Account Review:

Resource Governance:

Health Probe Review:

Network Policy Review:

Service Exposure Review:

PodDisruptionBudget Review:

Runtime Security Tests:

Admission Control Results:

Critical Findings:

High Findings:

Medium Findings:

Residual Risks:

Required Remediation:

Security Exceptions:

Overall Security Rating:

Production Decision:

Production Ready

Conditionally Approved

Not Production Ready

Approvals:

Application Owner:

Platform Engineering:

Cloud Security:

Operations:

Task 37 — Evidence Collection

Collect evidence for:

• Cluster and node health
• Namespace labels
• Restricted admission rejection
• ConfigMap manifest
• Hardened Deployment manifest
• Service manifest
• Network Policy manifests
• PodDisruptionBudget manifest
• Pod inventory
• Runtime identity
• Privilege escalation validation
• Capability validation
• Read-only filesystem testing
• Writable directory validation
• Seccomp validation
• Service Account token validation
• Resource configuration
• Health probe configuration
• Authorised connectivity test
• Unauthorised connectivity denial
• Replica recovery test
• PDB status
• Final readiness report

Suggested evidence names:

01-cluster-health.txt
02-production-namespace-labels.txt
03-admission-control-rejection.txt
04-production-configmap.yaml
05-production-deployment.yaml
06-production-service.yaml
07-network-policies.yaml
08-pod-disruption-budget.yaml
09-runtime-identity.txt
10-privilege-escalation.txt
11-capabilities.txt
12-readonly-filesystem.txt
13-seccomp-status.txt
14-service-account-token.txt
15-resource-governance.txt
16-health-probes.txt
17-authorised-connectivity.txt
18-unauthorised-connectivity.txt
19-replica-recovery.txt
20-production-readiness-report.md

Task 38 — Clean Up

Delete the unauthorised test Pod.

kubectl delete pod unauthorised-test \
  -n production-app \
  --ignore-not-found

Delete the authorised test Pod.

kubectl delete pod production-security-test \
  -n production-app

Delete the application resources.

kubectl delete deployment production-web \
  -n production-app

kubectl delete service production-web \
  -n production-app

Delete the Network Policies.

kubectl delete networkpolicy --all \
  -n production-app

Delete the PodDisruptionBudget.

kubectl delete poddisruptionbudget production-web-pdb \
  -n production-app

Delete the ConfigMap.

kubectl delete configmap production-web-content \
  -n production-app

Delete the namespace.

kubectl delete namespace production-app

Verify cleanup.

kubectl get namespace production-app

Expected:

NotFound

Enterprise Production Pod Checklist

Control	Status
Restricted Pod Security Standard enforced	☐
Non-root execution configured	☐
Explicit UID and GID configured	☐
Privileged mode disabled	☐
Privilege escalation disabled	☐
All Linux capabilities dropped	☐
RuntimeDefault seccomp enabled	☐
Root filesystem read-only	☐
Writable paths explicitly mounted	☐
Service Account token disabled	☐
Host namespaces disabled	☐
HostPath volumes absent	☐
Explicit image version or digest used	☐
Trusted image source used	☐
Resource requests configured	☐
Resource limits configured	☐
Startup probe configured	☐
Readiness probe configured	☐
Liveness probe configured	☐
Multiple replicas configured	☐
Rolling update configured	☐
PodDisruptionBudget configured	☐
Internal ClusterIP Service used	☐
Default-deny Network Policies configured	☐
Approved communication explicitly allowed	☐
Ownership metadata configured	☐
Runtime controls validated	☐
Evidence collected	☐
Production approval documented	☐

Risk Classification

Critical

Examples:

• Privileged production container
• Root execution with broad capabilities
• HostPath mounted to sensitive node paths
• Publicly exposed database or administrative Service
• No admission control for production workloads
• Unrestricted cross-namespace access

High

Examples:

• Privilege escalation enabled
• Writable root filesystem
• Missing seccomp profile
• Service Account token mounted unnecessarily
• No Network Policies
• Image using an untrusted registry
• No health probes for customer-facing applications

Medium

Examples:

• Missing resource limits
• Single application replica
• Missing PodDisruptionBudget
• Incomplete ownership metadata
• Floating image tag
• Incomplete runtime evidence

Low

Examples:

• Naming inconsistencies
• Missing descriptive annotations
• Evidence-format improvements
• Documentation gaps

Remediation Priorities

Immediate

• Remove privileged and root execution.
• Disable privilege escalation.
• Apply default-deny Network Policies.
• Remove unnecessary external Service exposure.
• Disable unnecessary Service Account tokens.
• Reject workloads that fail the Restricted Pod Security Standard.

Short-Term

• Enable read-only root filesystems.
• Drop unnecessary capabilities.
• Configure RuntimeDefault seccomp.
• Add resource controls and health probes.
• Add multiple replicas and disruption protection.
• Pin images to approved versions or digests.

Long-Term

• Add image signature verification.
• Enforce policies using Kyverno or Gatekeeper.
• Integrate workload scanning into CI/CD.
• Implement runtime threat detection.
• Apply GitOps-based configuration management.
• Automate production readiness evidence.
• Perform recurring workload security assessments.

Skills Developed

By completing this lab, you will be able to:

• Design hardened production Kubernetes workloads
• Enforce Restricted Pod Security Standards
• Configure defence-in-depth container security
• Implement immutable filesystem patterns
• Apply least-privilege runtime access
• Protect Kubernetes API credentials
• Configure resource governance
• Implement Kubernetes health probes
• Apply application network isolation
• Protect application availability
• Test authorised and unauthorised communication
• Validate runtime security controls
• Assess residual production risk
• Produce enterprise production readiness reports
• Approve or reject workloads for production

Knowledge Check

Question 1

Which combination best represents a hardened production container?

• A. Root user, privileged mode, writable filesystem
• B. Non-root user, privilege escalation disabled, capabilities dropped, read-only filesystem
• C. Host networking, host PID, and HostPath access
• D. No resource limits and no health probes

Answer: B

Question 2

What is the purpose of a readiness probe?

• A. Restart the Kubernetes node
• B. Determine whether a Pod should receive Service traffic
• C. Increase CPU limits
• D. Create a new namespace

Answer: B

Question 3

Why should a production Service normally use ClusterIP behind an Ingress Controller?

• A. To expose every Pod directly to the internet
• B. To keep backend workloads internal and centralise external access
• C. To disable Kubernetes networking
• D. To remove TLS support

Answer: B

Question 4

What is the purpose of a PodDisruptionBudget?

• A. Restrict Linux capabilities
• B. Protect application availability during voluntary disruptions
• C. Configure container users
• D. Scan container images

Answer: B

Question 5

Why are both resource requests and limits important?

• A. They encrypt container traffic.
• B. Requests support scheduling, while limits constrain maximum resource usage.
• C. They replace Network Policies.
• D. They disable Service Account tokens.

Answer: B

Question 6

Which control prevents unauthorised Pods from connecting to the production application?

• A. ConfigMap
• B. NetworkPolicy
• C. ReplicaSet
• D. PersistentVolume

Answer: B

Question 7

Why should container images use explicit versions or immutable digests?

• A. To make DNS faster
• B. To ensure predictable and auditable deployments
• C. To enable privileged mode
• D. To remove resource limits

Answer: B

Question 8

What does automountServiceAccountToken: false achieve?

• A. It prevents unnecessary Kubernetes API credentials from being mounted into the Pod.
• B. It disables application networking.
• C. It removes the container image.
• D. It creates a PodDisruptionBudget.

Answer: A

Question 9

Which probe should protect an application that requires additional time to initialise?

• A. Startup probe
• B. Network Policy
• C. Resource limit
• D. Pod Security admission

Answer: A

Question 10

What is the most appropriate production decision when critical workload security findings remain unresolved?

• A. Production Ready
• B. Ignore the findings
• C. Not Production Ready
• D. Increase the replica count only

Answer: C

Lab Summary

In this lab, you built and validated a hardened production Kubernetes workload using defence-in-depth security and reliability controls.

You implemented:

• Restricted Pod Security enforcement
• Non-root execution
• Disabled privileged mode
• Disabled privilege escalation
• Dropped Linux capabilities
• RuntimeDefault seccomp
• A read-only root filesystem
• Controlled writable storage
• Disabled Service Account token mounting
• CPU and memory governance
• Startup, readiness, and liveness probes
• Multiple application replicas
• Rolling updates
• Internal ClusterIP exposure
• Default-deny Network Policies
• Explicit application access
• A PodDisruptionBudget
• Production metadata and ownership
• Runtime attack simulation
• Enterprise readiness assessment

These controls reduce the likelihood and impact of container compromise while improving application availability, operational consistency, auditability, and production resilience.

A production-ready Pod is not secured by a single YAML setting. It requires coordinated controls across image security, admission, runtime privileges, filesystem access, networking, availability, monitoring, and governance.

What’s Next?

Next Lab: Lab 05 — Runtime Security Validation

In the next lab, you will validate Kubernetes workloads during runtime by monitoring process execution, filesystem changes, network activity, privilege use, suspicious commands, container drift, and indicators of compromise using enterprise runtime security techniques and tools.