Project 04 — Secure Multi-Tenant Kubernetes Cluster

Project Overview

Welcome to Project 04 — Secure Multi-Tenant Kubernetes Cluster.

In this enterprise project, you will design and secure a Kubernetes platform that safely hosts multiple teams, applications and business units within the same cluster.

Most enterprise organizations do not deploy one Kubernetes cluster per application.

Instead, they operate shared enterprise Kubernetes platforms where hundreds of teams deploy workloads into the same cluster.

This introduces several security challenges:

Tenant isolation
Identity separation
Resource governance
Network segmentation
Secret isolation
Compliance
Access governance
Platform administration

Your responsibility is to build a production-ready multi-tenant Kubernetes platform that follows enterprise security best practices.

Project Mission

CloudNova Technologies is consolidating multiple Kubernetes clusters into one enterprise platform to reduce operational costs and simplify management.

As the Kubernetes Security Engineer, your responsibilities are to:

Design tenant isolation
Implement secure namespaces
Enforce least-privilege access
Prevent tenant-to-tenant communication
Protect sensitive workloads
Govern resource usage
Secure workload identities
Collect audit evidence
Produce production-ready documentation

Business Scenario

CloudNova Technologies currently has separate Kubernetes clusters for every development team.

Management has decided to migrate to a shared enterprise Kubernetes platform.

The following departments will share the cluster:

Customer Portal Team
Payments Team
HR Applications
Finance Applications
DevOps Platform
SOC Platform
Security Engineering
AI Engineering
Data Analytics

The platform must ensure one tenant cannot impact another.

Business Requirements

The enterprise platform must:

Support multiple business units
Enforce namespace isolation
Implement least privilege
Prevent lateral movement
Support secure CI/CD
Protect sensitive workloads
Provide centralized logging
Support compliance requirements
Enable secure monitoring
Scale to hundreds of namespaces

Project Objectives

By completing this project you will learn how to:

Design enterprise multi-tenancy
Build namespace isolation
Configure tenant RBAC
Implement Pod Security Admission
Configure ResourceQuotas
Configure LimitRanges
Secure Service Accounts
Implement Network Policies
Separate workload identities
Secure Secrets
Prevent privilege escalation
Produce enterprise governance documentation

Estimated Time

14–20 Hours

Difficulty

Advanced

Recommended Tools

Amazon EKS
kubectl
eksctl
Terraform
Helm
Kyverno
OPA Gatekeeper
AWS IAM
EKS Pod Identity / IRSA
AWS Secrets Manager
Prometheus
Grafana
Falco
AWS CloudWatch
AWS Security Hub

Enterprise Architecture

Amazon EKS Cluster

│
├── kube-system
├── monitoring
├── security
├── platform
│
├── customer-portal
├── payments
├── hr
├── finance
├── ai
├── analytics
├── development
└── testing

Each Namespace

↓

Dedicated RBAC

↓

Dedicated Service Accounts

↓

Dedicated Network Policies

↓

Dedicated Secrets

↓

Dedicated Resource Quotas

↓

Dedicated Logging

Security Architecture

Enterprise Identity

↓

AWS IAM Identity Center

↓

Temporary IAM Roles

↓

EKS Access Entries

↓

Namespace RBAC

↓

Service Accounts

↓

Pod Identity

↓

Applications

↓

AWS Resources

Project Deliverables

Technical

Namespace Architecture
RBAC Policies
ResourceQuota Policies
LimitRanges
Network Policies
Pod Security Policies
Service Accounts
Secrets Architecture

Security

Threat Model
Tenant Isolation Design
Risk Register
Security Assessment
Compliance Mapping

Management

Architecture Document
Tenant Governance Guide
Production Readiness Report
Executive Summary

Project Folder Structure

04-secure-multi-tenant-cluster/

├── README.md
├── 01-requirements/
├── 02-architecture/
├── 03-namespaces/
├── 04-rbac/
├── 05-network-policies/
├── 06-service-accounts/
├── 07-resource-governance/
├── 08-security/
├── 09-monitoring/
├── 10-evidence/
├── 11-report/
└── 12-runbooks/

Phase 1 — Requirements

Task 1.1 — Identify Tenants

Document:

Tenant	Owner	Environment
Customer Portal	Digital Team	Production
Payments	Finance	Production
HR	HR Team	Production
Finance	Finance Team	Production
AI	AI Engineering	Production
Analytics	Data Team	Production

Task 1.2 — Classify Workloads

Assign classifications:

Public
Internal
Confidential
Restricted

Task 1.3 — Identify Trust Boundaries

Define:

User access
Namespace boundaries
Application boundaries
Network boundaries
AWS account boundaries

Phase 2 — Namespace Design

Create namespaces:

customer-portal

payments

finance

hr

analytics

ai

platform

security

monitoring

Label every namespace.

Example:

metadata:
  labels:
    owner: payments
    environment: production
    classification: confidential

Phase 3 — RBAC Design

Each tenant receives:

Namespace Admin
Developer
Read Only
Auditor

Example Role:

kind: Role

rules:

- apiGroups:
  - apps

  resources:
  - deployments

  verbs:

  - get
  - list
  - watch
  - create
  - update

Avoid ClusterRole where namespace Role is sufficient.

Phase 4 — Service Accounts

Every application receives its own Service Account.

Never use:

default

Example:

serviceAccountName: payment-api

Disable automatic token mounting unless required.

Phase 5 — Resource Governance

Create ResourceQuotas.

Example:

cpu: 20

memory: 64Gi

pods: 100

Create LimitRanges.

Prevent resource exhaustion.

Phase 6 — Pod Security

Apply Pod Security Admission.

Restricted

Validate:

Non-root
Read-only filesystem
Drop capabilities
No privilege escalation
RuntimeDefault seccomp

Phase 7 — Network Isolation

Every namespace receives:

Default Deny

↓

Allow DNS

↓

Allow Monitoring

↓

Allow Required Application Traffic

↓

Deny Everything Else

Example:

kind: NetworkPolicy

policyTypes:

- Ingress

- Egress

Validate:

Customer Portal

↓

Payment API

Allowed

Customer Portal

↓

HR Database

Denied

Phase 8 — Secrets Isolation

Every tenant receives:

Dedicated Secrets

↓

Dedicated IAM Role

↓

Dedicated KMS Key

↓

Dedicated Access Policy

Never share Secrets across tenants.

Phase 9 — Workload Identity

Use:

EKS Pod Identity

IRSA

Each workload gets:

Dedicated IAM Role

↓

Least Privilege

↓

CloudTrail Logging

Phase 10 — Logging

Enable:

Audit Logs
CloudWatch
Prometheus
Grafana
Falco

Create dashboards per tenant.

Phase 11 — Monitoring

Monitor:

CPU
Memory
Pods
Restarts
Privilege Escalation
Runtime Events
Network Policy Violations

Phase 12 — Security Validation

Validate:

Namespace Isolation

Tenant cannot list Pods in another namespace.

kubectl auth can-i list pods \
-n finance \
--as developer

Secrets

Tenant cannot read Secrets.

kubectl auth can-i get secrets

Network

Connectivity tests:

Allowed

↓

Customer Portal → Payment API

Denied

↓

Customer Portal → HR Database

Pod Security

Deploy:

Privileged Pod

Expected:

Denied

Resource Governance

Attempt:

100 CPU Pods

Expected:

Quota exceeded

Threat Model

Threats include:

Tenant escape
Privilege escalation
Secret theft
Lateral movement
Resource exhaustion
Compromised developer account
Malicious container
Supply-chain attack

Evidence Collection

Collect:

Namespace YAML
RBAC YAML
Network Policies
ResourceQuota
LimitRange
Service Accounts
IAM Policies
kubectl outputs
Screenshots
CloudWatch Logs
Falco Alerts

Findings Template

Finding ID

Severity

Affected Tenant

Description

Evidence

Risk

Recommendation

Owner

Status

Example Finding

Finding:

TENANT-001

Severity:

High

Issue:

Finance namespace has no Network Policy.

Risk:

Tenant workloads can communicate with other namespaces.

Recommendation:

Implement default deny Network Policy.

Production Readiness Checklist

Executive Dashboard

Domain	Status
Namespace Isolation	✅
RBAC	✅
Secrets	✅
Network Policies	✅
Pod Security	✅
Resource Governance	✅
Runtime Security	✅
Monitoring	✅

Knowledge Check

1. Why is namespace isolation important?

Answer: It prevents tenants from accessing each other’s workloads, resources and sensitive information, reducing the risk of lateral movement.

2. Why should every application use its own Service Account?

Answer: Dedicated Service Accounts enable least-privilege access, improve auditability and limit the impact if a workload is compromised.

3. Why are ResourceQuotas important in a multi-tenant cluster?

Answer: They prevent one tenant from consuming excessive CPU, memory or other cluster resources, ensuring fair resource allocation and platform stability.

4. Why should Network Policies follow a default-deny approach?

Answer: Default-deny Network Policies block all traffic by default and require only approved communication paths to be explicitly allowed, reducing the attack surface.

5. Why should Pod Security Admission be enforced for every tenant?

Answer: It provides a consistent baseline that prevents insecure workload configurations such as privileged containers, root execution and unrestricted Linux capabilities.

Project Success Criteria

Portfolio Outcome

By completing this project, you will demonstrate the ability to:

Design secure enterprise multi-tenant Kubernetes platforms
Implement namespace and tenant isolation
Apply Zero Trust principles within shared clusters
Govern compute resources across multiple business units
Secure identities, workloads and networks using enterprise best practices
Produce professional architecture, governance and security documentation suitable for real-world enterprise environments

➡️ Next Project: Project 05 — Build a Production Kubernetes Security Architecture