Lab 03 — Deploy Prometheus & Grafana

Lab Overview

Monitoring is one of the most important responsibilities of a Cloud Security Engineer.

A secure Kubernetes cluster is not simply deployed—it must be continuously monitored for:

• Infrastructure health
• Cluster performance
• Security events
• Resource utilisation
• Application availability
• Capacity planning
• Operational anomalies

In enterprise environments, Prometheus collects metrics while Grafana transforms those metrics into interactive dashboards.

Together they provide real-time visibility across Amazon EKS clusters and are commonly integrated with SIEM, Security Hub and enterprise SOC platforms.

By the end of this lab, you will deploy Prometheus and Grafana, collect Kubernetes metrics and build your first enterprise monitoring dashboard.

---

Mission Information

Mission Objective

Deploy Prometheus and Grafana on Amazon EKS and monitor the health, performance and security of Kubernetes workloads using enterprise dashboards.

---

Estimated Time

75–90 Minutes

---

Difficulty

Intermediate

---

Lab Type

• Amazon EKS
• Kubernetes Monitoring
• Observability
• Prometheus
• Grafana
• Security Operations

---

Learning Objectives

After completing this lab, you will be able to:

• Deploy Prometheus
• Deploy Grafana
• Verify monitoring components
• Collect Kubernetes metrics
• Access Grafana dashboards
• Create custom dashboards
• Understand enterprise observability architecture

---

Business Scenario

You are a Cloud Security Engineer working for a global e-commerce company.

The organisation recently migrated hundreds of applications to Amazon EKS.

Although Audit Logs and Falco are already deployed, operations teams have limited visibility into:

• Cluster health
• CPU utilisation
• Memory usage
• Pod failures
• Node availability
• Container resource consumption

Management requires a centralized monitoring platform that provides real-time dashboards for both Operations and Security teams.

You have been tasked with deploying Prometheus and Grafana across the production monitoring environment.

---

Prerequisites

Before beginning this lab, ensure you have:

• AWS Account
• Amazon EKS Cluster
• kubectl installed
• Helm installed
• AWS CLI configured
• Cluster administrator access

Verify cluster connectivity.

kubectl get nodes

Expected output:

STATUS

Ready

---

Lab Architecture

Applications

↓

Pods

↓

Node Exporter

↓

kube-state-metrics

↓

Prometheus

↓

Grafana

↓

SOC Dashboard

---

Enterprise Architecture

Amazon EKS

↓

Prometheus

↓

Amazon Managed Grafana

↓

CloudWatch

↓

Enterprise SIEM

↓

SOC

---

Lab Tasks

Task	Description
Task 1	Verify Cluster Health
Task 2	Install Monitoring Repository
Task 3	Deploy Prometheus Stack
Task 4	Verify Components
Task 5	Access Grafana
Task 6	Explore Dashboards
Task 7	Create Security Dashboard
Task 8	Enterprise Monitoring Review

---

Task 1 — Verify Cluster Health

Confirm your Amazon EKS cluster is operational.

kubectl get nodes

Check namespaces.

kubectl get namespaces

Verify Pods.

kubectl get pods -A

---

Task 2 — Add the Helm Repository

Add the Prometheus Community repository.

helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts

Update Helm.

helm repo update

Verify repository.

helm search repo kube-prometheus-stack

Example output:

prometheus-community/kube-prometheus-stack

---

Understanding kube-prometheus-stack

The Helm chart installs:

• Prometheus
• Grafana
• Alertmanager
• Node Exporter
• kube-state-metrics
• Prometheus Operator

This provides a complete enterprise monitoring platform.

---

Task 3 — Deploy the Monitoring Stack

Create the monitoring namespace.

kubectl create namespace monitoring

Deploy the stack.

helm install monitoring \
prometheus-community/kube-prometheus-stack \
--namespace monitoring

Deployment may take several minutes.

---

Task 4 — Verify Installation

Verify Pods.

kubectl get pods -n monitoring

Example:

grafana

Running

prometheus

Running

alertmanager

Running

node-exporter

Running

kube-state-metrics

Running

Verify services.

kubectl get svc -n monitoring

Verify deployments.

kubectl get deployments -n monitoring

---

Understanding Monitoring Components

Prometheus

Responsible for:

• Collecting metrics
• Storing time-series data
• Executing PromQL queries

---

Grafana

Responsible for:

• Dashboards
• Charts
• Alert visualisation
• Monitoring reports

---

Alertmanager

Responsible for:

• Alert routing
• Notification management
• Alert grouping
• Alert suppression

---

Node Exporter

Collects metrics including:

• CPU
• Memory
• Disk
• Network
• File systems

---

kube-state-metrics

Collects Kubernetes object metrics including:

• Pods
• Deployments
• Nodes
• StatefulSets
• DaemonSets
• ReplicaSets

---

Task 5 — Access Grafana

Retrieve the Grafana administrator password.

kubectl get secret \
monitoring-grafana \
-n monitoring \
-o jsonpath="{.data.admin-password}" \
| base64 --decode

Forward the Grafana service.

kubectl port-forward \
svc/monitoring-grafana \
3000:80 \
-n monitoring

Open your browser.

http://localhost:3000

Default username:

admin

Password:

Use the password retrieved from the Kubernetes Secret.

---

Task 6 — Explore Dashboards

Browse the pre-built dashboards.

Common dashboards include:

• Kubernetes Cluster Overview
• Node Health
• Pod Metrics
• Namespace Resource Usage
• API Server Metrics
• Kubernetes Networking
• CPU Utilisation
• Memory Usage

Spend time exploring the available visualisations.

---

Task 7 — Create a Security Dashboard

Create a new Grafana dashboard.

Add panels for:

• Cluster CPU Utilisation
• Node Memory Usage
• Pod Restart Count
• Node Availability
• Container CPU Usage
• Namespace Resource Consumption
• Failed Pods
• Kubernetes API Server Availability

Save the dashboard as:

Enterprise Kubernetes Security Dashboard

This dashboard provides operational visibility for both platform engineers and SOC analysts.

---

Example PromQL Queries

CPU Usage

rate(container_cpu_usage_seconds_total[5m])

Memory Usage

container_memory_usage_bytes

Running Pods

kube_pod_status_phase

Node Availability

up

These queries form the basis of many enterprise dashboards.

---

Task 8 — Enterprise Monitoring Review

Imagine your organisation operates:

• 400 Amazon EKS clusters
• 6 AWS Regions
• Thousands of Kubernetes workloads

Your monitoring platform should answer questions such as:

• Which cluster is unhealthy?
• Which node has high CPU utilisation?
• Which Pods are restarting frequently?
• Which namespaces consume the most resources?
• Are Kubernetes API Servers healthy?
• Are worker nodes available?
• Is abnormal resource usage occurring?
• Should alerts be triggered?

Enterprise monitoring enables rapid operational decision-making.

---

Validation Checklist

Confirm that:

• Helm repository added successfully
• Monitoring namespace created
• Prometheus deployed
• Grafana deployed
• Alertmanager running
• Node Exporter running
• kube-state-metrics running
• Grafana accessible
• Dashboards displaying metrics

---

Common Issues

Pods Not Starting

Check Pod status.

kubectl get pods -n monitoring

Describe the Pod.

kubectl describe pod <pod-name> -n monitoring

Review logs.

kubectl logs <pod-name> -n monitoring

---

Grafana Login Failed

Retrieve the administrator password again.

kubectl get secret monitoring-grafana \
-n monitoring \
-o jsonpath="{.data.admin-password}" \
| base64 --decode

Ensure the port-forward session is active.

---

No Metrics Displayed

Verify Prometheus is running.

kubectl get pods -n monitoring

Confirm Prometheus targets are healthy by opening the Prometheus UI if required.

---

Enterprise Best Practices

As a Cloud Security Engineer:

• Deploy Prometheus across every production cluster.
• Secure Grafana using enterprise authentication such as IAM, SSO or OIDC.
• Restrict dashboard access using RBAC.
• Configure Prometheus data retention according to organisational policies.
• Integrate Alertmanager with enterprise notification platforms.
• Build dashboards for Operations, Security and Management teams.
• Monitor resource trends to identify capacity and security anomalies.
• Back up Grafana dashboards and configuration.
• Regularly review monitoring coverage.
• Use Amazon Managed Prometheus and Amazon Managed Grafana where appropriate to reduce operational overhead.

---

Real-World Scenario

A multinational banking organisation observes increased latency in one of its customer-facing applications.

Grafana dashboards reveal:

• A sharp increase in CPU utilisation on several worker nodes.
• Multiple Pods entering a CrashLoopBackOff state.
• Memory consumption nearing configured limits.
• An increase in container restart counts.

Prometheus metrics are correlated with Falco runtime alerts and Kubernetes Audit Logs.

The SOC determines that an unauthorised cryptocurrency mining process was executing inside a compromised container.

The affected workload is isolated, new Pods are deployed from trusted images and monitoring thresholds are adjusted to detect similar behaviour in the future.

Because comprehensive monitoring was already in place, the issue is identified and resolved before customer services are significantly affected.

---

Lab Summary

Congratulations!

You have successfully deployed Prometheus and Grafana on Amazon EKS.

During this lab you learned how to:

• Deploy the Kubernetes monitoring stack
• Collect infrastructure and Kubernetes metrics
• Access Grafana dashboards
• Create custom monitoring dashboards
• Understand enterprise observability
• Support security monitoring and operational visibility

These skills are essential for Cloud Security Engineers, Platform Engineers, DevSecOps Engineers and SOC Analysts responsible for operating enterprise Kubernetes environments.

---

Key Takeaways

• Prometheus is the standard metrics collection platform for Kubernetes.
• Grafana transforms metrics into real-time operational dashboards.
• Alertmanager automates alert routing and notifications.
• Node Exporter and kube-state-metrics provide infrastructure and Kubernetes telemetry.
• Enterprise organisations rely on Prometheus and Grafana for continuous monitoring, capacity planning and security operations.

---

What’s Next?

In the next lab, you will investigate a simulated Kubernetes security incident by combining Audit Logs, Falco alerts, Prometheus metrics and Grafana dashboards to perform a complete SOC-style incident investigation.

➡️ Next Lab: Lab 04 — Investigate a Kubernetes Security Incident