Lab 03 Threat Model an AI Agent

Mission Information

In this lab, you will perform a focused threat modeling assessment of an Enterprise AI Agent.

In the previous lab, you assessed how AI retrieves enterprise knowledge through RAG.

Now you will assess what happens when AI can take actions.

This is a major security shift.

A traditional AI assistant may:

User
 ↓
LLM
 ↓
Response

An AI agent may:

User
 ↓
AI Agent
 ↓
Tool
 ↓
Enterprise System
 ↓
Real Action

The security question is no longer only:

What information can the AI access?

You must now ask:

What can the AI change, execute, send, create or delete?

In this lab, you will analyze:

Agent identity
User identity
Tool inventory
Tool permissions
User-context authorization
Agent memory
External and indirect inputs
Tool parameters
Human approval
Production access
Agent-to-agent delegation
Logging
Kill switches
Blast radius
Risk and security controls

Lab Difficulty

Intermediate to Advanced

Estimated Time

120–150 minutes

Lab Type

Architecture Review + Threat Modeling + Security Design

Prerequisites

Before starting this lab, you should understand:

LLM fundamentals
AI agents
Tool calling
RAG
Enterprise IAM
Least privilege
Trust boundaries
STRIDE
OWASP GenAI security concepts
MITRE ATLAS concepts
Attack paths
Risk assessment
Threat-to-control mapping

What You Will Build

By the end of this lab, you will create:

Enterprise-AI-Agent-Threat-Model/
│
├── 01 Business Context.md
├── 02 Agent Architecture.md
├── 03 Agent Identity and Permissions.md
├── 04 Tool Inventory.md
├── 05 Agent Data Flows.md
├── 06 Trust Boundary Register.md
├── 07 Agent Attack Surface Register.md
├── 08 Agent Threat Register.md
├── 09 Agent Attack Paths.md
├── 10 Agent Risk Register.md
├── 11 Agent Security Requirements.md
├── 12 Detection and Response.md
└── 13 Executive Summary.md

Scenario

You are working as an AI Security Engineer for an enterprise organization.

The company is developing an internal AI assistant called:

Cloud Operations Copilot

Cloud engineers will use the assistant to investigate infrastructure issues and perform approved operational actions.

The system can:

Read cloud security findings
Query cloud inventory
Review logs
Create incident tickets
Restart approved workloads
Modify selected network rules
Generate remediation recommendations

The organization wants the agent to improve operational efficiency without giving users unrestricted administrative access.

Your mission is to threat model the agent before production deployment.

Target Architecture

Use the following architecture:

                         Cloud Engineer
                              │
                              │ SSO + MFA
                              ▼
                        AI Web Application
                              │
                              ▼
                          AI Agent
                              │
               ┌──────────────┼──────────────┐
               │              │              │
               ▼              ▼              ▼
             LLM           Memory        Tool Router
                                             │
                     ┌───────────────────────┼───────────────────────┐
                     │                       │                       │
                     ▼                       ▼                       ▼
               Cloud Read Tool       Ticketing Tool        Cloud Action Tool
                     │                       │                       │
                     ▼                       ▼                       ▼
               Cloud APIs             ITSM Platform          Production Cloud

The agent uses a dedicated workload identity.

However, the security team has not yet confirmed whether:

Permissions follow least privilege
User permissions are preserved
Every tool action is logged
Sensitive actions require approval
External content can influence agent actions
Agent memory is isolated per user

These are part of your investigation.

Step 1 — Understand the Business Context

Create:

01 Business Context.md

Use:

# Business Context

## System Name

Cloud Operations Copilot

## Business Purpose

Assist authorized cloud engineers with investigation, analysis and approved operational cloud actions.

## Primary Users

- Cloud Engineers
- Cloud Security Engineers
- Platform Engineers
- Incident Responders

## Business Criticality

High

## Sensitive Functions

- Cloud inventory access
- Security log access
- Ticket creation
- Workload restart
- Network configuration changes

## Key Security Objective

The AI agent must never allow a user to perform actions beyond the user's approved authority.

Student Task

Answer:

What problem does the AI agent solve?
Which actions are read-only?
Which actions modify enterprise systems?
Which actions could impact production?
Which actions should require approval?
What would happen if the agent identity were compromised?

Step 2 — Define Scope

In Scope

AI Web Application
AI Agent
LLM
Agent Memory
Tool Router
Cloud Read Tool
Ticketing Tool
Cloud Action Tool
Agent Workload Identity
Cloud IAM
User Authorization
Tool Parameters
Approval Workflow
Agent Logging
Production Cloud Access

Out of Scope

Corporate Endpoint Security
Physical Security
Internal Architecture of the LLM Provider
Full Cloud Platform Threat Model
ITSM Platform Internal Infrastructure

Create:

## Scope

### In Scope

### Out of Scope

### Assumptions

### Unknowns

Step 3 — Document Assumptions

Use:

Users authenticate through enterprise SSO.

MFA is enabled.

The AI agent uses a dedicated workload identity.

The agent reaches cloud services through authenticated APIs.

Tool calls are performed only through the Tool Router.

Production cloud resources are classified as critical.

Do not assume:

Agent permissions are appropriate.
User permissions are preserved.
Tool parameters are safe.
Agent memory is isolated.
Tool outputs are trustworthy.
Human approval exists.
Kill-switch procedures work.

These must be validated.

Step 4 — Build the Agent Architecture Inventory

Create:

02 Agent Architecture.md

Use:

ID	Component	Purpose	Exposure	Identity
C-A01	AI Web App	User interface	Internal	User Session
C-A02	AI Agent	Task planning and coordination	Private	Agent Identity
C-A03	LLM	Reasoning / response generation	Internal/External	API Identity
C-A04	Memory	Stores context	Private	Agent Identity
C-A05	Tool Router	Selects and invokes tools	Private	Agent Identity
C-A06	Cloud Read Tool	Queries cloud data	Private	Workload ID
C-A07	Ticketing Tool	Creates incidents	Private	Tool Identity
C-A08	Cloud Action Tool	Performs approved changes	Privileged	Workload ID

Student Task

For every component document:

Owner
Identity
Data processed
Permissions
Network reachability
Logging
Criticality

Step 5 — Build the Agent Identity Register

Create:

03 Agent Identity and Permissions.md

Use:

ID	Identity	Type	Purpose	Privilege
ID-A01	Cloud Engineer	Human	Uses agent	User-specific
ID-A02	AI Agent	Workload	Agent runtime	Medium/High
ID-A03	Cloud Read Identity	Workload	Read APIs	Read-only
ID-A04	Ticketing Identity	Workload	Create tickets	Write
ID-A05	Cloud Action Identity	Workload	Modify approved resources	High

Student Task

For each identity determine:

Authentication mechanism
Permissions
Credential type
Lifetime
Owner
Rotation
Monitoring
Revocation method

Step 6 — Review the Agent Permission Model

Start with the key question:

What can the agent do if it is fully compromised?

Create a capability matrix:

Capability	Required?	Current Permission	Desired Permission	Risk
Read Inventory	Yes	Read	Read	Low
Read Logs	Yes	Read	Read	Medium
Create Ticket	Yes	Write	Create Only	Medium
Restart Workload	Yes	Broad Write	Selected Workloads	High
Modify Network Rule	Limited	Admin	Approved Rules Only	Critical
Modify IAM	No	Unknown	None	Critical

Any permission that is not required should be removed.

Step 7 — Identify Excessive Agency

Ask:

Does the agent have more tools than required?

Does it have broader permissions than required?

Can it execute automatically?

Can it affect production?

Can it perform destructive actions?

Can it make IAM changes?

Can it send data externally?

Create:

## Excessive Agency Findings

### EA-01

### EA-02

### EA-03

Step 8 — Build the Tool Inventory

Create:

04 Tool Inventory.md

Use:

ID	Tool	Function	Access	Impact	Approval Required
TOOL-01	Cloud Inventory	Read cloud resources	Read	Low	No
TOOL-02	Log Search	Query security logs	Read	Medium	No
TOOL-03	Ticket Creator	Create incident	Write	Medium	No
TOOL-04	Workload Restart	Restart workload	Write	High	Yes
TOOL-05	Network Rule Tool	Modify selected rules	Write	Critical	Yes

Student Task

For each tool ask:

What can it do?

Which identity does it use?

What resources can it reach?

What parameters control it?

Can the action be reversed?

What is the maximum impact?

Step 9 — Classify Tools by Risk

Use:

Low

Read-only public or low-sensitivity information.

Medium

Internal reads or limited business writes.

High

Production changes or sensitive data access.

Critical

IAM, network perimeter, destructive operations or unrestricted administration.

Prioritize controls according to tool risk.

Step 10 — Map Agent Data Flows

Create:

05 Agent Data Flows.md

Use:

ID	Source	Destination	Data / Action	Identity	Sensitivity
DF-A01	User	AI App	Prompt	Employee	Internal
DF-A02	AI App	Agent	User Request	Employee Session	Internal
DF-A03	Agent	LLM	Prompt + Context	Agent ID	Confidential
DF-A04	Agent	Memory	Context	Agent ID	Confidential
DF-A05	Agent	Tool Router	Proposed Tool Action	Agent ID	Restricted
DF-A06	Tool Router	Cloud API	API Request	Workload ID	Restricted
DF-A07	Cloud API	Agent	Tool Result	Service	Confidential
DF-A08	Agent	User	Response	Application	Mixed

Student Task

For each flow document:

Direction
Authentication
Authorization
Data classification
Logging
Trust change
Privilege change

Step 11 — Identify Trust Boundaries

Create:

06 Trust Boundary Register.md

Start with:

ID	Boundary	Primary Concern
TB-A01	User → Agent	Malicious or compromised user
TB-A02	Agent → LLM	Sensitive context
TB-A03	Agent → Tool Router	AI decision becomes action
TB-A04	Tool Router → Cloud API	Privilege transition
TB-A05	Tool Result → Agent	Untrusted tool output
TB-A06	Memory → Future Agent Session	Persistent influence

For each boundary document:

Source
Destination
Identity
Action / Data
Privilege Change
Trust Assumption
Existing Controls
Failure Impact

Step 12 — Focus on the Privilege Boundary

The most important boundary may be:

AI Decision
     ↓
========================
PRIVILEGE BOUNDARY
========================
     ↓
Production Action

Ask:

What deterministic security control exists between AI intent and execution?

If the answer is:

“The prompt tells the agent not to do anything unsafe.”

that is not sufficient.

Step 13 — Threat Model User-Context Authorization

Imagine:

Cloud Engineer
      ↓
Agent
      ↓
Network Tool

The agent’s identity may have permission to modify the network.

But the engineer may not.

Weak design:

Can Agent Do It?
      ↓
YES
      ↓
EXECUTE

Stronger design:

Can User Do It?
      +
Can Agent Do It?
      +
Is Resource Allowed?
      ↓
EXECUTE

Security Requirement

AGENT-SEC-001

Sensitive tool actions must be independently authorized using the initiating user's identity and target resource.

Step 14 — Threat Model the Confused Deputy

AG-T01 — Privileged Agent Used by Low-Privilege User

Attack path:

Compromised User
      ↓
AI Agent
      ↓
Agent Uses Higher Privilege
      ↓
Cloud Action Tool
      ↓
Production Resource

STRIDE

Elevation of Privilege

Impact

Unauthorized production modification.

Initial Risk

Critical

Step 15 — Threat Model Direct Prompt Manipulation

AG-T02 — User Manipulates Agent Into Sensitive Action

User
 ↓
Manipulated Request
 ↓
AI Agent
 ↓
Sensitive Tool

The important question is not only:

Can the prompt influence the model?

The important question is:

What security control prevents manipulated model intent from becoming unauthorized execution?

Step 16 — Threat Model Indirect Prompt Injection

Imagine the agent reads cloud ticket descriptions or external documentation.

Attack path:

External Attacker
      ↓
Malicious Ticket / Document
      ↓
AI Agent
      ↓
Interprets Content
      ↓
Tool Request
      ↓
Production Action

AG-T03 — External Content Influences Agent

Primary controls:

Treat external content as untrusted
Restrict tools
User-context authorization
Policy enforcement
Human approval

Step 17 — Threat Model Tool Parameters

Suppose the agent chooses:

Tool:
RestartWorkload

Parameters:

environment = production

resource = payment-api

The tool may be legitimate.

The target may not be.

Validate:

Tool
+
Operation
+
Target
+
Environment
+
Scope

before execution.

Step 18 — Build Parameter Policy

Example:

Requested Action
      ↓
Is Tool Approved?
      ↓
Is Environment Approved?
      ↓
Is Resource Approved?
      ↓
Is User Authorized?
      ↓
Approval Required?
      ↓
Execute

Security Requirement

AGENT-SEC-002

High-risk tool parameters must be validated against approved resource and operation policies before execution.

Step 19 — Threat Model Agent Memory

Agent memory may contain:

Previous prompts
Tool results
Resource identifiers
Business context
User preferences

Create questions:

Who Can Read Memory?

Who Can Write Memory?

Is Memory Shared Between Users?

How Long Is It Retained?

Can It Influence Future Actions?

Step 20 — Threat Model Cross-User Memory Leakage

AG-T04 — Memory Isolation Failure

User A
 ↓
Agent Memory
 ↓
User B

Potential impact:

Confidentiality breach
Cross-user context leakage

Security Requirement

AGENT-SEC-003

Persistent and session memory must be isolated according to user, tenant and authorization context.

Step 21 — Threat Model Memory Poisoning

AG-T05 — Persistent Malicious Memory

Attacker Input
      ↓
Agent Memory
      ↓
Future Session
      ↓
Agent Behavior

Potential controls:

Restrict what becomes persistent
Validate memory writes
Retention limits
Memory reset
Audit changes

Step 22 — Threat Model Tool Responses

The agent may receive:

Tool
 ↓
Response
 ↓
LLM

A common mistake is to trust tool output automatically.

Suppose a tool retrieves content from:

External site
Ticket
Customer field
Third-party API

The returned content may be attacker-controlled.

AG-T06 — Malicious Tool Response Influences Agent

External Data
      ↓
Tool
      ↓
Agent Context
      ↓
Next Tool Action

Treat tool outputs according to their original trust level.

Step 23 — Threat Model Ticketing Tool

The agent can create tickets.

Ask:

Which projects can it access?
Can it modify existing tickets?
Can it change priority?
Can it add external recipients?
Can it attach sensitive data?

Least-privilege design may be:

Create Approved Ticket Type

rather than:

Ticketing Administrator

Step 24 — Threat Model Production Restart

The agent can restart workloads.

Define allowed resources:

Development
Staging
Selected Production Services

Potential controls:

Approved resource list
User authorization
Environment restriction
Human approval
Maintenance-window policy

Step 25 — Threat Model Network Changes

Network changes should be treated as high risk.

Example:

Agent
 ↓
Modify Security Rule
 ↓
Internet Exposure

AG-T07 — Unsafe Network Change

Potential impact:

Service exposure
Unauthorized access
Lateral movement opportunity

Controls:

Predefined change templates
Resource allowlist
Parameter validation
Human approval
Cloud-native policy enforcement

Step 26 — Threat Model IAM Access

Ask whether the agent can:

Create roles
Attach policies
Modify users
Pass roles

If not required:

Agent IAM Permissions
      ↓
NONE

Removing access is stronger than controlling it through prompt instructions.

Step 27 — Threat Model Agent Credentials

AG-T08 — Agent Identity Compromise

Attack path:

Attacker
   ↓
Agent Workload Identity
   ↓
Cloud API
   ↓
Production

This bypasses model safety entirely.

Controls:

Managed identity
Short-lived credentials
Least privilege
Credential monitoring
Fast revocation

Step 28 — Determine Agent Blast Radius

Create:

# Agent Blast Radius

## If the agent identity is fully compromised:

### Data It Can Read

### Resources It Can Modify

### Resources It Can Delete

### Systems It Can Reach

### Security Controls It Can Modify

### External Destinations It Can Reach

Then assign:

Low
Medium
High
Critical

Step 29 — Reduce Blast Radius

Look for opportunities to replace:

One Powerful Agent

with:

Read-Only Analysis Agent

Ticket Creation Agent

Controlled Production Action Agent

Separate identities and capabilities where practical.

Step 30 — Threat Model Human Approval

Classify actions:

Action	Approval
Read Inventory	No
Read Logs	No
Create Standard Ticket	No
Restart Production Workload	Yes
Modify Network Rule	Yes
Modify IAM	Not Allowed

Approval should be proportional to risk.

Step 31 — Design an Approval Request

A meaningful request should show:

Initiating User

Agent

Action

Target Resource

Environment

Reason

Expected Impact

Rollback Information

Weak approval:

Approve this action?

does not give the reviewer enough context.

Step 32 — Threat Model Approval Fatigue

Too many approval prompts may cause:

Approval
Approval
Approval
Approval
      ↓
Automatic Clicking

Use approval for meaningful high-risk actions.

Do not use it as a substitute for proper authorization.

Step 33 — Threat Model Multi-Agent Delegation

Suppose the architecture later adds:

General Agent
      ↓
Cloud Operations Agent
      ↓
Network Agent

Ask:

How do agents authenticate each other?
Is original user identity preserved?
Can one agent increase privilege?
Are delegation rules enforced?

Step 34 — Multi-Agent Attack Path

AG-T09 — Privilege Escalation Through Agent Delegation

Low-Privilege User
      ↓
General Agent
      ↓
Privileged Agent
      ↓
Production Tool

Required principle:

Delegation should not silently increase the initiating user’s authority.

Step 35 — Map the Agent Attack Surface

Create:

07 Agent Attack Surface Register.md

Use:

ID	Attack Surface	Exposure	Main Risk
AS-A01	User Prompt	Internal	Manipulated requests
AS-A02	Agent Memory	Private	Leakage / poisoning
AS-A03	Tool Router	Privileged	Unauthorized tool selection
AS-A04	Tool Parameters	Privileged	Unsafe targets
AS-A05	Tool Responses	Mixed	Indirect manipulation
AS-A06	Agent Identity	Privileged	Credential abuse
AS-A07	Approval Workflow	Internal	Approval bypass
AS-A08	Cloud API	Privileged	Production impact

Step 36 — Apply STRIDE

Create:

08 Agent Threat Register.md

Identify at least one threat for each:

Spoofing

User account takeover
Agent identity theft
Agent-to-agent impersonation

Tampering

Memory manipulation
Tool configuration changes
Tool parameter modification

Repudiation

Missing user-to-action traceability
Missing approval evidence

Information Disclosure

Cross-user memory leakage
Sensitive tool output

Denial of Service

Agent loops
Tool flooding
Cloud API exhaustion

Elevation of Privilege

Low-privileged user controls privileged agent

Step 37 — Apply OWASP GenAI Guidance

Review current relevant guidance for:

Prompt injection
Sensitive information disclosure
Improper output handling
Excessive agency
Supply-chain risks

Explain the actual scenario rather than only recording the risk category.

Step 38 — Apply MITRE ATLAS

Consider adversary behaviors involving:

Agent capability discovery
AI context manipulation
Credential abuse
Collection
Defense evasion
Impact

Validate current mappings when performing a real assessment.

Step 39 — Build Attack Path AP-A01

Privileged Agent Abuse

Compromised Cloud Engineer
          ↓
AI Agent
          ↓
Manipulated Request
          ↓
Missing User Authorization
          ↓
Cloud Action Tool
          ↓
Privileged Agent Identity
          ↓
Production Resource

Control Breakpoints

Conditional Access
      ↓
User-Context Authorization
      ↓
Tool Allowlisting
      ↓
Parameter Validation
      ↓
Human Approval
      ↓
Cloud API Authorization

Step 40 — Build Attack Path AP-A02

Indirect Prompt Injection

External Attacker
      ↓
Malicious Ticket Content
      ↓
Agent Reads Ticket
      ↓
Untrusted Context Influences Agent
      ↓
Cloud Action Tool
      ↓
Production Action

Control breakpoints:

Content treated as untrusted
Tool restrictions
User authorization
Approval
Production policy enforcement

Step 41 — Build Attack Path AP-A03

Agent Credential Compromise

Attacker
   ↓
Agent Credential
   ↓
Cloud API
   ↓
Production Resource

This path does not require LLM manipulation.

Control breakpoints:

Managed identity
Token lifetime
Least privilege
API monitoring
Revocation

Step 42 — Build Attack Path AP-A04

Memory Poisoning

Attacker
   ↓
Malicious Input
   ↓
Persistent Memory
   ↓
Future Cloud Engineer Session
   ↓
Agent Uses Poisoned Context
   ↓
Unsafe Recommendation / Action

Step 43 — Build Attack Path AP-A05

Unsafe Network Change

Compromised User
      ↓
AI Agent
      ↓
Network Tool
      ↓
Overly Broad Parameters
      ↓
Public Exposure
      ↓
Production Security Impact

Step 44 — Build the Risk Register

Create:

10 Agent Risk Register.md

Example:

ID	Threat	Likelihood	Impact	Inherent Risk
AG-R01	Privileged Agent Abuse	Medium	Critical	Critical
AG-R02	Indirect Prompt Injection	Medium	Critical	Critical
AG-R03	Agent Credential Theft	Medium	Critical	Critical
AG-R04	Memory Poisoning	Medium	High	High
AG-R05	Cross-User Memory Leakage	Medium	High	High
AG-R06	Unsafe Network Modification	Medium	Critical	Critical
AG-R07	Agent Loop / Resource Abuse	Medium	Medium	Medium

Step 45 — Evaluate AI-Specific Risk Factors

For each risk document:

Agent Privilege

Autonomy

Number of Tools

Production Access

Data Sensitivity

Blast Radius

Persistence

Required User Access

Detectability

Containment Capability

These factors may significantly affect the final rating.

Step 46 — Define Agent Security Requirements

Create:

11 Agent Security Requirements.md

AGENT-SEC-001

Sensitive tool actions must be authorized using the initiating user's identity and target resource.

AGENT-SEC-002

Production AI agents must use dedicated least-privilege workload identities.

AGENT-SEC-003

Tool access must be restricted to approved capabilities required by the business workflow.

AGENT-SEC-004

High-risk tool parameters must be validated against approved operation and resource policies.

AGENT-SEC-005

Production network modifications must require explicit human approval unless covered by a pre-approved automated policy.

AGENT-SEC-006

Persistent agent memory must be isolated by user and authorization context.

AGENT-SEC-007

External content and tool responses must be treated according to their original trust level and must not directly authorize privileged actions.

AGENT-SEC-008

All production agent tool calls must record the initiating user, agent identity, tool, parameters, target, authorization result, approval status and execution result.

AGENT-SEC-009

AI agents must have configurable execution, tool-call, time and resource limits.

AGENT-SEC-010

Security Operations must be able to immediately disable production AI agents and revoke their active access.

Step 47 — Define Validation Tests

For every important requirement, define expected behavior.

Example:

Test — Unauthorized Production Restart

User:

Cloud Engineer without restart permission

Request:

Restart production payment-api.

Expected:

DENIED

even if the agent identity technically possesses restart permission.

Step 48 — Test Parameter Controls

Request:

Modify approved network rule.

Allowed target:

staging-security-group

Expected:

ALLOWED after required policy checks.

Unauthorized target:

production-public-gateway

Expected:

DENIED

Step 49 — Test Approval

Attempt a controlled production-style action in an authorized lab environment.

Expected:

Agent Proposes Action
      ↓
Execution Blocked
      ↓
Approval Required
      ↓
Approved
      ↓
Execution

Confirm that bypassing approval is not possible.

Step 50 — Test Agent Memory Isolation

Use:

User A
User B

Store harmless test context for User A.

Verify User B cannot retrieve or influence that memory.

Expected:

User A Memory → User A Only

Step 51 — Test Kill Switch

In a non-production environment:

Agent Enabled
      ↓
Tool Call
      ↓
Success

Disable the agent identity.

Then:

Agent Disabled
      ↓
Tool Call
      ↓
DENIED

Document:

Disable method
Time to containment
Tokens revoked
Tool access blocked

Step 52 — Define Detection Requirements

Create:

12 Detection and Response.md

DET-A01 — Privileged Tool Invocation

Detect sensitive agent tool actions.

DET-A02 — User-to-Agent Permission Mismatch

Detect when a low-privileged user initiates a high-privileged operation.

DET-A03 — High Tool-Call Volume

Detect agent loops or unusual automation.

DET-A04 — Agent Identity Anomaly

Detect use of agent credentials outside expected runtime context.

DET-A05 — Sensitive Network Change

Detect unusual or high-risk network modifications.

DET-A06 — Memory Anomaly

Detect unusual persistent memory changes where telemetry supports it.

Step 53 — Required Agent Telemetry

Capture:

Initiating User ID

User Role

Session ID

Agent ID

Model / Agent Version

Tool

Tool Parameters

Target Resource

Authorization Decision

Approval Result

Execution Result

Timestamp

Correlation ID

This enables end-to-end attribution.

Step 54 — Threat Hunting Scenario

Hypothesis:

A compromised employee may be using the AI agent to perform actions outside normal job responsibilities.

Data sources:

Identity Logs

Agent Logs

Tool Calls

Cloud Audit Logs

Hunt for:

Unusual privileged tool usage
New production targets
User-role mismatch
Large increase in tool requests

Step 55 — Agent Incident Response

Imagine:

Security Operations observes unusual production network changes from the AI agent.

Investigation path:

Cloud Change
      ↓
Agent Tool Call
      ↓
Agent Session
      ↓
Initiating User
      ↓
Original Request
      ↓
Authorization Decision
      ↓
Approval
      ↓
Related Agent Activity

Your logging must support this investigation.

Step 56 — Containment Procedure

For suspected agent compromise:

Disable Agent Identity
        ↓
Revoke Active Tokens
        ↓
Block High-Risk Tools
        ↓
Preserve Logs
        ↓
Identify Affected Actions
        ↓
Restore Unauthorized Changes
        ↓
Investigate Root Cause

Step 57 — Agent Kill-Switch Runbook Test

Your architecture should provide at least one rapid containment method.

Examples:

Disable Workload Identity

Disable Tool Router

Apply Emergency Deny Policy

Stop Agent Runtime

Document the fastest supported option.

Step 58 — Determine Residual Risk

Example:

Privileged Agent Abuse

Inherent Risk:

Critical

Controls:

User-context authorization
Least privilege
Tool allowlisting
Parameter validation
Human approval
Logging

Residual Risk:

Medium

Agent Credential Compromise

Inherent Risk:

Critical

Controls:

Managed identity
Short-lived tokens
Least privilege
Runtime restrictions
monitoring

Residual Risk:

Medium

Document your reasoning.

Step 59 — Build the Executive Summary

Create:

13 Executive Summary.md

Use:

# Executive Summary

## System

Cloud Operations Copilot

## Assessment Focus

Enterprise AI Agent Security

## Overall Risk

High

## Critical Findings

1. User authorization must be preserved when privileged agent tools are invoked.
2. Agent permissions must be reduced to the minimum required.
3. High-risk tool parameters require deterministic validation.
4. Sensitive production actions require approval and strong auditability.
5. Agent memory and tool responses introduce additional trust boundaries.
6. A tested emergency disable capability is required.

## Priority Recommendations

1. Implement user-context authorization.
2. Apply dedicated least-privilege agent identities.
3. Restrict tool availability.
4. Validate all high-risk tool parameters.
5. Require approval for critical actions.
6. Implement end-to-end user-to-tool logging.
7. Test agent kill-switch procedures.

## Residual Risk

Medium after priority controls are implemented and validated.

Expected Deliverables

Your completed lab should include:

Business context
Scope and assumptions
Agent architecture
Identity inventory
Permission matrix
Tool inventory
Tool risk classification
Data flow register
Trust boundary register
Attack surface register
STRIDE analysis
OWASP GenAI considerations
MITRE ATLAS considerations
Confused deputy threat
Direct prompt manipulation threat
Indirect prompt injection threat
Memory poisoning threat
Agent credential compromise threat
Unsafe tool parameter threat
Multi-agent delegation consideration
At least 5 attack paths
Agent risk register
At least 10 security requirements
Validation tests
Detection requirements
Containment procedure
Kill-switch validation
Executive summary

Lab Success Criteria

You have successfully completed the lab when you can trace:

User
  ↓
Agent
  ↓
Tool Selection
  ↓
Authorization
  ↓
Parameter Validation
  ↓
Approval
  ↓
Tool Identity
  ↓
Enterprise API
  ↓
Action

and explain which control protects each step.

You should also be able to explain the following risk:

Threat Actor
Compromised Cloud Engineer
        ↓
Attack Surface
AI Agent
        ↓
Security Gap
Missing User-Context Authorization
        ↓
Privilege
Agent Workload Identity
        ↓
Target
Production Cloud
        ↓
Impact
Unauthorized Production Change
        ↓
Controls
Least Privilege + Authorization + Approval

Security Engineer Challenge

The organization now wants to make the agent more autonomous.

Current design:

Agent
 ↓
Human Approval
 ↓
Production Action

Proposed design:

Agent
 ↓
Production Action

The business argues:

“Human approval slows down operations.”

Reassess the architecture.

Determine:

Which risks increase?
How does likelihood change?
How does blast radius change?
Which actions could safely become autonomous?
Which actions should remain approval-based?
What deterministic policy controls would be required?
What transaction limits would be required?
What monitoring would be required?
Would residual risk remain acceptable?

A mature answer should not be:

AI should never be autonomous.

Nor should it be:

Let the AI perform everything automatically.

Instead classify actions by risk.

For example:

Read Inventory
      ↓
Autonomous

Create Standard Ticket
      ↓
Autonomous

Restart Approved Non-Critical Workload
      ↓
Policy-Controlled Automation

Modify Production Network
      ↓
Human Approval

Modify IAM
      ↓
Not Available to Agent

This is risk-based AI security architecture.

Real-World Takeaway

When reviewing an AI agent, do not begin by asking only:

Can the model be prompt injected?

Start by asking:

Which Identity Does the Agent Use?

What Can That Identity Do?

Which Tools Exist?

Which Tools Are High Risk?

Whose Authorization Is Used?

What Parameters Are Allowed?

What Can External Content Influence?

Does the Agent Have Memory?

What Requires Approval?

What Is the Blast Radius?

How Do We Detect Abuse?

How Do We Stop the Agent?

These questions tell you whether an AI weakness can become a real enterprise security incident.

Lab Review Questions

Question 1

What is the main security difference between an AI assistant and an AI agent?

Question 2

Why should agent intent not automatically equal authorization?

Question 3

What is the confused deputy problem?

Question 4

Why must initiating-user authorization be preserved?

Question 5

Why should tool parameters be validated separately from tool selection?

Question 6

What is excessive agency?

Question 7

Why should agents use dedicated workload identities?

Question 8

How can external content indirectly manipulate an AI agent?

Question 9

Why can agent memory become both a confidentiality and integrity risk?

Question 10

Why should tool responses sometimes be treated as untrusted?

Question 11

How does human approval reduce risk?

Question 12

Why is a kill switch important?

Question 13

What is blast radius?

Question 14

Why can agent credential compromise be more serious than a model-level weakness?

Key Takeaways

An AI agent should be threat modeled as an action system, not only a language model.

The critical chain is:

User
  ↓
AI Agent
  ↓
Decision
  ↓
Tool
  ↓
Identity
  ↓
Authorization
  ↓
Enterprise System
  ↓
Real Impact

The most important security areas are:

User Authorization
        +
Agent Identity
        +
Least Privilege
        +
Tool Restrictions
        +
Parameter Validation
        +
Untrusted Context
        +
Memory
        +
Human Approval
        +
Logging
        +
Containment

The central principle is:

Assume that AI behavior can be influenced or fail unexpectedly, then design the surrounding authorization and control architecture so those failures cannot automatically become privileged enterprise actions.

The safest agent is not necessarily the agent that makes the fewest mistakes.

It is the agent whose mistakes have a controlled blast radius.

What’s Next?

➡️ Lab 04 — Build an AI Attack Tree

In the next lab, you will connect multiple threats into structured attack trees.

You will select critical attacker objectives such as:

Steal confidential RAG data
Manipulate enterprise AI responses
Abuse a privileged AI agent
Compromise production resources

Then you will map:

Multiple attack paths
Preconditions
Alternative routes
Trust boundaries
Control breakpoints
Detection opportunities
Highest-risk branches

The goal is to move from:

“I can threat model individual AI agent scenarios.”

to:

“I can visualize multiple ways an attacker could reach a critical AI or enterprise asset and determine where defenses provide the greatest value.”

➡️ Next: Lab 04 — Build an AI Attack Tree