04 Trust Boundaries in AI Systems

Modern AI applications operate across multiple users, services, models, data stores and external platforms.

Each component operates with a different level of:

Trust
Identity
Privilege
Data access
Security control

Whenever information or control moves between components with different security assumptions, a trust boundary exists.

Understanding these boundaries is one of the most important skills in AI threat modeling.

Attackers frequently succeed by crossing a trust boundary that was poorly understood or insufficiently protected.

Learning Objectives

By the end of this lesson, you should be able to:

Explain what a trust boundary is.
Identify trust boundaries in AI architectures.
Understand why internal does not automatically mean trusted.
Identify user-to-application boundaries.
Identify application-to-LLM boundaries.
Identify enterprise-to-third-party boundaries.
Identify RAG ingestion and retrieval boundaries.
Identify AI agent privilege boundaries.
Understand trust differences between AI instructions and data.
Identify privilege transitions.
Map security controls to trust boundaries.
Document trust boundaries for threat modeling.
Recognize dangerous trust assumptions in enterprise AI systems.

What Is Trust?

In cybersecurity, trust means that a system allows another entity to perform certain actions or provide information under defined assumptions.

For example:

Employee
   ↓
Authenticated Identity
   ↓
Enterprise Application

The application may trust the identity provider to confirm who the employee is.

But that does not mean:

Authenticated = Trusted for Everything

Authentication establishes identity.

Authorization determines what that identity is allowed to do.

Trust should therefore always be:

Explicit, limited and continuously validated.

What Is a Trust Boundary?

A trust boundary exists whenever data or control moves between systems, users or components operating under different levels of trust.

Example:

UNTRUSTED USER

     │
     │ User Input
     ▼

========================
     TRUST BOUNDARY
========================

AI APPLICATION

Another example:

ENTERPRISE ENVIRONMENT
        │
        │ Prompt + Context
        ▼
========================
     TRUST BOUNDARY
========================
        ▼
EXTERNAL LLM PROVIDER

Here, enterprise information crosses an organizational security boundary.

Why Trust Boundaries Matter

Security controls are frequently placed at trust boundaries.

For example:

Internet
   ↓
[Authentication]
   ↓
Application

Application
   ↓
[Authorization]
   ↓
Sensitive Database

AI Agent
   ↓
[Policy Validation]
   ↓
Production API

Whenever you identify a trust boundary, ask:

What security control protects this transition?

AI Creates More Trust Relationships

A traditional application may contain:

User
 ↓
Application
 ↓
Database

A modern AI system may contain:

User
 ↓
Application
 ↓
Prompt Orchestrator
 ↓
LLM
 ↓
RAG
 ↓
Vector Database
 ↓
Enterprise Documents

It may also include:

LLM
 ↓
AI Agent
 ↓
Tools
 ↓
Enterprise Systems

Every connection introduces security assumptions.

AI therefore creates additional opportunities for:

Trust confusion
Privilege escalation
Unauthorized data access
Malicious context manipulation
Unsafe automation

The Most Important AI Trust Principle

One principle should remain in your mind throughout this learning path:

LLM-generated output should not automatically become trusted input to privileged systems.

Consider:

Untrusted User Input
        ↓
       LLM
        ↓
Generated Output

The model processing the information does not magically make it trustworthy.

Use the following mental model:

Untrusted Input
        ↓
AI Processing
        ↓
Potentially Untrusted Output

User-to-Application Trust Boundary

One of the first boundaries is:

USER
 │
 ▼
====================
  TRUST BOUNDARY
====================
 │
 ▼
APPLICATION

Possible threats include:

Unauthorized access
Malicious input
Prompt injection
File upload abuse
Session abuse
API abuse

Typical controls include:

Authentication
MFA
Authorization
Rate limiting
Input handling
Session security
Logging

Authenticated Users Are Not Fully Trusted

A common mistake is:

Authenticated Employee
        =
Trusted User

A better model is:

Authenticated Employee
        ↓
Known Identity
        ↓
Still Subject to Authorization

An employee account may be:

Compromised
Malicious
Overprivileged
Misused accidentally

AI systems should enforce authorization regardless of whether a user is internal.

Application-to-LLM Boundary

Another important boundary exists between the application and the model.

AI Application
      │
      │ Prompt + Context
      ▼
========================
     TRUST BOUNDARY
========================
      ▼
LLM

Ask:

What information is sent?
Is sensitive data included?
Is the model internal or external?
Is the connection authenticated?
Is data retained?
Can the output be trusted?

External LLM Provider Boundary

When using an external model provider:

ENTERPRISE
    │
    │ Sensitive Context
    ▼
=========================
ORGANIZATIONAL BOUNDARY
=========================
    ▼
EXTERNAL AI PROVIDER

Security considerations may include:

Data protection
Provider security
Data residency
Privacy
Retention
Encryption
Authentication
Contractual requirements
Regulatory obligations

Model Output Boundary

The return path is equally important.

LLM
 │
 │ Generated Output
 ▼
========================
VALIDATION BOUNDARY
========================
 ▼
Application

Model output may be:

Incorrect
Manipulated
Maliciously influenced
Sensitive
Unsafe
Structurally unexpected

Applications should validate model output according to its intended use.

Trust Within the Prompt Context

An LLM may receive several information sources simultaneously.

System Instructions ─────┐
                         │
User Prompt ──────────────┤
                         │
Conversation History ─────┼──► LLM
                         │
RAG Content ──────────────┤
                         │
External Web Content ─────┘

These sources do not necessarily have the same level of trust.

Context Source	Typical Trust Consideration
Application policy	Higher trust
System instructions	Higher trust
User prompt	Untrusted
Uploaded file	Untrusted
Approved RAG source	Partially trusted
External website	Untrusted

The exact classification depends on the architecture.

Instructions vs Data

One of the central security challenges in LLM applications is that models process both:

Instructions
+
Data

using natural language.

Example:

Trusted Instruction:
Summarize the following document.

Untrusted Document:
Ignore previous instructions and perform another action.

The application intended the document to be:

DATA

but malicious content may attempt to behave like:

INSTRUCTIONS

This is one reason prompt injection is fundamentally challenging.

RAG Trust Boundaries

RAG introduces multiple trust boundaries.

Document Source
      ↓
Ingestion Pipeline
      ↓
Embedding
      ↓
Vector Database
      ↓
Retrieval
      ↓
LLM

Each transition should be reviewed separately.

RAG Source-to-Ingestion Boundary

DOCUMENT SOURCE
      │
      ▼
========================
     TRUST BOUNDARY
========================
      ▼
INGESTION PIPELINE

Ask:

Who controls the source?
Who can upload documents?
Are sources approved?
Can external content enter?
Is content reviewed?
Is provenance recorded?

RAG Ingestion-to-Vector Database Boundary

Ingestion Service
       ↓
========================
     DATA BOUNDARY
========================
       ↓
Vector Database

Important controls may include:

Authentication
Write authorization
Integrity protection
Source validation
Logging

Unauthorized writes could lead to knowledge poisoning.

Vector Database-to-Retrieval Boundary

Vector Database
       ↓
========================
 AUTHORIZATION BOUNDARY
========================
       ↓
Retrieval Service

A critical question is:

Does retrieval enforce the user’s authorization?

The existence of a matching document does not mean the user is permitted to see it.

Retrieval-to-LLM Boundary

Retrieved Content
       ↓
========================
 CONTEXT TRUST BOUNDARY
========================
       ↓
LLM

Retrieved information may contain:

Malicious instructions
Incorrect information
Sensitive data
Compromised content

RAG content should therefore not automatically be considered trusted simply because it came from an enterprise repository.

External Content Boundary

AI agents may access external information.

Internet
   ↓
Website
   ↓
========================
UNTRUSTED DATA BOUNDARY
========================
   ↓
AI Agent

External content may intentionally contain instructions designed to influence AI behavior.

This is particularly important for:

Browsing agents
Email assistants
Document-processing agents
Research agents
Automated workflows

Email as an Untrusted Boundary

Consider:

External Sender
      ↓
Email
      ↓
AI Assistant
      ↓
Enterprise Workflow

An email is not trusted simply because an enterprise AI assistant can read it.

A safer model is:

External Email
      ↓
UNTRUSTED CONTENT
      ↓
AI Processing

File Upload Boundary

Uploaded files introduce another boundary.

User
 ↓
File Upload
 ↓
========================
UNTRUSTED INPUT BOUNDARY
========================
 ↓
AI Processing

Possible concerns include:

Malicious content
Prompt injection
Malware
Sensitive information
Unexpected file types
Resource exhaustion

AI Agent Trust Boundaries

AI agents introduce some of the most important trust boundaries in AI architecture.

User
 ↓
AI Agent
 ↓
LLM
 ↓
Tool
 ↓
Enterprise System

The most important transition may be:

AI DECISION
     ↓
========================
PRIVILEGE BOUNDARY
========================
     ↓
REAL-WORLD ACTION

Why Agent Boundaries Are Critical

A chatbot may produce:

Incorrect Response

An agent may produce:

Incorrect Decision
      ↓
Tool Invocation
      ↓
Production Change

The second scenario converts AI behavior into enterprise impact.

Agent-to-Tool Boundary

AI Agent
    ↓
Tool Request
    ↓
========================
 POLICY BOUNDARY
========================
    ↓
Enterprise Tool

Ask:

Is the requested tool allowed?
Are parameters validated?
Is the user authorized?
Is the agent authorized?
Is approval required?
Is the action logged?

Tool-to-Enterprise-System Boundary

Agent Tool
    ↓
========================
AUTHORIZATION BOUNDARY
========================
    ↓
Production API

Even when an AI agent requests an action, the enterprise API should independently enforce authorization.

Dangerous Agent Architecture

A risky pattern:

User
 ↓
LLM
 ↓
Agent
 ↓
Administrator Credential
 ↓
Production

A stronger pattern:

User
 ↓
LLM
 ↓
Proposed Action
 ↓
Policy Engine
 ↓
Authorization
 ↓
Human Approval
 ↓
Scoped Service Identity
 ↓
Production

The model proposes an action.

Trusted systems determine whether the action is allowed.

Privilege Boundaries

A privilege boundary exists whenever a workflow moves from lower privilege to higher privilege.

Example:

Normal User
    ↓
AI Application
    ↓
AI Agent
    ↓
Privileged Cloud API

Ask:

Can a low-privileged user indirectly cause a higher-privileged agent to perform an action?

If yes, authorization must be carefully designed.

The Confused Deputy Problem

AI agents can create a classic security problem known as the confused deputy.

Low-Privilege User
        ↓
AI Agent
        ↓
High-Privilege Identity
        ↓
Sensitive System

The user cannot directly access the sensitive system.

But the agent may possess greater permissions.

If the agent blindly performs the requested action, the user may indirectly gain capabilities they should not have.

The authorization model should therefore consider:

The initiating user
The agent identity
The requested operation
The target resource

Identity Boundaries

Different identities may be involved throughout one AI workflow.

Employee Identity
       ↓
Application Identity
       ↓
RAG Service Identity
       ↓
Agent Identity
       ↓
Cloud Workload Identity

For each transition ask:

Which identity is used?
Why does it require access?
What permissions does it have?
Can it be impersonated?
Is activity attributable?

Service-to-Service Trust

AI architectures contain many service relationships.

Application
    ↓
AI Service
    ↓
RAG Service
    ↓
Vector Database

Do not assume services can trust each other simply because they are on the same network.

Use:

Strong workload identity
Authentication
Authorization
Encryption
Least privilege
Logging

Cloud Trust Boundaries

AI applications frequently span cloud services.

AI Application
      ↓
Cloud AI Service
      ↓
Object Storage
      ↓
Database
      ↓
Secrets Manager

Trust boundaries may exist between:

Accounts
Subscriptions
Projects
Virtual networks
Clusters
Namespaces
Services
Regions

Cloud architecture should therefore appear in the AI threat model when relevant.

Container and Kubernetes Boundaries

AI workloads may operate inside containers and Kubernetes.

Internet
   ↓
Ingress
   ↓
Kubernetes Service
   ↓
AI Pod
   ↓
Model Service

Potential boundaries include:

Internet → Ingress
Namespace → Namespace
Pod → Pod
Workload → Cloud API
Container → Host

AI security does not replace infrastructure security.

Development-to-Production Boundary

AI development pipelines introduce another critical boundary.

Developer
    ↓
Source Repository
    ↓
CI/CD
    ↓
========================
DEPLOYMENT BOUNDARY
========================
    ↓
Production

Security controls may include:

Code review
Branch protection
Security scanning
Artifact integrity
Deployment approval
Workload identity

Model Supply-Chain Boundary

Models may enter the organization from external repositories.

External Model Repository
          ↓
========================
 SUPPLY CHAIN BOUNDARY
========================
          ↓
Enterprise Model Registry
          ↓
Production

Ask:

Is the source trusted?
Is model provenance known?
Is integrity verified?
Has the model been assessed?
Who approved deployment?

Dataset Supply-Chain Boundary

The same concept applies to datasets.

External Dataset
       ↓
========================
 DATA TRUST BOUNDARY
========================
       ↓
Training Pipeline

Potential concerns include:

Poisoning
Privacy
Data quality
Malicious content
Provenance

Logging Trust Boundary

Security telemetry also requires protection.

AI Application
       ↓
Agent Activity
       ↓
Security Logs
       ↓
========================
SECURITY DATA BOUNDARY
========================
       ↓
SIEM

Logs should maintain appropriate:

Integrity
Availability
Confidentiality

Attackers should not easily erase or manipulate evidence of AI misuse.

Human Approval Boundary

Human approval can create another important security boundary.

AI Agent
   ↓
Proposed Action
   ↓
========================
HUMAN CONTROL BOUNDARY
========================
   ↓
Approved Action
   ↓
Execution

Human approval is useful only when the reviewer receives enough information to understand:

The requested action
The target resource
The reason
The potential impact

Trust Boundary vs Network Boundary

These concepts are related but different.

A network boundary may be:

Internet
 ↓
Firewall
 ↓
Private Network

A trust boundary can exist entirely within the private network:

Normal Application
       ↓
Privileged Administrative API

Do not identify trust boundaries only by looking at network diagrams.

Identity, privilege, organizational ownership and data sensitivity can also create trust boundaries.

Trust Boundary vs Data Flow

A data flow describes:

Where information moves.

A trust boundary describes:

Where security assumptions change.

Example:

User
 │
 │ Data Flow
 ▼
====================
 Trust Boundary
====================
 ▼
Application

Trust Boundary vs Attack Surface

An attack surface is somewhere an attacker may interact with a system.

A trust boundary represents a change in trust.

They often overlap.

For example:

Internet User
     ↓
Public API

The API may represent both:

An attack surface
A trust boundary

However, an internal privilege transition may also be a critical trust boundary without being publicly accessible.

Map Controls to Trust Boundaries

For every important boundary, identify the controls protecting it.

Boundary	Example Controls
User → Application	Authentication, MFA, authorization
Application → LLM	Authentication, encryption, data controls
RAG → Vector DB	Workload identity, authorization
Enterprise → External AI	Encryption, provider controls, governance
Agent → Tool	Policy enforcement, tool restrictions
Tool → Production	Authorization, least privilege
AI → Sensitive Action	Validation and human approval where appropriate

Boundary Control Layers

A useful model is:

Source
  ↓
Authentication
  ↓
Authorization
  ↓
Input / Context Validation
  ↓
Policy Enforcement
  ↓
Destination

Not every boundary requires every control.

Apply controls according to risk.

Boundary Failure Analysis

For every trust boundary ask:

What happens if this boundary fails?

Example:

Boundary:
RAG Authorization

Failure:
Unauthorized document retrieval

Impact:
Confidential information disclosure

Another:

Boundary:
Agent Policy Enforcement

Failure:
Unauthorized tool execution

Impact:
Production modification

This directly connects trust boundaries to threat scenarios.

Trust Boundary Register

Create an Obsidian note named:

AI Trust Boundary Register.md

Use:

ID	Source	Destination	Trust Change	Data / Action	Primary Concern	Controls
TB-01	User	AI Application	External → Internal	Prompt	Malicious input	Authentication, authorization, logging
TB-02	Application	External LLM	Enterprise → Third Party	Prompt + context	Data exposure	Encryption, provider controls
TB-03	RAG	Vector DB	Service → Sensitive Data	Retrieval	Unauthorized access	Service identity, authorization
TB-04	Agent	Cloud API	AI → Privileged System	Tool action	Unauthorized action	Least privilege, policy enforcement
TB-05	Internet Content	Agent	Untrusted → AI Context	External content	Indirect manipulation	Isolation, validation, restricted actions

Trust Boundary Assessment Template

Create reusable notes using:

# TB-XX — Boundary Name

## Source

## Destination

## Purpose

## Identity

## Data or Action Crossing the Boundary

## Data Classification

## Trust Assumption

## Privilege Change

## Potential Threats

## Existing Controls

## Required Controls

## Logging Requirements

## Failure Impact

## Notes

Example — AI Agent to Cloud API

# TB-04 — AI Agent to Cloud API

## Source

AI Agent

## Destination

Production Cloud API

## Purpose

Allow approved infrastructure operations.

## Identity

Dedicated AI agent workload identity.

## Data or Action Crossing the Boundary

Cloud API request.

## Data Classification

Restricted administrative operation.

## Trust Assumption

Agent requests are expected to represent authorized user actions.

## Privilege Change

High.

## Potential Threats

- Agent manipulation
- Unauthorized tool invocation
- Excessive agency
- Privilege abuse

## Existing Controls

- Dedicated workload identity
- API authentication

## Required Controls

- Least privilege
- Independent authorization
- Tool restrictions
- Parameter validation
- Approval for high-risk actions
- Comprehensive logging

## Failure Impact

Unauthorized production modification.

Prioritize High-Risk Boundaries

Not every boundary requires equal attention.

Prioritize boundaries involving:

Restricted data
Administrative privileges
External parties
Untrusted content
AI agents
Production systems
Credentials
Sensitive RAG sources

A useful mental model is:

Untrusted Input
      +
Sensitive Asset
      +
Privilege Transition
      =
High-Priority Trust Boundary

Common Trust Boundary Mistakes

Trusting Internal Users

Internal users and accounts can be compromised or abused.

Trusting RAG Content

Enterprise repositories may contain compromised or malicious content.

Trusting LLM Output

Model-generated information should not automatically be trusted.

Trusting AI Agents

Agents should operate under explicit authorization and limited privilege.

Ignoring Third Parties

External AI providers create important organizational boundaries.

Focusing Only on Networks

Identity, data and privilege can create trust boundaries without network segmentation.

Ignoring Return Flows

Information returned from models, tools and external services may also be untrusted.

Assuming Authentication Equals Authorization

Knowing who a user is does not mean every requested operation is allowed.

Real-World AI Security Engineer Workflow

When reviewing an AI architecture:

Architecture Diagram
        ↓
Identify Actors
        ↓
Identify Data Flows
        ↓
Identify Identities
        ↓
Identify Privilege Changes
        ↓
Identify Trust Boundaries
        ↓
Document Trust Assumptions
        ↓
Identify Existing Controls
        ↓
Ask What Happens If They Fail
        ↓
Identify Threat Scenarios

This is where architecture analysis begins turning into adversarial thinking.

Trust Boundary Mindset

Whenever you examine an AI architecture, ask:

Where Does Trust Change?
        ↓
What Crosses the Boundary?
        ↓
Who Controls That Input?
        ↓
Which Identity Is Used?
        ↓
Does Privilege Increase?
        ↓
What Sensitive Asset Is Reachable?
        ↓
What Validates the Transition?
        ↓
What Happens If Validation Fails?
        ↓
How Would We Detect Abuse?

Knowledge Check

Question 1

What is a trust boundary?

Question 2

Why does authentication not make a user fully trusted?

Question 3

Why should LLM-generated output not automatically be trusted?

Question 4

Why does RAG create additional trust boundaries?

Question 5

Why can enterprise documents still contain untrusted AI instructions?

Question 6

What makes the AI agent-to-tool boundary particularly important?

Question 7

What is a privilege boundary?

Question 8

How can an AI agent create a confused deputy problem?

Question 9

Why is an external LLM provider a trust boundary?

Question 10

What is the difference between a network boundary and a trust boundary?

Question 11

Why should security controls be mapped to trust boundaries?

Question 12

Which types of boundaries should receive the highest priority?

Key Takeaways

Trust boundaries show where:

Identity Changes
      +
Trust Changes
      +
Data Sensitivity Changes
      +
Privilege Changes
      +
Organizational Control Changes

AI systems introduce important boundaries involving:

Users
Applications
LLMs
RAG pipelines
Vector databases
External content
Third-party AI providers
AI agents
Enterprise tools
Production systems

The most important principle from this lesson is:

Never allow an AI model to become the primary security boundary protecting a critical enterprise asset.

Use trusted systems for:

Authentication
Authorization
Policy enforcement
Least privilege
Approval
Logging
Monitoring

Design the architecture so that even if the AI behaves unexpectedly, the potential impact remains controlled.

What’s Next?

➡️ 05 — AI Threat Actors and Attack Surface Identification

You now understand:

Where trust changes across AI architectures.
Why internal users and systems cannot automatically be trusted.
How RAG introduces new content trust boundaries.
Why AI agents create privilege boundaries.
How third-party AI services change enterprise trust.
How security controls map to important boundaries.

The next step is to identify:

Who might attack the AI system, what they want, and where they can interact with it.

In the next lesson, you will learn how to:

Identify external AI attackers.
Identify malicious and compromised insiders.
Identify supply-chain adversaries.
Understand indirect attackers using malicious content.
Identify AI-specific attack entry points.
Map LLM attack surfaces.
Map RAG attack surfaces.
Map AI agent attack surfaces.
Map model and data supply-chain surfaces.
Connect threat actors to assets and trust boundaries.
Build an AI Attack Surface Register.

The goal is to move from:

“I know where trust changes.”

to:

“I know who may attempt to cross those boundaries, where they can enter, and which assets they could ultimately reach.”

➡️ Next: 05 — AI Threat Actors and Attack Surface Identification