02 Prompt Injection Attacks

Prompt Injection is one of the most important security challenges affecting applications built with Large Language Models.

Traditional applications usually separate:

Code
  ↓
Data

An LLM application often processes both instructions and data through natural language.

For example:

System Instruction
      +
User Input
      +
Retrieved Documents
      ↓
      LLM

The model must determine which information represents:

• Trusted application instructions

• User requests

• Reference data

• External content

• Tool results

An attacker may attempt to exploit this ambiguity by introducing instructions that influence the model in ways the application developer did not intend.

This is called Prompt Injection.

For an AI Security Engineer, the important question is not simply:

Can the model be manipulated?

The more important question is:

What security impact becomes possible if the model is manipulated?

Learning Objectives

By the end of this lesson, you should be able to:

• Explain what prompt injection is.

• Understand why prompt injection exists.

• Distinguish instructions from data.

• Understand direct prompt injection.

• Understand how system instructions may be challenged.

• Recognize common prompt-manipulation patterns.

• Understand how prompt injection affects RAG.

• Understand how prompt injection affects AI agents.

• Identify possible attack paths.

• Understand why prompt filtering alone is insufficient.

• Apply defense-in-depth controls.

• Perform safe prompt injection testing in authorized environments.

• Document prompt injection findings professionally.

What Is Prompt Injection?

Prompt Injection occurs when attacker-controlled input influences the behavior of an LLM application in an unintended way.

Conceptually:

Trusted Application Instructions
             +
Attacker-Controlled Input
             ↓
            LLM
             ↓
     Unexpected Behavior

The attacker attempts to make their input act like an instruction rather than ordinary data.

Simple Example

Imagine an application designed to summarize documents.

Expected workflow:

Document
   ↓
LLM
   ↓
Summary

The application instructs the model:

Summarize the document provided by the user.

Now imagine the document contains text telling the model to abandon the summarization task and follow a different instruction.

The system expected the document to be:

DATA

but the model may interpret part of the document as:

INSTRUCTION

This is the fundamental prompt injection problem.

Why Prompt Injection Exists

Large Language Models process natural language context.

That context may combine:

System Instructions
+
Developer Instructions
+
User Input
+
Conversation History
+
Retrieved Documents
+
Tool Results

All of this becomes input to the model.

The model must infer which information should influence its behavior.

Unlike traditional code execution, there may not always be a perfect technical separation between:

Instruction

and:

Data

This creates an attack surface.

Traditional Injection vs Prompt Injection

Traditional injection vulnerabilities occur when untrusted input is interpreted as instructions by another system.

For example, conceptually:

Untrusted Input
     ↓
Interpreter
     ↓
Unintended Command

Prompt Injection has a similar security pattern:

Untrusted Natural Language
        ↓
       LLM
        ↓
Unintended Model Behavior

However, LLM behavior is probabilistic rather than deterministic.

This makes prompt injection different from traditional injection vulnerabilities.

Prompt Injection Is Not Always Predictable

A traditional vulnerable application may behave consistently when given the same exploit input.

An LLM may behave differently depending on:

• Model version

• Context

• Temperature

• System instructions

• Conversation history

• Retrieved information

• Application logic

Therefore:

Same Prompt
     ↓
May Not Always Produce
Exactly the Same Result

Security testing may require multiple observations rather than one successful response.

Direct Prompt Injection

Direct Prompt Injection occurs when the attacker directly interacts with the LLM application.

The attack path is:

Attacker
   ↓
User Input
   ↓
LLM Application
   ↓
Model Behavior Changes

The attacker attempts to influence how the model follows instructions.

Direct Prompt Injection Scenario

Imagine an internal AI assistant designed to answer questions only about company policies.

Expected behavior:

Employee
   ↓
Policy Question
   ↓
AI Assistant
   ↓
Approved Policy Answer

An attacker attempts to provide instructions designed to override the intended task.

The security concern is not necessarily that the model discusses something unexpected.

The real concern is whether the manipulation allows:

• Unauthorized data access

• Access-control bypass

• Tool invocation

• Sensitive information disclosure

• Unsafe automated action

Prompt Injection vs Jailbreaking

These concepts overlap, but they are not always identical.

Prompt Injection

Focuses on manipulating an LLM application’s intended instructions or workflow.

Example concept:

Application Instruction
        ↓
Attacker Input
        ↓
Application Behavior Changes

Jailbreaking

Generally focuses on bypassing behavioral or safety restrictions placed on a model.

Conceptually:

Model Restriction
       ↓
Adversarial Prompt
       ↓
Attempted Safety Bypass

Prompt Injection often has stronger relevance to application security because it may influence connected systems.

Jailbreaking often focuses more directly on model behavior and safety controls.

You will examine jailbreaking separately later.

System Prompt Manipulation

Many applications use a system prompt.

Example:

You are an internal security assistant.

Answer only cybersecurity questions.

Do not reveal confidential information.

Developers may assume the system prompt provides a strong security boundary.

It does not.

A system prompt can help define expected model behavior, but it should not replace:

• Authentication

• Authorization

• Data filtering

• Agent permissions

• Policy enforcement

Weak Security Architecture

Consider:

Sensitive Data
     ↓
LLM
     ↓
System Prompt:
"Do not reveal sensitive data."

The model already has access to the sensitive information.

The organization is relying on model behavior to protect confidentiality.

Stronger Security Architecture

Instead:

User
 ↓
Authentication
 ↓
Authorization
 ↓
Allowed Data Only
 ↓
LLM

The model never receives unauthorized information.

This is significantly stronger.

The Fundamental Security Rule

Remember:

Use prompts to guide model behavior. Use deterministic security controls to enforce security boundaries.

This principle appears repeatedly throughout LLM Security.

Prompt Hierarchy

LLM applications may have different instruction sources.

Conceptually:

System Instructions
        ↓
Developer Instructions
        ↓
Application Context
        ↓
User Instructions
        ↓
Retrieved Content

The exact implementation depends on the model and application.

Security engineers should understand:

• Which instruction sources exist

• Which sources are trusted

• Which sources are user-controlled

• Whether untrusted content can influence higher-level behavior

Prompt Injection Attack Surface

Prompt injection can enter through more than the chat box.

Possible input sources include:

User Prompt
Document
Email
Website
Database
API Response
Search Result
RAG Content
Agent Memory
Tool Output

This is why prompt injection is a system-level security problem.

User-Controlled Prompt Injection

The simplest path is:

User
 ↓
Prompt
 ↓
LLM

The user intentionally attempts to influence model behavior.

Security impact depends heavily on what the LLM can access.

Prompt Injection Through File Uploads

Consider:

User
 ↓
Uploads Document
 ↓
AI Document Analyzer
 ↓
LLM

The document itself may contain instructions intended to affect the model.

The application sees:

Uploaded File

but from the model’s perspective, it is text inside the context.

Prompt Injection Through Websites

AI browsing systems introduce another attack surface.

AI Agent
   ↓
Visits Website
   ↓
Reads Content
   ↓
LLM

If web content contains manipulative instructions, the site may influence the AI.

The attacker may never communicate directly with the application user.

Prompt Injection Through Email

Consider an AI assistant that summarizes email.

External Sender
      ↓
Email
      ↓
AI Assistant
      ↓
LLM

A malicious sender may place model-directed instructions inside the email.

This becomes particularly important if the assistant can also:

• Send messages

• Access files

• Create events

• Invoke tools

Prompt Injection Through API Results

Agents frequently interact with APIs.

AI Agent
   ↓
External API
   ↓
API Response
   ↓
LLM

If API responses contain attacker-controlled text, that content may influence subsequent model behavior.

Tool results therefore should not automatically be treated as trusted instructions.

Prompt Injection and RAG

RAG is one of the most important environments for prompt injection.

Recall:

User Question
      ↓
Retriever
      ↓
Enterprise Documents
      ↓
LLM

If one document contains manipulative content:

Malicious Document
       ↓
RAG
       ↓
LLM Context
       ↓
Unexpected Behavior

This is often called Indirect Prompt Injection.

We will cover it in depth in the next lesson.

Prompt Injection and AI Agents

Prompt injection becomes significantly more dangerous when the LLM can perform actions.

Consider:

User
 ↓
AI Agent
 ↓
LLM
 ↓
Tool
 ↓
Enterprise System

If the attacker manipulates the model, the impact may move from:

Unexpected Text

to:

Unauthorized Action

This is one of the most important differences between traditional chatbots and AI agents.

Example — Read-Only Assistant

Consider:

User
 ↓
LLM
 ↓
Text Response

The model has:

• No sensitive data

• No tools

• No enterprise permissions

Prompt manipulation may produce inappropriate or incorrect output.

The operational impact may be limited.

Example — Privileged Agent

Now consider:

User
 ↓
LLM
 ↓
Agent
 ↓
Cloud Administrator Tool
 ↓
Production

A successful prompt injection may now influence:

• Cloud resources

• Security controls

• Production systems

This is why prompt injection severity depends heavily on architecture.

Prompt Injection Risk Factors

When evaluating risk, consider:

Input Exposure
      +
Data Access
      +
Agent Capability
      +
Tool Permission
      +
Business Impact

A useful conceptual model is:

Prompt Injection Risk
        ≈
Manipulability
        ×
Available Capability
        ×
Privilege
        ×
Impact

This is not a formal scoring formula.

It is a useful engineering mindset.

Common Prompt Manipulation Patterns

Attackers may use many strategies to influence model behavior.

Rather than memorizing payloads, understand the categories.

Pattern 1 — Instruction Override

The attacker attempts to replace the original task.

Conceptually:

Original Instruction
       ↓
Attacker Provides Conflicting Instruction
       ↓
Model Chooses Unexpected Behavior

Pattern 2 — Role Manipulation

The attacker attempts to redefine the model’s role or task.

Example concept:

"You are no longer performing Task A.
Act as Task B."

The wording may vary.

The security issue is whether this changes access or capability.

Pattern 3 — Context Manipulation

The attacker introduces information designed to alter how the model interprets later instructions.

Conceptually:

Attacker Context
      ↓
Model Interpretation Changes
      ↓
Later Request Behaves Differently

Pattern 4 — Instruction Smuggling

Instructions may be placed inside content that the application expects to be data.

Examples:

• Documents

• Web pages

• Emails

• Code comments

This is particularly relevant to indirect prompt injection.

Pattern 5 — Obfuscation

Attackers may transform instructions using:

• Encoding

• Alternate wording

• Character substitutions

• Multiple languages

• Formatting

This can make simple keyword filtering less reliable.

Pattern 6 — Multi-Step Manipulation

An attacker may spread manipulation across multiple turns.

Message 1
 ↓
Establish Context

Message 2
 ↓
Change Assumption

Message 3
 ↓
Trigger Behavior

This demonstrates why conversation history can become security-relevant.

Pattern 7 — Tool-Oriented Manipulation

The attacker attempts to influence the LLM toward using a particular tool.

Example architecture:

Prompt
 ↓
LLM
 ↓
Tool Selection
 ↓
Database

The key question becomes:

Can the model decide to perform something the user is not authorized to do?

Prompt Injection Attack Path

A professional assessment should look beyond the prompt.

Example:

Attacker
   ↓
Malicious Input
   ↓
Prompt Injection
   ↓
LLM Behavior Changes
   ↓
Agent Selects Tool
   ↓
Tool Uses Powerful Identity
   ↓
Sensitive Resource

Now we can identify several controls.

Control 1 — Authentication

Determine who can access the application.

Control 2 — Authorization

Ensure the user is permitted to request the action.

Control 3 — Agent Policy

Restrict which actions the agent may perform.

Control 4 — Least Privilege

Restrict the underlying tool identity.

Control 5 — Human Approval

Require approval for sensitive actions.

Control 6 — Logging

Record the complete action chain.

This demonstrates defense in depth.

Why Prompt Filtering Alone Is Not Enough

A common response to prompt injection is:

Block suspicious prompts.

Filtering can help.

But it has limitations.

Natural language is flexible.

The same intent can be expressed in many ways.

Attackers may use:

• Synonyms

• Multiple languages

• Encodings

• Indirect instructions

• Multi-turn conversations

• Retrieved content

Therefore:

Prompt filtering can reduce risk, but it should not be the primary security boundary.

Defense in Depth

A stronger architecture assumes some prompt manipulation attempts may reach the model.

Then it asks:

What prevents that manipulation from causing security impact?

For example:

Untrusted Input
      ↓
Input Controls
      ↓
LLM
      ↓
Restricted Context
      ↓
Agent Policy
      ↓
Least-Privilege Tool
      ↓
Human Approval
      ↓
Logging

Even if one layer fails, additional controls remain.

Defense 1 — Minimize Model Access

Do not give the model information it does not need.

Bad:

LLM
 ↓
Entire Enterprise Knowledge Base

Better:

User
 ↓
Authorization
 ↓
Relevant Approved Data
 ↓
LLM

Prompt injection cannot expose information the model never received.

Defense 2 — Enforce Authorization Outside the Model

Do not ask:

LLM:
"Is this user allowed to access the document?"

Use:

Authorization Service
        ↓
Access Decision
        ↓
Allowed Document
        ↓
LLM

The model should not control critical authorization.

Defense 3 — Separate Instructions and Untrusted Content

Where possible, application design should clearly label or structure:

Trusted Instructions

and:

Untrusted Content

However, remember:

Structural separation reduces risk but does not guarantee perfect resistance to prompt injection.

Additional controls are still required.

Defense 4 — Minimize Agent Tools

An agent should only receive tools required for its business purpose.

Instead of:

AI Agent
 ├── Email
 ├── Cloud Admin
 ├── Shell
 ├── Database
 └── File System

use:

AI Agent
 └── Read Security Alerts

Smaller capability means smaller potential impact.

Defense 5 — Least Privilege

The underlying identity should have minimum permissions.

Example:

Requirement:
Read Cloud Security Findings

Permission:
Read Cloud Security Findings

not:

Permission:
Administrator

This protects the environment even if model behavior is manipulated.

Defense 6 — Human Approval

For high-impact operations:

AI Agent
   ↓
Proposed Action
   ↓
Human Review
   ↓
Approved?

Examples include:

• Deleting resources

• Changing IAM

• Sending external communications

• Disabling users

• Deploying code

Human approval adds an independent trust boundary.

Defense 7 — Output Validation

If the model generates:

• SQL

• Code

• Commands

• URLs

• API parameters

validate them before use.

Example:

LLM Output
    ↓
Validation
    ↓
Approved Action

Prompt injection should not automatically turn generated text into system execution.

Defense 8 — Context Minimization

Reduce unnecessary model context.

Example:

Bad:

Entire Customer Database
        ↓
LLM

Better:

Authorized Customer Record
        ↓
LLM

Less exposed information means less potential disclosure.

Defense 9 — Protect RAG Sources

Controls may include:

• Approved ingestion sources

• Document ownership

• Content validation

• Change control

• Authorization-aware retrieval

This reduces indirect manipulation opportunities.

Defense 10 — Monitoring

Prompt injection cannot always be completely prevented.

Detection therefore matters.

Potential indicators may include:

• Repeated unusual prompts

• Attempts to access unrelated information

• Abnormal tool requests

• Unexpected agent actions

• Policy denials

Logging should allow investigation without unnecessarily retaining sensitive information.

Detection Architecture

A useful chain might be:

User Identity
      +
Request Metadata
      +
Model Interaction
      +
RAG Retrieval
      +
Tool Invocation
      +
Authorization Decision
      ↓
Security Monitoring

This allows security teams to understand what happened.

Prompt Injection vs Business Impact

Consider two situations.

Scenario A

Public documentation chatbot.

No enterprise access.

No sensitive information.

A user manipulates the model into discussing an unrelated topic.

Impact:

Mostly Behavioral

Scenario B

AI agent connected to production cloud resources.

A manipulation influences tool selection.

Impact:

Potential Enterprise Security Incident

The same broad attack class can have very different severity.

Safe Prompt Injection Testing

Prompt injection testing should occur only against:

• Your own applications

• Lab environments

• Dedicated test systems

• Systems where explicit authorization exists

A professional assessment should not begin with random payload experimentation.

Begin with the architecture.

Step 1 — Understand the Application

Document:

Business Purpose

Users

Model

RAG

Agents

Tools

Sensitive Data

Step 2 — Identify Instruction Sources

List:

System Prompt

Developer Instructions

User Input

Documents

Web Content

API Results

Tool Output

Determine which sources are attacker-controlled.

Step 3 — Identify Security Boundaries

Ask:

What should the user never access?

What should the model never trigger?

Which actions require authorization?

Which data is sensitive?

Step 4 — Define Expected Behavior

Example:

User:
Standard employee

Expected:
Can retrieve public employee policies

Must Not:
Retrieve restricted HR records

Without expected behavior, it is difficult to determine whether a security test succeeded.

Step 5 — Test Behavioral Manipulation

In an authorized lab, evaluate whether untrusted instructions can cause the application to deviate from expected behavior.

Focus on:

• Instruction following

• Context handling

• Data boundary behavior

• Tool-selection behavior

Avoid unnecessary destructive actions.

Step 6 — Validate Security Impact

A successful behavioral manipulation is not automatically a high-severity vulnerability.

Ask:

Did sensitive data become accessible?

Was authorization bypassed?

Could a tool be invoked?

Could an enterprise system be affected?

Could the behavior persist?

This determines actual security impact.

Step 7 — Collect Evidence

Document:

Expected Behavior

Input Source

Observed Behavior

Affected Component

Security Boundary

Potential Impact

Do not unnecessarily collect real sensitive information.

Use synthetic test data whenever possible.

Step 8 — Recommend Architecture Controls

Recommendations should focus on root cause.

Avoid only:

Change the prompt.

Consider:

Authorization

Least Privilege

Tool Restrictions

Data Minimization

Human Approval

Output Validation

Monitoring

Step 9 — Retest

After remediation:

Original Scenario
       ↓
Retest
       ↓
Security Boundary Holds?

Test whether the real risk was reduced.

Example Assessment Scenario

Consider:

Employee
   ↓
AI HR Assistant
   ↓
RAG
   ↓
HR Knowledge

The security requirement is:

Employees:
General HR Policies

HR Team:
Restricted HR Records

Expected Control

User Identity
      ↓
Authorization
      ↓
Allowed Documents
      ↓
RAG

Weak Architecture

User
 ↓
RAG Searches Everything
 ↓
LLM
 ↓
System Prompt:
"Do not show restricted information."

Prompt injection may become very dangerous because the model already has access to restricted information.

Stronger Architecture

User
 ↓
Authorization
 ↓
General HR Collection Only
 ↓
RAG
 ↓
LLM

Now prompt injection cannot retrieve documents that the application never exposes.

This is the difference between:

Prompt-Based Security

and:

Architecture-Based Security

Example Agent Scenario

Consider:

SOC Analyst
   ↓
AI Security Agent
   ↓
Cloud Tool

The agent requirement is:

Read Security Alerts

But the agent has:

Cloud Administrator

This means prompt injection may have a much larger impact than necessary.

Better Design

SOC Analyst
   ↓
AI Agent
   ↓
Policy Check
   ↓
Read-Only Security Role

Now even if manipulation influences the agent, its capability remains constrained.

Prompt Injection Finding Template

Use a structure like:

Finding:
Prompt Injection Can Influence Agent Tool Selection

Affected Component:
AI Security Agent

Entry Point:
User-controlled prompt

Expected Behavior:
Agent should only query approved read-only tools.

Observed Behavior:
Model behavior could be influenced toward an unintended tool request.

Security Impact:
If combined with excessive agent permissions,
the application may attempt unauthorized operations.

Root Cause:
Security relies heavily on model behavior rather than
independent tool authorization.

Recommendation:
Apply deterministic tool authorization, least privilege,
and approval for high-risk operations.

This is more useful than:

"The chatbot can be prompt injected."

Prompt Injection Risk Assessment

When evaluating severity, consider:

Exposure

Who can provide input?

Data

What sensitive data is available?

Tools

What systems can the model access?

Privilege

What permissions exist?

Autonomy

Can actions occur automatically?

Impact

What happens if manipulation succeeds?

A useful model is:

Input Control
     ↓
Model Manipulation
     ↓
Available Capability
     ↓
Security Impact

Prompt Injection Security Checklist

Architecture

• Application architecture is documented.

• Model capabilities are understood.

• Trust boundaries are identified.

Input

• User input is treated as untrusted.

• External content is treated as untrusted.

• File uploads are considered potential instruction sources.

System Prompt

• No secrets are stored in system prompts.

• System prompts are not treated as authorization controls.

Context

• Sensitive context is minimized.

• Conversation history is appropriately isolated.

• Untrusted sources are identified.

RAG

• Retrieval authorization is enforced.

• Document sources are controlled.

• RAG content is not automatically trusted as instruction.

Agents

• Agent tools are minimized.

• Agent permissions follow least privilege.

• Tool authorization is deterministic.

• Sensitive actions require approval where appropriate.

Output

• Model output is validated where necessary.

• Generated commands are not blindly executed.

Monitoring

• Suspicious usage can be investigated.

• Tool invocations are logged.

• Security decisions are observable.

Common Beginner Mistakes

Mistake 1 — Treating Prompt Injection as Only a Chatbot Problem

It can enter through documents, websites, APIs and tools.

Mistake 2 — Testing Only Whether the Model Ignores Instructions

The important question is whether security impact results.

Mistake 3 — Relying Only on Better Prompt Wording

Prompt design may help, but architecture controls are more important.

Mistake 4 — Giving the Model Sensitive Data and Asking It Not to Reveal It

Authorization should prevent unauthorized data from reaching the model.

Mistake 5 — Ignoring AI Agents

Agent capabilities can dramatically increase impact.

Mistake 6 — Giving Agents Broad Permissions

Least privilege is one of the strongest controls.

Mistake 7 — Ignoring Indirect Input

External content can manipulate the model without direct attacker interaction.

Mistake 8 — Treating Every Successful Manipulation as Critical

Severity depends on actual business impact.

Interview Perspective

You may be asked:

What is Prompt Injection?

A strong answer is:

Prompt Injection occurs when attacker-controlled input influences an LLM application in unintended ways by being interpreted as instructions rather than ordinary data. The security impact depends on the architecture, especially what sensitive information, tools and permissions are available to the model or connected agents.

Another question may be:

How is Prompt Injection different from traditional injection?

A strong answer is:

Both involve untrusted input influencing an interpreter, but traditional injection often abuses deterministic command or query syntax, while prompt injection targets probabilistic natural-language model behavior. This makes complete prevention difficult and increases the importance of architectural controls such as authorization, least privilege and output validation.

Another question may be:

How would you defend against Prompt Injection?

A strong answer is:

I would use defense in depth rather than relying only on prompt filtering. Sensitive data should be authorization-filtered before reaching the model, agent tools should be minimized and least privileged, high-impact actions should require independent authorization or human approval, model output should be validated and important activity should be logged and monitored.

Another question may be:

Why does Prompt Injection become more dangerous with AI agents?

A strong answer is:

A chatbot primarily produces text, while an agent can invoke tools and interact with enterprise systems. If attacker-controlled input manipulates the model and the agent has excessive permissions, a behavioral issue can become an unauthorized real-world action.

Key Takeaways

Prompt Injection occurs because LLM applications process:

Trusted Instructions
        +
Untrusted Natural Language
        ↓
       LLM

The model may interpret attacker-controlled data as instructions.

Prompt Injection may enter through:

User Prompts
+
Documents
+
Websites
+
Email
+
APIs
+
RAG
+
Tool Results

The security impact depends heavily on:

Data Access
+
Tool Access
+
Permission
+
Autonomy
+
Business Impact

Do not rely solely on:

Prompt Filtering

Use:

Authorization
+
Data Minimization
+
Least Privilege
+
Tool Restrictions
+
Human Approval
+
Output Validation
+
Monitoring

Most importantly:

Design the application so that successful prompt manipulation does not automatically become successful enterprise compromise.

What’s Next?

➡️ 03 — Indirect Prompt Injection

Direct Prompt Injection occurs when the attacker provides malicious instructions directly to the AI application.

But modern AI systems increasingly consume information from:

• Websites

• Documents

• Email

• Search engines

• Enterprise knowledge bases

• APIs

• RAG

• External tools

This introduces a more difficult security challenge:

Indirect Prompt Injection.

In the next lesson, you will learn:

• What Indirect Prompt Injection is

• Direct vs indirect prompt injection

• How malicious instructions enter through external content

• RAG-based attacks

• Web and email scenarios

• AI agent exposure

• Trust boundaries

• Persistent indirect injection

• Why content filtering is difficult

• Secure RAG and agent design

• Detection and response strategies

• Safe testing methodology

You will move from:

Attacker
   ↓
Direct Prompt
   ↓
LLM

to:

Attacker
   ↓
External Content
   ↓
AI Retrieves Content
   ↓
LLM
   ↓
Potential Agent / Data Impact

➡️ Next: 03 — Indirect Prompt Injection