05 Sensitive Information Disclosure

Large Language Model applications often process far more information than users realize.

An enterprise AI application may have access to:

Internal documents
Customer records
Employee information
Source code
Security logs
Incident data
API responses
Conversation history
AI agent tool output
Application instructions
Business-sensitive information

This creates a critical security question:

What happens if the AI returns information that the user should never have been able to access?

This is the problem of Sensitive Information Disclosure.

For an AI Security Engineer, Sensitive Information Disclosure is not simply about a model accidentally revealing something.

The real problem may exist anywhere across:

Identity
   +
Authorization
   +
Context
   +
RAG
   +
Memory
   +
Model
   +
Output
   +
Logging

The objective is to ensure sensitive information is protected throughout the complete LLM application lifecycle.

Learning Objectives

By the end of this lesson, you should be able to:

Explain Sensitive Information Disclosure in LLM applications.
Identify common categories of sensitive information.
Understand how sensitive data reaches model context.
Recognize RAG-related disclosure risks.
Understand cross-user and cross-tenant data leakage.
Recognize conversation memory risks.
Understand system prompt and hidden context exposure.
Recognize credential and secret exposure scenarios.
Understand output and logging risks.
Apply data minimization.
Apply authorization before retrieval.
Understand privacy-aware AI design.
Perform safe disclosure testing.
Document findings professionally.

What Is Sensitive Information Disclosure?

Sensitive Information Disclosure occurs when an AI application exposes information that should only be accessible to authorized users, applications or systems.

Conceptually:

Sensitive Information
        ↓
AI Application
        ↓
LLM
        ↓
Unauthorized User

The information may be intentionally requested or exposed accidentally.

Sensitive information can enter the system through many different paths.

What Is Sensitive Information?

Sensitive information depends on the organization and business context.

Examples may include:

Personal Information

Customer Records

Employee Records

Financial Information

Authentication Credentials

API Keys

Access Tokens

Source Code

Security Architecture

Incident Data

Internal Documents

Legal Information

Trade Secrets

Business Strategy

The security requirement is not simply:

Keep secrets secret.

It is:

Ensure each user and system can access only the information required for their authorized purpose.

Data Classification

Organizations commonly classify information.

A simple model might be:

Public
   ↓
Internal
   ↓
Confidential
   ↓
Restricted

Public

Information approved for public access.

Examples:

Public website content
Marketing materials
Published documentation

Internal

Information intended for employees or approved partners.

Examples:

Internal procedures
General corporate documentation

Confidential

Information requiring stronger access controls.

Examples:

Customer information
Internal architecture
Business plans

Restricted

Highly sensitive information.

Examples:

Credentials
Private keys
Authentication secrets
Highly regulated data

AI applications should understand which categories they are allowed to process.

How Sensitive Information Reaches an LLM

A model may receive information from:

System Prompt
      +
User Input
      +
Conversation History
      +
RAG Documents
      +
Database Results
      +
Tool Output
      +
Application Metadata
      ↓
     LLM

Each path creates a potential disclosure risk.

Disclosure Is Often an Architecture Problem

Consider an employee asking:

Show me executive salary information.

The application may retrieve the information and send it to the LLM.

The system prompt says:

Do not reveal confidential salary information.

This is weak architecture.

Why?

Because unauthorized data already reached the model.

A stronger architecture is:

Employee
   ↓
Authentication
   ↓
Authorization
   ↓
Salary Record Access?
   ↓
DENIED

The information never reaches the LLM.

This leads to one of the strongest principles in LLM Security:

Prevent unauthorized information from reaching the model instead of relying on the model not to reveal it.

Disclosure Path 1 — User Input

Users themselves may provide sensitive information.

For example:

User
 ↓
Prompt:
"My password is..."
 ↓
LLM

This creates questions such as:

Is the prompt stored?
Is it sent to an external provider?
Is it retained?
Is it logged?
Can administrators view it?

Sensitive disclosure does not always involve an attacker.

Users may expose information accidentally.

Example — Accidental Credential Exposure

An engineer asks an AI assistant:

Why is this API request failing?

API_KEY=...

The secret may now exist in:

Prompt History
Logs
Provider Request
Conversation Storage

Security awareness and technical controls should reduce this risk.

Disclosure Path 2 — System Prompts

System prompts may contain internal application information.

For example:

You are the internal finance assistant.

Use database X.

Internal workflow:
...

Organizations sometimes place information in system prompts that users were never intended to see.

This can create hidden context exposure risk.

Never Store Secrets in System Prompts

Bad:

Database Password:
secret123

API Key:
abcxyz

inside a system prompt.

The model does not need these secrets as natural-language context.

Secrets should remain in dedicated security systems.

Use:

Application
   ↓
Secret Manager
   ↓
Authenticated Service

not:

Secret
   ↓
LLM Context

Disclosure Path 3 — RAG

RAG is one of the most important enterprise disclosure risks.

Consider:

Employee
   ↓
AI Assistant
   ↓
RAG
   ↓
Enterprise Knowledge

The knowledge base contains:

General Policies

Security Procedures

HR Records

Finance Documents

Executive Information

If retrieval does not preserve authorization, any employee may potentially retrieve sensitive content.

Weak RAG Architecture

Authenticated Employee
        ↓
Search Entire Knowledge Base
        ↓
Relevant Document
        ↓
LLM

Authentication exists.

Authorization does not.

Secure RAG Architecture

Authenticated Employee
        ↓
User Identity
        ↓
Authorization
        ↓
Allowed Collections
        ↓
Retriever
        ↓
Authorized Content
        ↓
LLM

This is called authorization-aware retrieval.

The RAG Security Rule

Remember:

The LLM should never become a shortcut around existing enterprise permissions.

If a user cannot open a document normally, the AI assistant should not make that document accessible through natural-language search.

Cross-User Leakage

AI applications may serve many users.

A serious security problem exists if information from one user’s session appears in another user’s session.

Conceptually:

User A
   ↓
Sensitive Conversation
   ↓
Shared Context / Memory
   ↓
User B

This should never happen without an explicit authorized collaboration model.

Session Isolation

Applications should separate:

User A Session

from:

User B Session

Security questions include:

Is memory scoped per user?
Are caches isolated?
Are conversation IDs securely generated?
Can users access another conversation?
Are temporary files isolated?

Traditional application security remains extremely important here.

Cross-Tenant Leakage

Multi-tenant applications create an even higher-risk scenario.

Consider:

AI SaaS Platform
     │
     ├── Company A
     ├── Company B
     └── Company C

A tenant isolation failure may expose:

Company A
    ↓
Company B Data

This can have significant:

Security
Privacy
Contractual
Regulatory

impact.

RAG Tenant Isolation

A stronger architecture might use:

Tenant Identity
      ↓
Tenant Authorization
      ↓
Tenant-Specific Collection
      ↓
Retriever

rather than:

All Tenants
    ↓
Shared Search

without effective filtering.

Disclosure Path 4 — Conversation History

AI applications frequently maintain previous messages.

Message 1
   ↓
Message 2
   ↓
Message 3
   ↓
Conversation History

This history may be sent back to the model during future interactions.

Sensitive information may therefore persist longer than the user expects.

Conversation History Risks

Potential risks include:

Sensitive data retention
Shared-device exposure
Cross-user leakage
Excessive provider retention
Administrative access
Prompt reuse

Applications should define:

What Is Stored?

Why Is It Stored?

For How Long?

Who Can Access It?

How Can It Be Deleted?

Disclosure Path 5 — Long-Term Memory

Some AI agents maintain longer-term memory.

Conceptually:

Conversation
    ↓
Memory Store
    ↓
Future Sessions

This may improve personalization.

It also creates security questions.

Memory Security Questions

Ask:

What information is written to memory?

Who can write it?

Who can read it?

How long is it retained?

Can users delete it?

Is memory isolated by user?

Is sensitive information filtered?

Long-term memory becomes another enterprise data store.

It requires access controls.

Disclosure Path 6 — Tool Output

AI agents may call enterprise systems.

For example:

LLM
 ↓
HR Tool
 ↓
Employee Record
 ↓
LLM

The tool may return more information than necessary.

Example:

The agent needs:

Employee Department

but the API returns:

Name
Address
Salary
National Identifier
Medical Information
Department

The model now receives far more sensitive information than required.

Minimize Tool Output

A stronger pattern is:

Agent Request
      ↓
Scoped API
      ↓
Required Fields Only
      ↓
LLM

This applies data minimization to agent tools.

Disclosure Path 7 — Model Output

Sensitive information may appear directly in model responses.

Example:

User
 ↓
AI Assistant
 ↓
Sensitive Information

Possible causes include:

Missing authorization
Excessive context
Retrieval weakness
Cross-user memory
Tool overexposure

The response itself may be the final symptom.

The root cause may exist earlier in the architecture.

Root Cause Thinking

Do not stop with:

Finding:
LLM exposed confidential information.

Ask:

Why did the LLM have that information?

Where did it come from?

Why was it retrieved?

What authorization should have blocked it?

This leads to better remediation.

Disclosure Path 8 — Logs

AI applications generate logs.

Potential logging sources include:

Application Logs

Prompt Logs

Response Logs

RAG Logs

Agent Logs

API Logs

Debug Logs

These may accidentally contain sensitive information.

Example

A developer logs the full request:

User Prompt:
Please analyze customer record 12345...

Context:
[Full customer record]

Now sensitive information exists in:

Application Log

which may be accessible to many operators.

Logging Can Create a Second Copy of Sensitive Data

Think:

Original Sensitive Data
        ↓
AI Application
        ↓
Logs

The organization now has an additional location that must be protected.

Therefore:

Logging should capture what security teams need, not everything the model sees.

Privacy-Aware Logging

Instead of storing complete prompts, organizations may log:

User ID

Timestamp

Application

Request ID

Data Source

Tool Invocation

Security Decision

and selectively store content where justified.

Exact requirements depend on the use case.

Disclosure Path 9 — Error Messages

Application errors may reveal:

Model names
File paths
Internal APIs
Database information
Stack traces
Prompt fragments
Secrets

Example:

Error:
Failed request using API token xyz...

Production error handling should minimize unnecessary internal details.

Disclosure Path 10 — External Model Providers

An organization may send prompts to an external provider.

Architecture:

Enterprise Application
        ↓
External LLM API

The security boundary has now crossed the organization.

Questions include:

What data is being sent?

Where is it processed?

Is it retained?

Who can access it?

Is it used for training?

What contractual controls exist?

Vendor and data-governance requirements become relevant.

Shadow AI Disclosure

Employees may use unapproved external AI tools.

Example:

Employee
   ↓
Copies Internal Source Code
   ↓
Public AI Tool

or:

Analyst
   ↓
Uploads Customer Report
   ↓
Unapproved AI Service

This can create sensitive data exposure even when the enterprise AI platform itself is secure.

Disclosure Path 11 — Training Data

Models may be trained or fine-tuned on sensitive information.

Training data may include:

Customer information
Internal source code
Business documents
Support conversations

Security teams should understand:

What entered training?

Was it authorized?

Was sensitive information required?

How is the dataset protected?

Training Data Does Not Disappear

After model training, the organization still has security obligations for:

Original datasets
Prepared datasets
Training environments
Model artifacts

The training lifecycle itself needs protection.

Disclosure Path 12 — Fine-Tuning Data

Organizations may fine-tune models using enterprise information.

Example:

Internal Support Tickets
        ↓
Fine-Tuning
        ↓
Enterprise Model

Security questions include:

Are tickets appropriately sanitized?
Do they contain customer secrets?
Who can access the dataset?
Is the model intended to process this information?

Fine-tuning should not become an uncontrolled path for sensitive data.

Model Memorization and Inference Risk

Certain security assessments may consider whether model behavior can reveal information associated with training or prior interactions.

The practical enterprise lesson is:

Do not assume sensitive information becomes harmless simply because it has been processed by an AI model.

Sensitive data should be minimized throughout the lifecycle.

Sensitive Information and Prompt Injection

Prompt injection may be used as one step toward disclosure.

Example:

Attacker Input
      ↓
Prompt Injection
      ↓
Model Behavior Changes
      ↓
Sensitive Context Exposed

However, ask:

Why did unauthorized sensitive context exist in the model session?

Often the strongest fix is not merely improving the prompt.

It is fixing:

Authorization
+
Context Minimization
+
Data Access

Sensitive Information and Jailbreaking

Similarly:

Jailbreak
   ↓
Model Ignores Restriction
   ↓
Sensitive Information Requested

If data confidentiality depends entirely on model refusal, the architecture is weak.

A stronger design ensures:

Unauthorized User
       ↓
Sensitive Data
       ↓
Never Reaches Model

Sensitive Information and AI Agents

Agents create additional risks because they may retrieve information dynamically.

Example:

User
 ↓
AI Agent
 ↓
Database Tool
 ↓
Sensitive Database

The security question becomes:

What determines which records the tool can retrieve?

The correct answer should not be:

Whatever the LLM decides.

Agent Authorization

A stronger pattern:

User Identity
      ↓
AI Agent
      ↓
Proposed Data Request
      ↓
Authorization
      ↓
Allowed Data
      ↓
Tool

The downstream system should enforce access.

Principle 1 — Authorization Before Retrieval

This is one of the strongest defenses.

Weak:

Retrieve Everything
      ↓
LLM
      ↓
Decide What to Reveal

Strong:

User Identity
      ↓
Authorization
      ↓
Retrieve Allowed Data
      ↓
LLM

This substantially reduces disclosure risk.

Principle 2 — Data Minimization

Provide only the minimum information required.

Example:

User asks:

What is the customer's support tier?

Bad:

Full Customer Profile
       ↓
LLM

Better:

Customer Support Tier
       ↓
LLM

This limits exposure.

Principle 3 — Purpose Limitation

Ask:

Why does this AI need access to this data?

Example:

A general IT support chatbot probably does not need:

Employee Salary Records

Remove unnecessary access.

This reduces both attack surface and privacy risk.

Principle 4 — Least Privilege for Data

Least privilege applies to information as well as actions.

An AI agent may need access to:

Security Alerts

but not:

Complete SIEM Administrator Data
+
HR Information
+
Financial Systems

Limit data sources to business requirements.

Principle 5 — Preserve Existing Permissions

If an enterprise document repository already has:

Document Permissions

the AI integration should preserve them.

Do not build:

Secure Document Platform
         ↓
Export Everything
         ↓
Shared AI Knowledge Base

and accidentally remove existing authorization.

Principle 6 — Separate Sensitive Collections

Where appropriate:

General Knowledge
      ↓
Collection A

HR Knowledge
      ↓
Collection B

Security Knowledge
      ↓
Collection C

Then enforce access to each collection.

This can simplify authorization.

Principle 7 — Remove Secrets From Context

Never intentionally expose:

Passwords
Private keys
Long-lived access tokens
API secrets

to the model unless a very specific architecture requires it and appropriate controls exist.

In most cases, the model does not need the secret itself.

Example

The AI needs to query an API.

Bad:

LLM Context
   ↓
API Key

Better:

Agent Tool
   ↓
Managed Identity / Secret Manager
   ↓
API

The model knows the tool exists.

It does not need to know the credential.

Principle 8 — Protect Memory

Long-term memory should be:

User-scoped
Access-controlled
Minimally retained
Deletable
Monitored

Sensitive information should not automatically become permanent AI memory.

Principle 9 — Protect Logs

Apply:

Access control
Encryption
Redaction
Retention policies
Monitoring

to AI logs.

Treat logs according to the sensitivity of the information they may contain.

Principle 10 — Control External Data Transfer

Before sending enterprise information to external AI services, determine:

Is this provider approved?

Is this data permitted?

What is retained?

What contractual protections exist?

The decision should be governed by enterprise policy.

Principle 11 — Output Controls

Applications may detect or restrict certain sensitive information in output.

Examples may include:

Secrets
Credentials
Specific regulated data
Restricted document classifications

Output controls can provide an additional layer.

However:

Output filtering should not replace authorization.

The strongest control remains preventing unauthorized data from reaching the model.

Principle 12 — Separate Environments

Avoid using production sensitive information unnecessarily in:

Development
Testing
Demonstrations
Security labs

Prefer synthetic data.

Example:

Customer:
TEST-USER-001

Account:
TEST-ACCOUNT-001

This allows security behavior to be tested without exposing real users.

Safe Disclosure Testing

Sensitive Information Disclosure testing should use controlled data whenever possible.

Do not prove a vulnerability by collecting unnecessary real confidential information.

Step 1 — Identify Data Categories

Document:

Public

Internal

Confidential

Restricted

and identify which categories the application processes.

Step 2 — Identify Users and Roles

Example:

Employee

HR Employee

Security Analyst

Administrator

Determine what each role should access.

Step 3 — Build an Authorization Matrix

Example:

Role	Public Policies	HR Records	Security Incidents
Employee	Allow	Deny	Deny
HR	Allow	Allow	Deny
SOC	Allow	Deny	Allow

This gives you expected security behavior.

Step 4 — Create Synthetic Sensitive Data

Example:

Document:
TEST-EXECUTIVE-COMPENSATION

Classification:
Restricted

Allowed Group:
HR-Leadership

Use synthetic content instead of real salary information.

Step 5 — Test Retrieval Boundaries

Ask whether an unauthorized user can cause the AI to retrieve the test document.

Test through:

Normal queries
Prompt manipulation
Context variations

The goal is to validate authorization.

Step 6 — Test Session Isolation

Use:

Test User A

Test User B

Verify that information from User A cannot appear in User B’s session.

Step 7 — Test Tenant Isolation

If applicable, create:

Tenant A Test Data

Tenant B Test Data

Verify cross-tenant access is impossible.

Step 8 — Review Memory

Determine whether synthetic sensitive information remains available later.

Ask:

Was it stored?

Where?

For how long?

Who can retrieve it?

Step 9 — Review Logs

Check whether test sensitive data appears in:

Application logs
Debug output
Model telemetry
Agent logs

This frequently reveals overlooked copies.

Step 10 — Review External Transfers

Determine whether test information is sent outside expected trust boundaries.

Document:

Application
   ↓
External Provider

where applicable.

Step 11 — Review Tool Access

If an agent retrieves data:

User
 ↓
Agent
 ↓
Tool
 ↓
Sensitive Store

verify the tool enforces user or application authorization.

Step 12 — Validate Impact

Ask:

What information was exposed?

Who received it?

Was authorization bypassed?

Was data persisted?

Did it cross a tenant boundary?

Did it leave the organization?

These questions determine actual severity.

Example — HR Assistant

Architecture:

Employee
   ↓
HR AI Assistant
   ↓
RAG
   ↓
HR Repository

Data:

Leave Policy

Benefits

Employee Salaries

Performance Reviews

Expected access:

Employee
   ↓
Leave Policy
Benefits

HR
   ↓
All Approved HR Information

Weak Design

All HR Documents
      ↓
Vector Database
      ↓
Every Employee

with:

System Prompt:
Do not reveal restricted HR information.

This depends too heavily on model behavior.

Strong Design

Employee Identity
      ↓
Authorization
      ↓
General HR Collection
      ↓
Retriever
      ↓
LLM

Restricted HR information never enters unauthorized context.

Example — Security Assistant

Architecture:

SOC Analyst
   ↓
AI Assistant
   ↓
SIEM Tool

The tool returns:

Alert
+
User Details
+
API Tokens
+
Raw Credentials

but only alert information is required.

Better:

SIEM Tool
   ↓
Filtered Response
   ↓
Required Alert Fields
   ↓
LLM

This reduces exposure.

Example — Coding Assistant

A coding assistant may process:

Source code
Configuration
Environment files

Potential risk:

.env
 ↓
AI Coding Assistant
 ↓
API Key Exposed

Controls may include:

Secret scanning
File exclusions
Repository permissions
External-provider policies

Example — AI Support Agent

An AI support agent has access to:

Customer Database

The user asks:

What is my current support case status?

The tool should query:

Authenticated User
      ↓
Own Customer Record

not:

LLM Generates Customer Search
      ↓
Unrestricted Database Search

Authorization should remain deterministic.

Sensitive Information Finding Template

Finding:
Unauthorized RAG Retrieval of Restricted Documents

Affected Component:
Enterprise AI Assistant

Data Classification:
Restricted

Expected Behavior:
Standard employees should access only general policy documents.

Observed Behavior:
A standard test user was able to retrieve synthetic
restricted HR content through the AI assistant.

Root Cause:
RAG retrieval does not preserve document-level authorization.

Impact:
Users may access sensitive enterprise information outside
their approved permissions.

Recommendation:
Implement identity-aware retrieval and enforce document-level
authorization before content is provided to the LLM.

Another Example Finding

Finding:
Sensitive Prompt Content Stored in Application Logs

Affected Component:
LLM Application Logging

Observed Behavior:
Full prompt content, including synthetic customer data,
was written to application debug logs.

Impact:
Sensitive information may become accessible to users
with log-platform access and may be retained beyond
the original AI interaction.

Recommendation:
Disable unnecessary prompt logging, implement redaction,
restrict log access and apply an appropriate retention policy.

Sensitive Information Risk Factors

Consider:

Data Sensitivity
       +
Number of Users
       +
Authorization Failure
       +
Persistence
       +
External Transfer
       +
Business Impact

The same technical issue can have very different severity.

Example Comparison

Scenario A

Public documentation accidentally exposed.

Impact:

Low

because it was already public.

Scenario B

Internal operating procedure exposed to another employee.

Impact:

Potentially Moderate

depending on classification.

Scenario C

Customer financial information exposed across tenants.

Impact:

Potentially High / Critical

depending on scope and environment.

Always assess actual data sensitivity.

Preventive Controls

Preventive controls may include:

Authentication

Authorization

Data Classification

Data Minimization

Tenant Isolation

Context Minimization

Secret Management

RAG Access Controls

Detective Controls

Detective controls may include:

Sensitive Data Access Monitoring

Unusual Retrieval Alerts

Cross-Tenant Access Detection

Secret Detection

Agent Activity Monitoring

Responsive Controls

Response capabilities may include:

Disable Account

Remove Sensitive Document

Rebuild RAG Index

Revoke Credentials

Delete Conversation Data

Rotate Exposed Secret

Notify Appropriate Teams

AI security should consider all three:

Prevent
+
Detect
+
Respond

Incident Response for AI Data Exposure

If sensitive information is unexpectedly exposed:

Detect
  ↓
Stop Further Exposure
  ↓
Identify Data
  ↓
Identify Affected Users
  ↓
Preserve Evidence
  ↓
Remove Access
  ↓
Rotate Secrets if Required
  ↓
Investigate Root Cause
  ↓
Remediate

Privacy, legal or compliance teams may also need involvement depending on the information.

Questions During Investigation

Ask:

What information was exposed?

Which user received it?

Where did the data originate?

How did it reach model context?

Was it stored in memory?

Was it logged?

Was it sent externally?

Can the event be reproduced?

Did other users experience the same issue?

This helps determine scope.

AI Security Engineer Checklist

Data

Sensitive data sources identified.
Data classification understood.
Unnecessary data access removed.
Data minimization applied.

Identity

Users authenticated.
User roles understood.
Service identities documented.

Authorization

Authorization enforced before retrieval.
Document permissions preserved.
Tenant isolation implemented.
Agent tool access restricted.

Context

Sensitive context minimized.
Secrets excluded.
Hidden context reviewed.
Conversation history protected.

RAG

Data sources approved.
Retrieval is identity-aware.
Sensitive collections isolated where appropriate.
Metadata protected.

Memory

Memory is user-scoped.
Retention is defined.
Sensitive data persistence minimized.
Deletion supported where required.

Logs

Prompt logging reviewed.
Response logging reviewed.
Sensitive fields redacted where appropriate.
Access to logs restricted.
Retention defined.

External Providers

Data transfer understood.
Approved provider used.
Sensitive data policy followed.

Agents

Tool output minimized.
Agent permissions least privileged.
Data-access decisions enforced outside the model.

Response

Exposed credentials can be rotated.
Sensitive content can be removed.
RAG indexes can be rebuilt.
Investigation evidence is available.

Common Beginner Mistakes

Mistake 1 — Treating Data Leakage as Only a Model Problem

The root cause may be authorization, RAG or application design.

Mistake 2 — Giving the Model Everything

Only necessary information should reach context.

Mistake 3 — Relying on “Do Not Reveal” Instructions

Prompts are not access-control systems.

Mistake 4 — Ignoring Conversation Memory

Sensitive information may persist beyond one interaction.

Mistake 5 — Ignoring Logs

Logs can create a second copy of sensitive data.

Mistake 6 — Treating Embeddings as Automatically Safe

Vector stores can still expose sensitive information or metadata.

Mistake 7 — Ignoring Tenant Isolation

Multi-tenant leakage can create major impact.

Mistake 8 — Putting Secrets in Prompts

Secrets belong in dedicated secret-management systems.

Mistake 9 — Proving Findings With Real Data

Use synthetic evidence whenever possible.

Mistake 10 — Fixing Only the Output

If the model should never have received the information, fix the upstream authorization.

AI Security Engineer Perspective

When you discover information disclosure, do not ask only:

Why did the model say this?

Ask:

Where did this information come from?

Why did the model have access to it?

Who was supposed to access it?

Where should authorization have occurred?

Was the data necessary for the task?

Was it stored elsewhere?

Did it cross another trust boundary?

This helps identify the real security problem.

Interview Perspective

You may be asked:

What is Sensitive Information Disclosure in an LLM application?

A strong answer is:

Sensitive Information Disclosure occurs when an LLM application exposes information to a user or system that is not authorized to receive it. The root cause may involve excessive model context, weak RAG authorization, cross-user memory, logging, agent tool access or other application-level controls rather than the model itself.

Another question may be:

How would you prevent an LLM from exposing confidential RAG data?

A strong answer is:

I would enforce user identity and document-level authorization before retrieval so that the LLM only receives information the user is permitted to access. I would also minimize retrieved context, preserve data classification and tenant boundaries and avoid relying solely on a system prompt telling the model not to disclose restricted information.

Another question may be:

Why is data minimization important in AI Security?

A strong answer is:

The more sensitive information available in model context, memory, tools and logs, the greater the potential impact of manipulation, application errors or access-control failures. Providing only the minimum data required for the task reduces both attack surface and privacy risk.

Another question may be:

Should secrets ever be stored in a system prompt?

A strong answer is:

No. System prompts are part of model context and should not be used as secret storage. Credentials should be kept in dedicated secret-management or workload-identity systems and accessed by authorized application components or tools without exposing the secret to the model.

Key Takeaways

Sensitive Information Disclosure can occur through:

Prompts
+
System Context
+
Conversation History
+
Memory
+
RAG
+
Vector Databases
+
Tool Output
+
Model Responses
+
Logs
+
External Providers

The strongest design is not:

Give Sensitive Data to LLM
        ↓
Tell Model Not to Reveal It

It is:

User Identity
      ↓
Authorization
      ↓
Data Minimization
      ↓
Allowed Data Only
      ↓
LLM

Important security principles include:

Classify sensitive information.
Preserve enterprise authorization.
Apply authorization before retrieval.
Minimize model context.
Isolate users and tenants.
Protect conversation memory.
Keep secrets out of prompts.
Minimize tool responses.
Protect logs.
Understand external data transfers.
Use synthetic data for testing.

Most importantly:

The safest sensitive information is information the unauthorized user can never cause the model to receive.

What’s Next?

➡️ 06 — Insecure Output Handling

So far, we have focused primarily on information entering and leaving the LLM.

But generated model output may itself become dangerous when another application trusts it.

An LLM may generate:

HTML
SQL
Shell commands
Source code
URLs
API parameters
Cloud configurations
Structured actions

The next lesson explores what happens when this output is passed directly into another system.

You will learn:

What Insecure Output Handling means
Why LLM output should not automatically be trusted
Output as a new input boundary
Generated HTML and web risks
Generated SQL
Generated shell commands
Code generation
Agent tool parameters
Output validation
Allowlisting
Sandboxing
Least privilege
Human approval
Safe testing methodology

You will move from:

What Information Can the LLM Expose?

to:

What Can Happen When Another System
Trusts What the LLM Generates?

➡️ Next: 06 — Insecure Output Handling