02 How AI and Machine Learning Systems Work

Before you can secure an AI system, you need to understand how it works.

You do not need to become a data scientist or machine learning researcher.

As an AI Security Engineer, your goal is to understand the major components, data flows and lifecycle stages well enough to identify:

• What needs protection
• Where trust boundaries exist
• Where attackers may interfere
• Where sensitive data may be exposed
• Where security controls should be applied

This lesson gives you that foundation.

Learning Objectives

By the end of this lesson, you should be able to:

• Explain the difference between AI, Machine Learning and Deep Learning.
• Understand how Machine Learning systems learn from data.
• Explain the difference between training and inference.
• Understand what models and datasets are.
• Recognize common Machine Learning approaches.
• Understand neural networks at a practical level.
• Understand the Machine Learning lifecycle.
• Identify security risks across the lifecycle.
• Understand how traditional ML connects to Generative AI and Large Language Models.

What Is Artificial Intelligence?

Artificial Intelligence is a broad term for computer systems designed to perform tasks that normally require some level of human intelligence.

These tasks may include:

• Understanding language
• Recognizing images
• Making predictions
• Detecting patterns
• Recommending actions
• Generating content
• Automating decisions
• Solving problems

AI is the broad category.

Machine Learning is one way of building AI systems.

A useful relationship is:

Artificial Intelligence
        │
        └── Machine Learning
                │
                └── Deep Learning
                        │
                        └── Generative AI / Modern LLMs

Each level builds on concepts from the previous one.

Traditional Programming vs Machine Learning

Traditional software usually follows explicit rules.

For example:

Input
  ↓
Programmed Rules
  ↓
Output

Imagine a simple security rule:

IF failed_login_attempts > 5
THEN generate_alert

The programmer defines the behavior.

Machine Learning works differently.

Instead of manually defining every rule, the system learns patterns from examples.

Historical Data
      ↓
Learning Algorithm
      ↓
Model
      ↓
Prediction

For example, a Machine Learning system may learn what suspicious login behavior looks like based on previous activity.

What Is Machine Learning?

Machine Learning is a method where computer systems learn patterns from data and use those patterns to make predictions or decisions.

A simplified process looks like:

Data
  ↓
Training
  ↓
Model
  ↓
New Input
  ↓
Prediction

The model is created during training.

The model is then used during inference.

What Is a Model?

A model is the result of a Machine Learning training process.

You can think of it as a mathematical representation of patterns learned from data.

For example, imagine a security team wants to identify suspicious login activity.

The training data may contain:

User
Location
Login Time
Device
Failed Attempts
Successful Login
Known Malicious?

The Machine Learning system analyzes patterns across this data.

The result is a model that may estimate:

Normal Login
or
Suspicious Login

The model does not simply memorize one rule.

It learns relationships from data.

What Is a Dataset?

A dataset is a collection of information used by a Machine Learning system.

Examples include:

• Images

• Text

• Security logs

• Network traffic

• Transactions

• Customer activity

• Source code

• Documents

• Malware samples

Datasets may be used for:

• Training

• Validation

• Testing

• Evaluation

Data quality has a major impact on the quality and security of the resulting model.

Training Data

Training data is the information used to teach the model.

For example:

Training Dataset
       │
       ├── Normal Login
       ├── Normal Login
       ├── Suspicious Login
       ├── Normal Login
       └── Suspicious Login

The algorithm uses these examples to learn patterns.

Security question:

Can attackers influence the training data?

If they can, they may be able to influence model behavior.

This becomes important later when we study data poisoning.

Validation Data

Validation data helps developers tune the model during development.

A simple workflow is:

Training Data
      ↓
Train Model
      ↓
Validation Data
      ↓
Tune Model

The validation dataset should normally be separate from the training data.

Test Data

Test data is used to evaluate the final model.

Model
  ↓
Test Dataset
  ↓
Performance Evaluation

The purpose is to understand how the model performs on data it did not directly learn from.

Training vs Inference

This distinction is very important.

Training

Training is the process of creating or improving a model.

Dataset
   ↓
Training Algorithm
   ↓
Model

Training may require:

• Large datasets

• Significant compute

• GPUs

• Specialized frameworks

• Long processing times

Inference

Inference is when the trained model is used.

New Input
   ↓
Model
   ↓
Prediction / Output

For example:

Login Event
    ↓
Security Model
    ↓
Suspicious: 92%

For an LLM:

Prompt
   ↓
LLM
   ↓
Generated Response

Security threats can affect both training and inference.

Why Training Security Matters

If an attacker can modify the training pipeline, they may affect the model itself.

Possible risks include:

• Data poisoning

• Malicious training samples

• Dataset tampering

• Model manipulation

• Compromised dependencies

• Unauthorized model changes

Think of the training environment as a software supply chain.

Dataset
   +
Code
   +
Dependencies
   +
Compute
   ↓
Training Pipeline
   ↓
Model

Every component requires security.

Why Inference Security Matters

Inference is where users and applications interact with the model.

Potential risks include:

• Malicious input

• Data leakage

• Unauthorized API access

• Model extraction

• Excessive resource usage

• Prompt manipulation

• Unsafe model output

In many enterprise environments, inference is the most visible part of the AI system.

Supervised Learning

In supervised learning, the training data contains known answers or labels.

Example:

Email 1 → Spam
Email 2 → Normal
Email 3 → Spam
Email 4 → Normal

The model learns relationships between inputs and known outputs.

Common uses include:

• Classification

• Fraud detection

• Malware classification

• Spam detection

• Image recognition

Security example:

A security team trains a model using:

Network Traffic → Benign
Network Traffic → Malicious

The model learns to classify future network activity.

Unsupervised Learning

In unsupervised learning, the data is not necessarily labeled.

The system attempts to discover patterns or groups.

For example:

User Activity
      ↓
Machine Learning
      ↓
Behavior Groups

This may help identify unusual behavior.

Security applications include:

• Anomaly detection

• User behavior analytics

• Network behavior analysis

• Threat detection

Reinforcement Learning

In reinforcement learning, a system learns through interactions and feedback.

A simplified view:

Agent
  ↓
Action
  ↓
Environment
  ↓
Reward / Penalty
  ↓
Agent Learns

This type of learning is important in many AI research areas.

Security considerations include:

• Reward manipulation

• Unsafe actions

• Environment manipulation

• Adversarial inputs

What Is Deep Learning?

Deep Learning is a type of Machine Learning that uses neural networks with multiple layers.

Deep Learning is widely used in:

• Image recognition

• Speech recognition

• Natural language processing

• Generative AI

• Large Language Models

A simple relationship is:

Machine Learning
       │
       └── Deep Learning
               │
               └── Neural Networks

What Is a Neural Network?

A neural network is a computational system made of interconnected layers.

A simplified structure is:

Input Layer
     ↓
Hidden Layers
     ↓
Output Layer

Example:

Security Event Data
        ↓
Neural Network
        ↓
Suspicious / Normal

Each layer transforms information.

The network learns internal parameters during training.

You do not need to understand all the mathematics at this stage.

For security engineering, focus on:

• What data enters the network

• Where the model came from

• How the model was trained

• What outputs it produces

• How it is accessed

• How attackers may manipulate the process

Parameters

A Machine Learning model contains internal numerical values called parameters.

During training, these parameters are adjusted.

Conceptually:

Input
  ↓
Model Parameters
  ↓
Output

Large modern models may contain very large numbers of parameters.

Parameters represent learned relationships in the model.

Features

In traditional Machine Learning, a feature is a measurable characteristic used by the model.

For example, a login risk model might use:

Login Time
Country
Device Type
Failed Attempts
IP Reputation
User History

These are features.

The model learns how those features relate to suspicious activity.

Labels

A label is the expected outcome in supervised learning.

Example:

Login 1 → Normal
Login 2 → Malicious
Login 3 → Normal

Normal and Malicious are labels.

If labels are incorrect or maliciously manipulated, model behavior may be affected.

The Machine Learning Lifecycle

A typical ML lifecycle may look like:

Business Problem
      ↓
Data Collection
      ↓
Data Preparation
      ↓
Model Development
      ↓
Training
      ↓
Validation
      ↓
Testing
      ↓
Deployment
      ↓
Inference
      ↓
Monitoring
      ↓
Retraining

Each stage creates security considerations.

Stage 1 — Business Problem

Every AI project should begin with a purpose.

Examples:

• Detect fraud

• Identify malware

• Recommend products

• Classify documents

• Detect unusual behavior

Security engineers should ask:

Is AI necessary for this use case?

What could happen if the model makes a wrong decision?

Is this a high-risk business process?

Understanding business impact helps determine security requirements.

Stage 2 — Data Collection

Data may come from:

• Internal systems

• Public sources

• Partners

• Third parties

• Sensors

• APIs

• Customer interactions

Security considerations include:

• Data source trust

• Privacy

• Integrity

• Licensing

• Sensitive information

• Data provenance

Stage 3 — Data Preparation

Raw data often needs to be cleaned and transformed.

This may involve:

• Removing errors

• Formatting

• Labeling

• Filtering

• Normalization

• Deduplication

Security question:

Who can modify the data preparation pipeline?

A compromised preprocessing stage can affect downstream model behavior.

Stage 4 — Model Development

Developers choose:

• Algorithms

• Frameworks

• Libraries

• Architectures

• Model sources

Security risks may include:

• Vulnerable libraries

• Malicious packages

• Untrusted models

• Compromised repositories

This is part of the AI supply chain.

Stage 5 — Training

The training environment may contain:

• Sensitive datasets

• Expensive GPU resources

• Proprietary model code

• Credentials

• Storage

• Compute infrastructure

Security controls may include:

• IAM

• Network restrictions

• Encryption

• Logging

• Secrets management

• Environment isolation

Stage 6 — Validation and Testing

Models should be evaluated before deployment.

Security teams may also evaluate:

• Robustness

• Abuse cases

• Adversarial behavior

• Data leakage

• Security controls

This is where AI security testing begins to become important.

Stage 7 — Deployment

A trained model may be deployed through:

• API

• Cloud AI service

• Container

• Kubernetes

• Mobile application

• Embedded system

Deployment introduces traditional infrastructure security requirements.

Stage 8 — Inference

Users or applications send inputs to the model.

User / Application
        ↓
      API
        ↓
      Model
        ↓
     Output

Security controls may include:

• Authentication

• Authorization

• Input handling

• Rate limiting

• Logging

• Data protection

Stage 9 — Monitoring

Models must be monitored after deployment.

Organizations may monitor:

• Errors

• Performance

• Security alerts

• Unusual activity

• Input patterns

• Output behavior

• Infrastructure

• Data access

Security monitoring becomes increasingly important in enterprise AI environments.

Stage 10 — Retraining

Models may be updated with new data.

New Data
   ↓
Retraining
   ↓
Updated Model

This creates a continuous security lifecycle.

Security teams should consider whether attackers can influence the new data used for retraining.

A Security View of the ML Lifecycle

From an AI Security Engineer’s perspective:

Data
 │
 ├── Is it trusted?
 ├── Is it sensitive?
 └── Can it be manipulated?
 │
 ▼
Training
 │
 ├── Is the environment secure?
 ├── Are dependencies trusted?
 └── Who can modify the process?
 │
 ▼
Model
 │
 ├── Is it protected?
 ├── Is it authentic?
 └── Can it be replaced?
 │
 ▼
Deployment
 │
 ├── Is IAM secure?
 ├── Are APIs protected?
 └── Are secrets protected?
 │
 ▼
Inference
 │
 ├── Can input be malicious?
 ├── Can data leak?
 └── Can the model be abused?
 │
 ▼
Monitoring
 │
 ├── Are actions logged?
 ├── Can attacks be detected?
 └── Can incidents be investigated?

This is the security mindset you should begin developing.

AI Assets Across the Lifecycle

Different stages introduce different assets.

Stage	Important Assets
Data Collection	Raw data
Preparation	Cleaned datasets
Training	Training code and compute
Model Development	Source code and dependencies
Model Storage	Model files
Deployment	Infrastructure and configuration
Inference	APIs, prompts and outputs
Monitoring	Logs and telemetry
Retraining	Updated datasets and models

A security assessment should identify these assets.

Threat: Data Poisoning

Data poisoning occurs when training or related data is manipulated to influence model behavior.

Conceptually:

Legitimate Data
       +
Malicious Data
       ↓
Training
       ↓
Manipulated Model

The effect may be subtle.

The model may behave normally in most situations but incorrectly in specific scenarios.

You will explore this topic later.

Threat: Model Theft

Models may represent valuable intellectual property.

Attackers may attempt to:

• Download model files

• Copy model parameters

• Access proprietary APIs

• Reconstruct model behavior

Security controls may include:

• Access control

• Encryption

• Network restrictions

• Monitoring

• Rate limiting

Threat: Model Tampering

If an attacker can modify the model, they may alter its behavior.

Possible attack path:

Compromised Model Repository
          ↓
Modified Model
          ↓
Production Deployment

Model integrity therefore matters.

Threat: Untrusted Models

Organizations may download models from external repositories.

This creates supply-chain questions.

Before using an external model, teams should consider:

• Who published it?

• Where did it come from?

• What license applies?

• Has it been reviewed?

• Can it contain unsafe artifacts?

• Has its integrity been verified?

Do not automatically trust a model because it is publicly available.

Threat: Sensitive Training Data

Training datasets may contain:

• Customer information

• Source code

• Medical information

• Financial information

• Employee data

• Proprietary knowledge

Security teams should understand:

What data is being used?

Why is it needed?

Who can access it?

How is it protected?

How long is it retained?

Threat: Insecure ML Infrastructure

Machine Learning platforms often require powerful infrastructure.

Examples include:

• GPU servers

• Cloud compute

• Storage

• Databases

• Model registries

• Development notebooks

A compromised notebook or storage bucket could expose:

• Data

• Models

• Credentials

• Source code

Traditional cloud security is therefore highly relevant.

Threat: Vulnerable Dependencies

Machine Learning projects often use many software packages.

Example:

AI Application
      │
      ├── Python
      ├── ML Framework
      ├── Data Library
      ├── API Library
      └── Supporting Packages

A vulnerable or malicious dependency can compromise the environment.

AI security is therefore closely connected with software supply-chain security.

Threat: Resource Abuse

AI and ML workloads can consume significant resources.

Attackers may attempt to abuse:

• GPUs

• CPU

• API quotas

• Cloud credits

• Storage

• Inference capacity

This can create both availability and financial impact.

Machine Learning in Cybersecurity

Machine Learning is already used in security products.

Examples include:

• Malware detection

• Fraud detection

• Threat detection

• Email filtering

• User behavior analytics

• Endpoint detection

• Network anomaly detection

For example:

Security Events
      ↓
ML Model
      ↓
Risk Score
      ↓
Security Alert

As an AI Security Engineer, you may need to secure both:

AI used inside business applications

and

AI used inside security technologies.

False Positives and False Negatives

Security models are not perfect.

Imagine a malware detection model.

True Positive

Malware correctly identified as malware.

True Negative

Safe file correctly identified as safe.

False Positive

Safe file incorrectly identified as malware.

False Negative

Malware incorrectly identified as safe.

For security systems, false negatives can be particularly dangerous because malicious activity may go undetected.

False positives can overwhelm security teams.

Understanding model accuracy therefore has operational security implications.

Model Drift

Real-world environments change.

A model trained using historical data may become less effective over time.

This is called model drift.

Example:

Model Trained
     ↓
Environment Changes
     ↓
Model Accuracy Decreases

Security teams should understand that deployed AI systems require continuous monitoring.

Data Drift

Data patterns can also change.

For example:

Normal User Behavior in 2025
          ↓
Business Changes
          ↓
Normal User Behavior in 2026

The model may begin making inaccurate decisions if new data differs significantly from historical patterns.

Security Impact of Incorrect Models

An inaccurate model may cause security problems even without an attacker.

Examples:

• Legitimate users blocked

• Malicious activity missed

• Incorrect fraud decisions

• Incorrect access decisions

• Unsafe recommendations

Security engineers should therefore think about both:

Malicious Failure
        +
Accidental Failure

AI Systems Are Probabilistic

Traditional software often produces predictable results.

For example:

2 + 2 = 4

AI systems may operate probabilistically.

The same or similar input may sometimes produce different output.

This changes how security teams test systems.

You may need to evaluate:

• Repeated behavior

• Output variation

• Confidence

• Edge cases

• Unexpected responses

AI security testing often requires more than one test input.

From Machine Learning to Generative AI

Traditional Machine Learning commonly predicts or classifies.

For example:

Input:
Security Event

Output:
Malicious = 89%

Generative AI goes further.

It creates new content.

For example:

Input:
Summarize this security incident.

Output:
Generated Incident Summary

Generative systems may create:

• Text

• Images

• Code

• Audio

• Video

• Documents

Large Language Models are one important form of Generative AI.

From ML to LLMs

A simplified progression is:

Artificial Intelligence
       ↓
Machine Learning
       ↓
Deep Learning
       ↓
Neural Networks
       ↓
Transformer Models
       ↓
Large Language Models
       ↓
Generative AI Applications

You do not need to understand every mathematical detail.

But you should understand how each layer builds upon the previous concepts.

Security Changes With Generative AI

Traditional ML systems often follow:

Input
  ↓
Model
  ↓
Prediction

Generative AI applications may look more like:

User
 │
 ▼
Application
 │
 ├── System Instructions
 │
 ├── User Prompt
 │
 ├── Retrieved Documents
 │
 ├── Conversation Context
 │
 └── Tool Results
 │
 ▼
LLM
 │
 ▼
Generated Output

This creates many additional trust boundaries.

Each input source may affect model behavior.

Why This Matters for AI Security Engineers

When you see a Machine Learning or AI system, do not see only:

Model

Instead see:

Data
+
Code
+
Dependencies
+
Training Infrastructure
+
Model
+
Model Repository
+
API
+
Application
+
Identity
+
Cloud Infrastructure
+
Logs

Every component can create security risk.

Enterprise Scenario

Imagine a bank develops a Machine Learning system for fraud detection.

Architecture:

Transactions
     ↓
Data Pipeline
     ↓
Fraud Detection Model
     ↓
Risk Score
     ↓
Payment System

Think like an AI Security Engineer.

Question 1 — Data Integrity

Can attackers manipulate transaction data before it reaches the model?

Question 2 — Training Data

Who can modify the training dataset?

Question 3 — Model Integrity

Who can replace the production model?

Question 4 — Infrastructure

Where is the model running?

Question 5 — Identity

Which service accounts can access the model?

Question 6 — Decision Impact

Can a model decision automatically block transactions?

Question 7 — Monitoring

Would the organization detect unexpected model behavior?

Question 8 — Recovery

Can the organization quickly roll back to a known-good model?

These questions demonstrate how Machine Learning security becomes enterprise security.

Secure Machine Learning Lifecycle

A stronger architecture applies security throughout the lifecycle.

Trusted Data Sources
       ↓
Secure Data Pipeline
       ↓
Controlled Training Environment
       ↓
Validated Model
       ↓
Protected Model Registry
       ↓
Secure Deployment
       ↓
Authenticated API
       ↓
Monitoring
       ↓
Incident Response

This is similar to DevSecOps:

Security should be integrated into the process rather than added at the end.

AI Security Engineer Checklist

When reviewing an ML system, ask:

Business

• What decision does the model make?

• What happens if the model is wrong?

• Is the decision security-sensitive?

Data

• Where does training data come from?

• Is the source trusted?

• Is sensitive information present?

• Who can modify the dataset?

Development

• Which frameworks are used?

• Are dependencies trusted?

• Are external models used?

Training

• Who can access the environment?

• Are credentials protected?

• Is training activity logged?

Model

• Where is the model stored?

• Is model integrity validated?

• Who can replace the model?

Deployment

• How is the model exposed?

• Is authentication required?

• Are permissions restricted?

Operations

• Is model activity monitored?

• Is unexpected behavior detected?

• Is rollback possible?

This is a useful starting point for any Machine Learning security assessment.

Common Beginner Mistakes

Mistake 1 — Thinking AI Is Only the Model

The ecosystem around the model is equally important.

Mistake 2 — Ignoring Training Security

Attacks before deployment can affect everything that follows.

Mistake 3 — Trusting Public Datasets Automatically

Public availability does not guarantee integrity or quality.

Mistake 4 — Trusting Public Models Automatically

External models introduce supply-chain risk.

Mistake 5 — Ignoring Traditional Infrastructure

AI still runs on servers, cloud services, containers and APIs.

Mistake 6 — Treating Model Output as Guaranteed Truth

AI predictions may be incorrect.

Mistake 7 — Ignoring Monitoring

Models can change in effectiveness over time.

Interview Perspective

You may be asked:

What is the difference between AI, Machine Learning and Deep Learning?

A practical answer is:

Artificial Intelligence is the broad field of creating systems capable of performing tasks associated with human intelligence. Machine Learning is a subset of AI where systems learn patterns from data rather than relying only on explicitly programmed rules. Deep Learning is a subset of Machine Learning that uses multi-layer neural networks and forms the foundation of many modern Generative AI and Large Language Model systems.

Another important question is:

What is the difference between training and inference?

A strong answer is:

Training is the process of using data and algorithms to create or adjust a model. Inference is the process of using that trained model to make a prediction or generate an output from new input. Security risks exist in both stages, including data poisoning during training and malicious input or data exposure during inference.

Key Takeaways

Artificial Intelligence is the broad field.

Machine Learning allows systems to learn patterns from data.

Deep Learning uses multi-layer neural networks and powers many modern AI systems.

A Machine Learning system typically involves:

Data
  ↓
Training
  ↓
Model
  ↓
Deployment
  ↓
Inference
  ↓
Monitoring

Security must protect every stage.

Important security concerns include:

• Training data integrity

• Model integrity

• Sensitive data protection

• AI supply-chain security

• Secure infrastructure

• API security

• Access control

• Monitoring

• Model behavior

The most important lesson is:

Securing AI requires understanding the complete lifecycle, not simply the final model.

What’s Next?

➡️ 03 — Generative AI and LLM Architecture

You now understand the foundations of AI and Machine Learning.

The next step is understanding the technology driving many modern enterprise AI applications:

Generative AI and Large Language Models.

In the next lesson, you will learn:

• What Generative AI is

• What Large Language Models are

• How LLMs process text

• Tokens and tokenization

• Transformer architecture at a practical level

• Training and inference for LLMs

• System prompts and user prompts

• Context windows

• Embeddings

• Model APIs

• Why LLM architecture creates new security challenges

• How LLMs connect to RAG and AI agents

This will prepare you for the later modules on LLM Security, RAG Security and AI Red Teaming.

➡️ Next: 03 — Generative AI and LLM Architecture