Lab 02 — IOC Processing and Enrichment Pipeline

Mission Information

Difficulty: Beginner → Intermediate
Estimated Time: 90–120 minutes
Primary Language: Python
Security Domain: SOC / Threat Intelligence / Incident Response
Environment: Local controlled lab
Automation Type: Defensive Threat Intelligence Processing

Mission

Your SOC team has received indicators from several security sources.

The information contains:

IP ADDRESSES

DOMAIN NAMES

SHA-256 HASHES

DUPLICATES

INVALID VALUES

INCONSISTENT FORMATTING

Before analysts can use these indicators, the data must be cleaned and structured.

Your mission is to build:

RAW IOC DATA
     ↓
VALIDATION
     ↓
NORMALIZATION
     ↓
DEDUPLICATION
     ↓
CLASSIFICATION
     ↓
ENRICHMENT
     ↓
PRIORITIZATION
     ↓
ANALYST REPORT

The finished application will become a reusable defensive IOC-processing pipeline.

Why This Lab Matters

Threat intelligence rarely arrives perfectly formatted.

An organization may receive indicators from:

SIEM

EDR

EMAIL SECURITY

THREAT INTELLIGENCE FEEDS

INCIDENT REPORTS

INTERNAL INVESTIGATIONS

SECURITY VENDORS

One source might provide:

Example.COM

another:

example.com

and another:

example.com.

Without normalization, automation may treat these as different indicators.

Security automation therefore needs to understand:

VALIDITY

TYPE

NORMALIZATION

DUPLICATION

CONTEXT

CONFIDENCE

Learning Objectives

By completing this lab, you should be able to:

PROCESS IOC DATA

VALIDATE IP ADDRESSES

VALIDATE DOMAIN NAMES

VALIDATE SHA-256 HASHES

NORMALIZE INDICATORS

REMOVE DUPLICATES

CLASSIFY IOC TYPES

ADD DEFENSIVE CONTEXT

WORK WITH JSON

WORK WITH CSV

DESIGN SAFE API ENRICHMENT

HANDLE API FAILURES

GENERATE ANALYST REPORTS

Final Architecture

                  RAW IOC FILE
                       ↓
                 ┌───────────┐
                 │ INGESTION │
                 └─────┬─────┘
                       ↓
                 ┌───────────┐
                 │ VALIDATOR │
                 └─────┬─────┘
                       ↓
                 ┌───────────┐
                 │ NORMALIZE │
                 └─────┬─────┘
                       ↓
                 ┌───────────┐
                 │DEDUPLICATE│
                 └─────┬─────┘
                       ↓
                 ┌───────────┐
                 │ CLASSIFY  │
                 └─────┬─────┘
                       ↓
                 ┌───────────┐
                 │ ENRICHMENT│
                 └─────┬─────┘
                       ↓
                 ┌───────────┐
                 │ PRIORITIZE│
                 └─────┬─────┘
                       ↓
        ┌──────────────┼──────────────┐
        ↓              ↓              ↓
       CSV            JSON        MARKDOWN

Lab Scenario

Your SOC receives a file containing:

203.0.113.25
192.0.2.50
10.10.10.25
Example.COM
example.com
training.example.org
44d88612fea8a8f36de82e1278abb02f
not-an-ip
bad domain value

Some indicators are:

VALID

INVALID

DUPLICATED

PRIVATE

DOCUMENTATION / TRAINING VALUES

DIFFERENT IOC TYPES

Your first responsibility is not to declare anything malicious.

Your first responsibility is:

UNDERSTAND THE DATA

Important Security Principle

An IOC is:

AN OBSERVABLE

or:

A PIECE OF SECURITY CONTEXT

An IOC match does not automatically mean:

COMPROMISE CONFIRMED

For example:

IP ADDRESS MATCH
       ↓
REQUIRES CONTEXT
       ↓
TIME
       ↓
DIRECTION
       ↓
ASSET
       ↓
USER
       ↓
SOURCE CONFIDENCE
       ↓
RELATED ACTIVITY

Authorization and Safety

Use:

SYNTHETIC INDICATORS

DOCUMENTATION IP RANGES

TRAINING DOMAINS

LOCALLY GENERATED HASHES

AUTHORIZED THREAT-INTELLIGENCE APIs

Do not submit confidential organizational data, customer information, or sensitive internal indicators to third-party services unless organizational policy explicitly permits it.

01 — Create the Lab Workspace

Create:

ioc-enrichment-pipeline/
|
+-- data/
|
+-- reports/
|
+-- src/
|
+-- cache/
|
+-- tests/
|
+-- README.md

Linux/macOS:

mkdir -p ioc-enrichment-pipeline/{data,reports,src,cache,tests}
cd ioc-enrichment-pipeline

PowerShell:

mkdir ioc-enrichment-pipeline

cd ioc-enrichment-pipeline

mkdir data
mkdir reports
mkdir src
mkdir cache
mkdir tests

02 — Verify Python

Run:

python --version

or:

python3 --version

Recommended:

Python 3.10+

03 — Create the IOC Dataset

Create:

data/iocs.txt

Add:

203.0.113.25
192.0.2.50
198.51.100.75
10.10.10.25
172.16.20.15
Example.COM
example.com
training.example.org
security.example.net
44d88612fea8a8f36de82e1278abb02f
275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f
275A021BBFB6489E54D471899F7DB9D1663FC695EC2FE2A2C4538AABF651FD0F
not-an-ip
bad domain value
999.999.999.999

04 — Understand the Dataset

The file intentionally contains:

IPv4 ADDRESSES

PRIVATE ADDRESSES

DOMAIN NAMES

MD5-LENGTH HASH

SHA-256 HASH

DUPLICATE SHA-256

INVALID DATA

Your application must determine which is which.

05 — IOC Classification Strategy

Use this order:

RAW VALUE
    ↓
SHA-256?
    ↓ NO
IP ADDRESS?
    ↓ NO
DOMAIN?
    ↓ NO
INVALID

Why check hashes first?

Because a hash can look like a long hostname-like string if validation is poorly designed.

06 — Create the Python Application

Create:

src/ioc_processor.py

Add:

from pathlib import Path
import csv
import hashlib
import ipaddress
import json
import re

07 — Define Project Paths

Add:

BASE_DIR = Path(__file__).resolve().parent.parent

INPUT_FILE = BASE_DIR / "data" / "iocs.txt"
REPORT_DIR = BASE_DIR / "reports"
CACHE_DIR = BASE_DIR / "cache"

REPORT_DIR.mkdir(
    parents=True,
    exist_ok=True
)

CACHE_DIR.mkdir(
    parents=True,
    exist_ok=True
)

08 — Create IOC Types

For this lab:

IP

DOMAIN

SHA256

INVALID

Later you could extend the tool to support:

URL

EMAIL

IPv6

SHA1

MD5

FILE NAME

But keep the first version simple.

09 — Validate IP Addresses

Create:

def parse_ip(value):
    try:
        return ipaddress.ip_address(
            value
        )
    except ValueError:
        return None

10 — Test IP Validation

Examples:

203.0.113.25
→ Valid IP

10.10.10.25
→ Valid IP

999.999.999.999
→ Invalid IP

11 — Understand IP Context

Python’s ipaddress module can provide properties such as:

is_private

is_global

is_loopback

is_multicast

version

This allows us to enrich indicators without making any external API request.

12 — Build Local IP Enrichment

Create:

def enrich_ip_local(ip_obj):
    return {
        "ip_version":
            ip_obj.version,

        "is_private":
            ip_obj.is_private,

        "is_global":
            ip_obj.is_global,

        "is_loopback":
            ip_obj.is_loopback,

        "is_multicast":
            ip_obj.is_multicast
    }

13 — Documentation Address Warning

The ranges:

192.0.2.0/24

198.51.100.0/24

203.0.113.0/24

are reserved for documentation and examples.

That makes them useful for safe training material.

Do not interpret these lab addresses as real threat infrastructure.

14 — Validate SHA-256 Hashes

A SHA-256 hash contains:

64 HEXADECIMAL CHARACTERS

Create:

SHA256_PATTERN = re.compile(
    r"^[a-fA-F0-9]{64}$"
)

Then:

def is_sha256(value):
    return bool(
        SHA256_PATTERN.fullmatch(
            value
        )
    )

15 — Normalize SHA-256

Hashes should be normalized to lowercase:

def normalize_sha256(value):
    return value.lower()

Therefore:

275A021B...

and:

275a021b...

become the same value.

16 — What About MD5?

The dataset contains:

44d88612fea8a8f36de82e1278abb02f

This is 32 hexadecimal characters.

Your current lab supports only:

SHA-256

Therefore the value should not be accepted as SHA-256.

This teaches an important principle:

DO NOT GUESS
THE INDICATOR TYPE

17 — Domain Validation

Domain validation requires more care than simply checking for:

"."

Create:

DOMAIN_PATTERN = re.compile(
    r"^(?=.{1,253}$)"
    r"(?:"
    r"[a-zA-Z0-9]"
    r"(?:[a-zA-Z0-9-]{0,61}"
    r"[a-zA-Z0-9])?"
    r"\."
    r")+"
    r"[a-zA-Z]{2,63}$"
)

18 — Create Domain Validator

def is_domain(value):
    return bool(
        DOMAIN_PATTERN.fullmatch(
            value
        )
    )

19 — Normalize Domains

Create:

def normalize_domain(value):
    return (
        value
        .strip()
        .rstrip(".")
        .lower()
    )

Therefore:

Example.COM

becomes:

example.com

20 — Normalize Raw Input

Create:

def normalize_raw_value(value):
    return value.strip()

21 — Build the IOC Classifier

Create:

def classify_ioc(raw_value):
    value = normalize_raw_value(
        raw_value
    )

    if not value:
        return {
            "type": "invalid",
            "value": value,
            "reason": "Empty value"
        }

    if is_sha256(value):
        return {
            "type": "sha256",
            "value":
                normalize_sha256(value)
        }

    ip_obj = parse_ip(value)

    if ip_obj:
        return {
            "type": "ip",
            "value": str(ip_obj)
        }

    domain = normalize_domain(
        value
    )

    if is_domain(domain):
        return {
            "type": "domain",
            "value": domain
        }

    return {
        "type": "invalid",
        "value": value,
        "reason":
            "Unsupported or malformed IOC"
    }

22 — Test the Classifier

Input:

Example.COM

Expected:

{
    "type": "domain",
    "value": "example.com"
}

Input:

10.10.10.25

Expected:

{
    "type": "ip",
    "value": "10.10.10.25"
}

23 — Load the IOC File

Create:

def load_iocs():
    records = []

    with INPUT_FILE.open(
        "r",
        encoding="utf-8"
    ) as file:

        for line_number, line in enumerate(
            file,
            start=1
        ):
            raw = line.strip()

            if not raw:
                continue

            result = classify_ioc(
                raw
            )

            result["line"] = line_number
            result["raw"] = raw

            records.append(
                result
            )

    return records

24 — Test IOC Loading

Temporarily add:

records = load_iocs()

for record in records:
    print(record)

Run:

python src/ioc_processor.py

Review each classification.

25 — Separate Valid and Invalid IOCs

Create:

def split_records(records):
    valid = []
    invalid = []

    for record in records:
        if record["type"] == "invalid":
            invalid.append(record)
        else:
            valid.append(record)

    return valid, invalid

26 — Deduplication

Your dataset contains:

Example.COM

example.com

After normalization both become:

example.com

The SHA-256 value also appears twice using different letter casing.

These should be deduplicated.

27 — Build Deduplication Logic

Create:

def deduplicate(records):
    unique = []
    duplicates = []

    seen = set()

    for record in records:
        key = (
            record["type"],
            record["value"]
        )

        if key in seen:
            duplicates.append(
                record
            )
            continue

        seen.add(key)
        unique.append(record)

    return unique, duplicates

28 — Why Include IOC Type?

Use:

(
    record["type"],
    record["value"]
)

rather than only:

record["value"]

This makes the deduplication model explicit and easier to extend.

29 — Count IOC Types

Import:

from collections import Counter

Create:

def count_types(records):
    return Counter(
        record["type"]
        for record in records
    )

30 — Local Enrichment

Before using external intelligence services, enrich what you can locally.

For IP addresses:

IP VERSION

PRIVATE?

GLOBAL?

LOOPBACK?

MULTICAST?

For domains:

NORMALIZED DOMAIN

TRAINING / EXAMPLE DOMAIN?

For hashes:

ALGORITHM

LENGTH

31 — Enrich IP Records

Create:

def enrich_ip_record(record):
    ip_obj = ipaddress.ip_address(
        record["value"]
    )

    record.update(
        enrich_ip_local(
            ip_obj
        )
    )

    return record

32 — Enrich Domain Records

Because this lab uses example domains, create:

def enrich_domain_record(record):
    domain = record["value"]

    training_suffixes = (
        "example.com",
        "example.net",
        "example.org"
    )

    record["training_domain"] = (
        domain in training_suffixes
        or domain.endswith(
            tuple(
                "." + suffix
                for suffix
                in training_suffixes
            )
        )
    )

    return record

33 — Enrich Hash Records

Create:

def enrich_hash_record(record):
    record["algorithm"] = "sha256"
    record["length"] = len(
        record["value"]
    )

    return record

34 — Build the Local Enrichment Pipeline

Create:

def enrich_locally(records):
    enriched = []

    for record in records:
        item = record.copy()

        if item["type"] == "ip":
            item = enrich_ip_record(
                item
            )

        elif item["type"] == "domain":
            item = enrich_domain_record(
                item
            )

        elif item["type"] == "sha256":
            item = enrich_hash_record(
                item
            )

        enriched.append(item)

    return enriched

35 — Understand Enrichment

Before:

{
  "type": "ip",
  "value": "10.10.10.25"
}

After:

{
  "type": "ip",
  "value": "10.10.10.25",
  "ip_version": 4,
  "is_private": true,
  "is_global": false
}

The indicator now has more:

CONTEXT

36 — External Threat Intelligence

A production workflow might query an approved threat-intelligence service.

Conceptually:

IOC
 ↓
AUTHORIZED TI API
 ↓
REPUTATION / CONTEXT
 ↓
SOURCE
 ↓
CONFIDENCE
 ↓
OBSERVATION DATE

37 — Important External-Enrichment Rule

Before sending indicators to a third-party API ask:

IS THE INDICATOR SENSITIVE?

DOES POLICY ALLOW SHARING?

IS THE PROVIDER APPROVED?

WHAT DATA DOES THE SERVICE RETAIN?

DOES THE REQUEST EXPOSE
INTERNAL INVESTIGATION DETAILS?

38 — API Architecture

A safe design:

IOC PROCESSOR
     ↓
API CLIENT
     ↓
TIMEOUT
     ↓
AUTHORIZED TI API
     ↓
RESPONSE VALIDATION
     ↓
NORMALIZED ENRICHMENT

39 — Do Not Hard-Code API Keys

Avoid:

API_KEY = "real-secret-key"

Instead use:

ENVIRONMENT VARIABLE

or an approved:

SECRET MANAGER

40 — Read an API Token Safely

Example:

import os

api_token = os.getenv(
    "THREAT_INTEL_API_TOKEN"
)

if not api_token:
    print(
        "External enrichment disabled."
    )

This allows the lab to work without requiring a real external service.

41 — Build a Mock Enrichment Service

For the lab, use:

LOCAL MOCK DATA

instead of depending on a real threat-intelligence platform.

Create:

data/mock_intelligence.json

Add:

{
  "203.0.113.25": {
    "source": "training-feed",
    "confidence": 70,
    "classification": "review"
  },
  "example.com": {
    "source": "training-feed",
    "confidence": 10,
    "classification": "training"
  },
  "275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f": {
    "source": "training-feed",
    "confidence": 60,
    "classification": "review"
  }
}

These values are:

SYNTHETIC

and exist only for training.

42 — Load Mock Intelligence

Create:

MOCK_INTEL_FILE = (
    BASE_DIR /
    "data" /
    "mock_intelligence.json"
)

Then:

def load_mock_intelligence():
    if not MOCK_INTEL_FILE.exists():
        return {}

    with MOCK_INTEL_FILE.open(
        "r",
        encoding="utf-8"
    ) as file:

        return json.load(file)

43 — Add Mock Enrichment

Create:

def apply_mock_intelligence(
    records,
    intelligence
):
    results = []

    for record in records:
        item = record.copy()

        context = intelligence.get(
            item["value"]
        )

        if context:
            item["intel_match"] = True
            item["intel"] = context

        else:
            item["intel_match"] = False
            item["intel"] = None

        results.append(item)

    return results

44 — Why Use Mock Intelligence?

It allows students to learn:

LOOKUP

MATCHING

ENRICHMENT

CONFIDENCE

REPORTING

without:

API COST

REAL INDICATOR DISCLOSURE

ACCOUNT REQUIREMENTS

RATE LIMIT DEPENDENCY

45 — Threat Intelligence Confidence

Your synthetic feed contains:

confidence

But confidence should not be interpreted as:

70
=
70% CHANCE THE SYSTEM IS COMPROMISED

Confidence normally relates to the provider’s assessment of the intelligence.

Always understand the source’s definition.

46 — Intelligence Source

Always retain:

SOURCE

For example:

{
  "source": "training-feed",
  "confidence": 70
}

Context without provenance is weaker.

47 — Observation Time

A real enrichment pipeline should also consider:

FIRST SEEN

LAST SEEN

OBSERVATION TIME

FEED UPDATE TIME

because threat intelligence can become stale.

48 — Add Analysis Status

For this lab use:

INFORMATIONAL

REVIEW

Do not automatically label an indicator:

MALICIOUS

simply because it appears in the mock intelligence file.

49 — Create a Review Function

def determine_review_status(record):
    if not record.get(
        "intel_match"
    ):
        return "informational"

    intel = record.get(
        "intel"
    ) or {}

    confidence = intel.get(
        "confidence",
        0
    )

    if confidence >= 50:
        return "review"

    return "informational"

50 — Add Review Status

def add_review_status(records):
    results = []

    for record in records:
        item = record.copy()

        item["review_status"] = (
            determine_review_status(
                item
            )
        )

        results.append(item)

    return results

51 — Why Human Review?

Consider:

IOC MATCH
   +
HIGH CONFIDENCE

Even then you still need:

ASSET CONTEXT

EVENT CONTEXT

TIME CONTEXT

USER CONTEXT

NETWORK DIRECTION

52 — Complete Processing Pipeline

Create:

def process_iocs():
    records = load_iocs()

    valid, invalid = split_records(
        records
    )

    unique, duplicates = deduplicate(
        valid
    )

    enriched = enrich_locally(
        unique
    )

    intelligence = (
        load_mock_intelligence()
    )

    enriched = (
        apply_mock_intelligence(
            enriched,
            intelligence
        )
    )

    enriched = add_review_status(
        enriched
    )

    return {
        "raw_count":
            len(records),

        "valid_count":
            len(valid),

        "invalid_count":
            len(invalid),

        "unique_count":
            len(unique),

        "duplicate_count":
            len(duplicates),

        "type_counts":
            dict(
                count_types(unique)
            ),

        "records":
            enriched,

        "invalid":
            invalid,

        "duplicates":
            duplicates
    }

53 — Test the Complete Pipeline

Temporarily:

results = process_iocs()

print(
    json.dumps(
        results,
        indent=2
    )
)

Run:

python src/ioc_processor.py

54 — What Should You Observe?

Your application should identify:

VALID IPs

VALID DOMAINS

VALID SHA-256

INVALID VALUES

DUPLICATE DOMAINS

DUPLICATE SHA-256

PRIVATE IP CONTEXT

TRAINING DOMAIN CONTEXT

MOCK INTELLIGENCE MATCHES

55 — Export Valid IOCs

Create:

def export_valid_iocs(records):
    output = (
        REPORT_DIR /
        "valid-iocs.csv"
    )

    with output.open(
        "w",
        newline="",
        encoding="utf-8"
    ) as file:

        writer = csv.writer(file)

        writer.writerow([
            "type",
            "value",
            "review_status",
            "intel_match"
        ])

        for record in records:
            writer.writerow([
                record["type"],
                record["value"],
                record["review_status"],
                record["intel_match"]
            ])

56 — Export Invalid IOCs

Create:

def export_invalid_iocs(records):
    output = (
        REPORT_DIR /
        "invalid-iocs.csv"
    )

    with output.open(
        "w",
        newline="",
        encoding="utf-8"
    ) as file:

        writer = csv.writer(file)

        writer.writerow([
            "line",
            "raw",
            "reason"
        ])

        for record in records:
            writer.writerow([
                record["line"],
                record["raw"],
                record.get(
                    "reason",
                    ""
                )
            ])

57 — Export Duplicates

Create:

def export_duplicates(records):
    output = (
        REPORT_DIR /
        "duplicates.csv"
    )

    with output.open(
        "w",
        newline="",
        encoding="utf-8"
    ) as file:

        writer = csv.writer(file)

        writer.writerow([
            "line",
            "type",
            "raw",
            "normalized_value"
        ])

        for record in records:
            writer.writerow([
                record["line"],
                record["type"],
                record["raw"],
                record["value"]
            ])

58 — Export Enriched JSON

Create:

def export_enriched_json(
    results
):
    output = (
        REPORT_DIR /
        "enriched-iocs.json"
    )

    with output.open(
        "w",
        encoding="utf-8"
    ) as file:

        json.dump(
            results,
            file,
            indent=2
        )

59 — Generate Analyst Report

Create:

def generate_report(results):
    lines = []

    lines.append(
        "# IOC Processing Report"
    )

    lines.append("")

    lines.append(
        "## Executive Summary"
    )

    lines.append("")

    lines.append(
        f"- Raw indicators: "
        f"{results['raw_count']}"
    )

    lines.append(
        f"- Valid indicators: "
        f"{results['valid_count']}"
    )

    lines.append(
        f"- Invalid indicators: "
        f"{results['invalid_count']}"
    )

    lines.append(
        f"- Unique indicators: "
        f"{results['unique_count']}"
    )

    lines.append(
        f"- Duplicates: "
        f"{results['duplicate_count']}"
    )

    lines.append("")

    lines.append(
        "## IOC Types"
    )

    lines.append("")

    for ioc_type, count in (
        results["type_counts"].items()
    ):
        lines.append(
            f"- {ioc_type}: {count}"
        )

    lines.append("")

    lines.append(
        "## Indicators Requiring Review"
    )

    lines.append("")

    review_records = [
        record
        for record
        in results["records"]
        if record["review_status"]
        == "review"
    ]

    if review_records:
        for record in review_records:
            lines.append(
                f"- {record['type']}: "
                f"{record['value']}"
            )
    else:
        lines.append(
            "- No indicators currently "
            "require review."
        )

    lines.append("")

    lines.append(
        "## Analyst Guidance"
    )

    lines.append("")

    lines.append(
        "IOC enrichment provides context, "
        "not automatic proof of compromise. "
        "Correlate relevant indicators with "
        "endpoint, network, identity, asset, "
        "and timeline evidence before making "
        "an incident determination."
    )

    output = (
        REPORT_DIR /
        "ioc-investigation-report.md"
    )

    output.write_text(
        "\n".join(lines),
        encoding="utf-8"
    )

60 — Create main()

def main():
    results = process_iocs()

    export_valid_iocs(
        results["records"]
    )

    export_invalid_iocs(
        results["invalid"]
    )

    export_duplicates(
        results["duplicates"]
    )

    export_enriched_json(
        results
    )

    generate_report(
        results
    )

    print(
        "IOC processing complete."
    )

    print(
        f"Reports saved to: "
        f"{REPORT_DIR}"
    )

61 — Add Entry Point

if __name__ == "__main__":
    main()

62 — Run the Pipeline

Run:

python src/ioc_processor.py

Expected:

IOC processing complete.
Reports saved to: ...

63 — Review Your Reports

Your directory should contain:

reports/
|
+-- valid-iocs.csv
|
+-- invalid-iocs.csv
|
+-- duplicates.csv
|
+-- enriched-iocs.json
|
+-- ioc-investigation-report.md

64 — Review Valid Indicators

Example:

type,value,review_status,intel_match
ip,203.0.113.25,review,True
ip,192.0.2.50,informational,False
domain,example.com,informational,True
sha256,275a...,review,True

65 — Review Invalid Indicators

You should see entries such as:

not-an-ip

bad domain value

999.999.999.999

You may also see the unsupported MD5-length value classified as invalid.

66 — Review Duplicate Indicators

You should identify normalization-driven duplicates such as:

Example.COM
example.com

and the uppercase/lowercase SHA-256 pair.

67 — Build an IOC Processing Matrix

Stage	Purpose
Ingest	Read raw indicators
Validate	Reject malformed values
Normalize	Create consistent representation
Deduplicate	Remove repeated indicators
Classify	Determine IOC type
Enrich	Add useful context
Prioritize	Identify review candidates
Report	Provide analyst-ready output

68 — Add Logging

Import:

import logging

Configure:

logging.basicConfig(
    level=logging.INFO,
    format=(
        "%(asctime)s "
        "%(levelname)s "
        "%(message)s"
    )
)

Add:

logging.info(
    "IOC processing started"
)

and:

logging.info(
    "IOC processing completed"
)

69 — Do Not Log Secrets

If you later use external APIs, do not log:

API KEYS

AUTHORIZATION HEADERS

TOKENS

SESSION DATA

70 — Error Handling

Handle missing input:

try:
    results = process_iocs()

except FileNotFoundError:
    print(
        "IOC input file not found."
    )
    raise SystemExit(1)

71 — Test Missing Input

Temporarily rename:

iocs.txt

Run:

python src/ioc_processor.py

Verify the application:

FAILS CLEANLY

Restore the file afterward.

72 — Test Empty Input

Create an empty IOC file.

The application should return:

0 indicators

without crashing.

73 — Test Case Normalization

Add:

EXAMPLE.COM
Example.Com
example.com

Expected normalized value:

example.com

Expected unique count:

1

for those three domain entries.

74 — Test SHA-256 Normalization

Add the same hash as:

LOWERCASE

UPPERCASE

Verify that the second occurrence becomes:

DUPLICATE

75 — Test Invalid Hashes

Try:

12345

and:

gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg

Neither should be accepted as SHA-256.

76 — Test IPv6

Because:

ipaddress.ip_address()

supports IPv6, try a documentation address such as:

2001:db8::10

Your classifier should identify it as:

IP

77 — Add IOC Source Metadata

Real indicators should ideally include:

SOURCE

COLLECTION TIME

CASE ID

CONFIDENCE

TAGS

Instead of only:

indicator

78 — Better IOC Schema

A more mature internal record might look like:

{
  "indicator": "example.com",
  "type": "domain",
  "source": "training-feed",
  "first_seen": "2026-08-29T09:00:00Z",
  "confidence": 50,
  "tags": [
    "training"
  ]
}

79 — Provenance

Always ask:

WHERE DID THIS IOC COME FROM?

IOC provenance matters because indicators may originate from:

INTERNAL DETECTION

COMMERCIAL INTELLIGENCE

OPEN-SOURCE INTELLIGENCE

PARTNER FEED

INCIDENT INVESTIGATION

80 — Freshness

Indicators can become stale.

A domain may:

CHANGE OWNERSHIP

An IP may:

BE REASSIGNED

Infrastructure may:

CHANGE PURPOSE

Therefore:

IOC
+
TIME
+
SOURCE

is much more useful than an IOC alone.

81 — Expiration

A mature system may track:

FIRST SEEN

LAST SEEN

EXPIRATION

REVIEW DATE

Do not treat intelligence as permanently valid.

82 — Confidence vs Severity

Do not confuse:

CONFIDENCE

with:

SEVERITY

Confidence asks:

HOW MUCH DO WE TRUST
THIS INTELLIGENCE?

Severity asks:

WHAT COULD THE
SECURITY IMPACT BE?

83 — Add Asset Context

Suppose a matching IP is observed communicating with:

DEVELOPMENT TEST SYSTEM

versus:

DOMAIN CONTROLLER

The investigation priority may be very different.

84 — Contextual Enrichment Model

IOC MATCH
   +
ASSET CRITICALITY
   +
USER PRIVILEGE
   +
EVENT TYPE
   +
TIME
   +
SOURCE CONFIDENCE
   =
INVESTIGATION PRIORITY

85 — IOC Match Workflow

A mature SOC workflow could use:

IOC MATCH
   ↓
VALIDATE INTELLIGENCE
   ↓
CHECK FRESHNESS
   ↓
IDENTIFY AFFECTED ASSET
   ↓
IDENTIFY USER
   ↓
CHECK RELATED EVENTS
   ↓
BUILD TIMELINE
   ↓
ANALYST REVIEW

86 — Avoid Automatic Blocking

Do not design the first version as:

IOC MATCH
   ↓
BLOCK EVERYTHING

False positives or stale intelligence could interrupt legitimate services.

Prefer:

IOC MATCH
   ↓
ENRICH
   ↓
CORRELATE
   ↓
REVIEW
   ↓
APPROVED RESPONSE

87 — Caching

External enrichment can be:

SLOW

RATE LIMITED

EXPENSIVE

A cache can reduce repeated lookups.

Conceptually:

IOC
 ↓
CACHE?
 ├── YES → USE CACHED CONTEXT
 │
 └── NO
      ↓
     API
      ↓
    CACHE RESULT

88 — Cache Expiration

Do not cache intelligence forever.

Store:

LOOKUP TIME

and define:

CACHE TTL

based on the data source and operational requirements.

89 — API Failure Handling

Your external enrichment design should expect:

TIMEOUT

HTTP ERROR

RATE LIMIT

INVALID JSON

AUTHENTICATION FAILURE

SERVICE OUTAGE

90 — Graceful Degradation

If enrichment fails:

IOC PROCESSING
SHOULD NOT NECESSARILY FAIL
COMPLETELY

Instead:

VALIDATE
   ↓
NORMALIZE
   ↓
LOCAL ENRICHMENT
   ↓
EXTERNAL API FAILED
   ↓
MARK ENRICHMENT UNAVAILABLE
   ↓
CONTINUE REPORT

91 — Retry Strategy

For temporary API failures:

REQUEST
   ↓
FAIL
   ↓
WAIT
   ↓
RETRY

Use:

LIMITED RETRIES

BACKOFF

LOGGING

Never create:

INFINITE RETRIES

92 — Rate Limiting

If an API allows:

100 REQUESTS / MINUTE

your automation must respect that limit.

Otherwise:

AUTOMATION
   ↓
TOO MANY REQUESTS
   ↓
RATE LIMIT
   ↓
FAILED PIPELINE

93 — Batch Processing

For large datasets:

10,000 IOCs

process in manageable batches.

Example:

BATCH 01
  ↓
ENRICH
  ↓
SAVE

BATCH 02
  ↓
ENRICH
  ↓
SAVE

94 — Resume Capability

A mature pipeline should avoid restarting:

10,000 LOOKUPS

because lookup:

9,999

failed.

Persist processing state where appropriate.

95 — Generate a File Hash

To demonstrate hashing safely, create:

data/sample.txt

Add:

GoHackersCloud IOC Processing Lab

Calculate SHA-256:

def calculate_sha256(path):
    sha256 = hashlib.sha256()

    with path.open("rb") as file:
        for block in iter(
            lambda: file.read(8192),
            b""
        ):
            sha256.update(block)

    return sha256.hexdigest()

96 — Run the Hash Function

Example:

sample_file = (
    BASE_DIR /
    "data" /
    "sample.txt"
)

print(
    calculate_sha256(
        sample_file
    )
)

This demonstrates how file indicators can be generated from local lab artifacts.

97 — Hashing Mental Model

FILE
 ↓
SHA-256
 ↓
FIXED-LENGTH VALUE
 ↓
COMPARE

Hashes can support:

FILE IDENTIFICATION

INTEGRITY CHECKING

IOC MATCHING

98 — Hash Limitations

The same hash strongly identifies the same file content.

But:

ONE BYTE CHANGED

creates a different cryptographic hash.

Therefore hash-based indicators can be:

HIGHLY SPECIFIC

but:

FRAGILE TO FILE MODIFICATION

99 — Create a Test Suite

Create:

tests/test_ioc_processor.py

100 — Test IP Parsing

def test_valid_ip():
    assert (
        parse_ip(
            "10.10.10.25"
        )
        is not None
    )

101 — Test Invalid IP

def test_invalid_ip():
    assert (
        parse_ip(
            "999.999.999.999"
        )
        is None
    )

102 — Test SHA-256

def test_sha256():
    value = (
        "275a021bbfb6489e54d471899f7db9d"
        "1663fc695ec2fe2a2c4538aabf651fd0f"
    )

    assert is_sha256(value)

103 — Test Domain Normalization

def test_domain_normalization():
    assert (
        normalize_domain(
            "Example.COM"
        )
        == "example.com"
    )

104 — Test Deduplication

Provide:

Example.COM

example.com

Verify:

UNIQUE = 1

DUPLICATE = 1

105 — Test Unsupported Input

Provide:

this is not an indicator

Expected:

INVALID

106 — Analyst Investigation Exercise

Assume your enriched output identifies:

IOC
203.0.113.25

TYPE
IP

INTELLIGENCE MATCH
YES

CONFIDENCE
70

STATUS
REVIEW

What should you do next?

Do not immediately declare:

INCIDENT

Investigate:

Where was the IP observed?

Was traffic inbound or outbound?

Which asset communicated with it?

Which user was involved?

When did the communication occur?

Was it blocked?

Was the indicator current at that time?

Are related alerts present?

Is the intelligence source trustworthy?

107 — Analyst Investigation Exercise 02

Suppose:

example.com

matches your training intelligence.

But enrichment says:

training_domain = true

This demonstrates why:

MULTIPLE CONTEXT SOURCES

matter.

A simple:

MATCH = TRUE

is not enough.

108 — Analyst Investigation Exercise 03

Suppose a private IP appears:

10.10.10.25

An external reputation API may not be meaningful for it.

Instead investigate using:

INTERNAL ASSET INVENTORY

DHCP DATA

IDENTITY DATA

EDR DATA

NETWORK TELEMETRY

109 — Internal vs External Enrichment

PUBLIC IP
   ↓
EXTERNAL + INTERNAL CONTEXT

PRIVATE IP
   ↓
INTERNAL CONTEXT

This is an important SOC automation design principle.

110 — Build an Investigation Package

For every review candidate, ideally collect:

IOC

TYPE

SOURCE

CONFIDENCE

FIRST SEEN

LAST SEEN

AFFECTED ASSET

AFFECTED USER

RELATED ALERTS

REVIEW STATUS

ANALYST NOTES

111 — Portfolio Enhancement

Your Git repository should look like:

ioc-enrichment-pipeline/
|
+-- data/
|   +-- iocs.txt
|   +-- mock_intelligence.json
|   +-- sample.txt
|
+-- src/
|   +-- ioc_processor.py
|
+-- tests/
|   +-- test_ioc_processor.py
|
+-- reports/
|
+-- cache/
|
+-- README.md
|
+-- architecture.md

112 — README Structure

Include:

PROJECT OVERVIEW

SECURITY USE CASE

ARCHITECTURE

SUPPORTED IOC TYPES

INSTALLATION

USAGE

INPUT FORMAT

OUTPUT FORMAT

VALIDATION LOGIC

ENRICHMENT MODEL

TESTING

SECURITY CONSIDERATIONS

LIMITATIONS

FUTURE IMPROVEMENTS

113 — Document Security Considerations

Include:

No real threat indicators required

No production credentials included

No automatic blocking

No automatic containment

External API integration disabled by default

Analyst validation required

114 — Document Limitations

Your first version does not provide:

FULL URL PARSING

EMAIL INDICATORS

WHOIS

PASSIVE DNS

ASN ENRICHMENT

REAL-TIME TI FEEDS

HISTORICAL INTELLIGENCE

ASSET CORRELATION

IDENTITY CORRELATION

These are future improvements.

115 — Advanced Enhancement: URLs

A later version could support:

https://example.com/path

But URL normalization requires careful handling of:

SCHEME

HOST

PORT

PATH

QUERY STRING

Do not simply lowercase an entire URL because paths and parameters may be case-sensitive.

116 — Advanced Enhancement: IOC Relationships

Indicators often have relationships:

DOMAIN
  ↓
RESOLVES TO
  ↓
IP

or:

FILE HASH
    ↓
CONTACTS
    ↓
DOMAIN

A mature platform can model these relationships.

117 — IOC Graph Mental Model

         DOMAIN
        /      \
       ↓        ↓
      IP       HASH
       \        /
        ↓      ↓
          EVENT
            ↓
           ASSET
            ↓
           USER

This moves analysis from:

LIST OF INDICATORS

toward:

SECURITY RELATIONSHIPS

118 — Advanced Enhancement: STIX/TAXII

Threat-intelligence platforms may exchange structured information using technologies such as:

STIX

TAXII

You do not need to implement them in this lab.

Understand the concept:

STANDARDIZED
THREAT INTELLIGENCE
EXCHANGE

119 — Advanced Enhancement: SIEM Integration

Future architecture:

SIEM ALERT
    ↓
IOC EXTRACTOR
    ↓
IOC PROCESSOR
    ↓
THREAT INTELLIGENCE
    ↓
ENRICHED EVENT
    ↓
SOC ANALYST

120 — Advanced Enhancement: SOAR

A security orchestration platform could execute:

ALERT
 ↓
EXTRACT IOC
 ↓
ENRICH
 ↓
CORRELATE
 ↓
CREATE CASE
 ↓
ANALYST REVIEW

The same programming principles from this lab still apply.

121 — Common IOC Processing Mistakes

Avoid:

NO VALIDATION

NO NORMALIZATION

NO DEDUPLICATION

TRUSTING EVERY FEED EQUALLY

IGNORING INTELLIGENCE AGE

CONFUSING CONFIDENCE WITH SEVERITY

AUTOMATICALLY BLOCKING EVERY MATCH

SENDING SENSITIVE IOCs TO UNAPPROVED SERVICES

HARDCODING API KEYS

NO ERROR HANDLING

NO SOURCE ATTRIBUTION

NO HUMAN REVIEW

122 — Final Lab Validation

Confirm:

• Python environment works
• IOC file loads correctly
• IP addresses are validated
• IPv6 can be recognized
• Domains are validated
• Domains are normalized
• SHA-256 hashes are validated
• Hashes are normalized
• Unsupported indicators are rejected
• Duplicate domains are removed
• Duplicate hashes are removed
• Private IP context is identified
• Local enrichment works
• Mock intelligence loads
• Intelligence matches are recorded
• Confidence is retained
• Review status is calculated
• Valid IOC CSV is generated
• Invalid IOC CSV is generated
• Duplicate report is generated
• JSON report is generated
• Markdown analyst report is generated
• No secrets are hard-coded
• No automatic blocking occurs
• Limitations are documented

Mission Review

You started with:

RAW IOC LIST

containing:

VALID DATA

INVALID DATA

DUPLICATES

MULTIPLE IOC TYPES

You transformed it into:

RAW IOC
   ↓
VALIDATE
   ↓
CLASSIFY
   ↓
NORMALIZE
   ↓
DEDUPLICATE
   ↓
ENRICH
   ↓
PRIORITIZE
   ↓
REPORT

What You Built

You now have a reusable defensive pipeline capable of processing:

IP ADDRESSES

DOMAIN NAMES

SHA-256 HASHES

while handling:

INVALID DATA

DUPLICATES

LOCAL CONTEXT

INTELLIGENCE MATCHES

CONFIDENCE

ANALYST REVIEW

Key Security Lesson

The central lesson from this lab is:

IOC MATCH
≠
CONFIRMED COMPROMISE

Instead:

IOC
  +
SOURCE
  +
FRESHNESS
  +
CONFIDENCE
  +
ASSET CONTEXT
  +
IDENTITY CONTEXT
  +
RELATED EVENTS
  +
ANALYST JUDGMENT
  =
USEFUL SECURITY INTELLIGENCE

Final Mental Model

Whenever you receive threat indicators, think:

WHAT IS IT?
    ↓
IS IT VALID?
    ↓
HOW SHOULD IT BE NORMALIZED?
    ↓
HAVE I SEEN IT BEFORE?
    ↓
WHERE DID IT COME FROM?
    ↓
HOW CURRENT IS IT?
    ↓
WHAT CONTEXT CAN I ADD?
    ↓
WHERE WAS IT OBSERVED?
    ↓
WHAT ASSET / USER IS INVOLVED?
    ↓
DOES IT REQUIRE REVIEW?

The goal is not:

COLLECT MORE IOCs

The goal is:

TURN RAW INDICATORS
INTO USEFUL
INVESTIGATION CONTEXT

What’s Next?

➡️ Lab 03 — Linux Security Automation with Bash

The next lab moves from Python-based threat-intelligence processing into operating-system security automation.

You will build:

LINUX HOST
    ↓
BASH SECURITY SCRIPT
    ↓
SYSTEM INVENTORY
    ↓
USER REVIEW
    ↓
PERMISSION REVIEW
    ↓
PROCESS REVIEW
    ↓
SERVICE REVIEW
    ↓
LISTENING PORT REVIEW
    ↓
AUTHENTICATION LOG REVIEW
    ↓
SECURITY REPORT

You will combine:

BASH

grep

awk

sort

uniq

find

ss

systemctl

journalctl

to build a reusable Linux security assessment workflow for systems you own or are explicitly authorized to administer.