Runbook 03 Enterprise SOC Threat Hunting
Module: 13 – Security Operations Centre (SOC)
Enterprise Lab: 03
Estimated Time: 6–8 Hours
Difficulty: ⭐⭐⭐⭐⭐
Role: Tier-3 SOC Analyst / Cloud Threat Hunter / Detection Engineer
Estimated Cost: CloudWatch Logs Insights, Amazon Security Lake, Amazon Athena, AWS CloudTrail Lake, Amazon Detective, Lambda, EventBridge, and data storage may incur charges. Review service pricing and monitor AWS Cost Explorer throughout the runbook.
🎯 Objective
Section titled “🎯 Objective”As a Tier-3 SOC Analyst and Cloud Detection Engineer at CloudNova Technologies, you have been assigned to conduct a proactive threat-hunting exercise across the enterprise AWS environment.
Unlike alert-driven incident investigation, threat hunting begins with a hypothesis.
Your responsibility is to search for suspicious behaviour that may not have triggered an existing security alert, identify gaps in the current detection programme, create new detection logic, validate the detections safely, and improve the organisation’s SOC monitoring capability.
By completing this runbook, you will learn how to:
- Plan an enterprise cloud threat-hunting operation
- Develop threat-hunting hypotheses
- Identify high-value AWS telemetry sources
- Search CloudTrail activity for suspicious behaviour
- Hunt for compromised IAM identities
- Hunt for privilege-escalation techniques
- Detect persistence mechanisms
- Hunt for defence-evasion activity
- Analyse suspicious EC2 and network behaviour
- Hunt for data discovery and exfiltration
- Use Amazon Security Lake and Athena
- Use CloudWatch Logs Insights
- Use AWS CloudTrail Lake
- Correlate findings across multiple AWS accounts and Regions
- Map attacker behaviour to MITRE ATT&CK
- Create detection rules and monitoring logic
- Test new detections safely
- Measure detection effectiveness
- Document threat-hunting findings
- Build a continuous SOC improvement roadmap
🏢 Business Scenario
Section titled “🏢 Business Scenario”CloudNova Technologies has recently completed an investigation into a compromised IAM identity.
The incident was successfully contained, but the post-incident review identified several concerns:
- The attacker operated for several hours before the first high-severity alert
- Some malicious API calls did not generate GuardDuty findings
- CloudTrail events were available but were not actively monitored
- Several AWS Regions were not included in regular SOC searches
- Privilege-escalation activity was detected manually
- Suspicious role-trust changes did not trigger alerts
- No detection existed for newly created access keys followed by sensitive API activity
- SOC analysts relied heavily on vendor-generated alerts
- Existing detections produced several false positives
The CISO has requested a proactive threat-hunting and detection-engineering exercise.
As the Tier-3 SOC Analyst, you must search the environment for similar attacker behaviour, identify control gaps, create improved detections, and present a continuous-improvement plan to SOC leadership.
📋 Prerequisites
Section titled “📋 Prerequisites”Complete:
- Enterprise Runbook 01 — Build an Enterprise AWS Security Operations Centre
- Enterprise Runbook 02 — Enterprise SOC Security Monitoring & Incident Investigation
Required access:
- AWS Security Account
- Log Archive Account
- Member workload accounts
- Amazon GuardDuty
- AWS Security Hub
- Amazon Detective
- AWS CloudTrail
- AWS CloudTrail Lake, where configured
- CloudWatch Logs
- VPC Flow Logs
- AWS Config
- IAM Access Analyzer
- Amazon Security Lake
- Amazon Athena
- Amazon EventBridge
- AWS Lambda
- AWS CLI
- Visual Studio Code or Obsidian
🏗 Threat-Hunting Architecture
Section titled “🏗 Threat-Hunting Architecture”Enterprise AWS Accounts
├── AWS CloudTrail├── CloudTrail Lake├── VPC Flow Logs├── Route 53 Resolver Logs├── AWS Config├── GuardDuty├── Security Hub├── IAM Access Analyzer├── Amazon Inspector├── AWS WAF Logs└── Network Firewall Logs
│ ▼
Central Security Data Layer
├── Log Archive Account├── Amazon Security Lake├── Amazon S3├── Amazon Athena├── CloudWatch Logs Insights└── Amazon OpenSearch / External SIEM
│ ▼
Threat-Hunting and Detection Engineering
├── Hypothesis Development├── Behavioural Queries├── Entity Correlation├── ATT&CK Mapping├── Detection Development├── Detection Testing└── Detection Tuning
│ ▼
SOC Operations
├── New Alerts├── Updated Playbooks├── Investigation Runbooks├── Automation├── Dashboards└── Continuous ImprovementMission Information
Section titled “Mission Information”| Field | Value |
|---|---|
| Hunt ID | CN-HUNT-2026-003 |
| Hunt Name | AWS Identity Abuse and Persistence |
| Hunt Owner | Tier-3 SOC / Detection Engineering |
| Hunt Scope | All AWS Accounts and Regions |
| Priority | High |
| Investigation Window | Previous 30 Days |
| Primary Data Source | AWS CloudTrail |
| Supporting Data Sources | GuardDuty, Security Hub, VPC Flow Logs, AWS Config, Security Lake |
| Status | Active |
Step 1 — Define the Threat-Hunting Scope
Section titled “Step 1 — Define the Threat-Hunting Scope”Document the scope before running queries.
Include:
- AWS accounts
- AWS Regions
- Business units
- Production environments
- Development environments
- Shared-services accounts
- Security accounts
- Identity types
- Workload types
- Investigation period
- Data sources
Threat-Hunting Scope
Section titled “Threat-Hunting Scope”| Scope Area | Analyst Entry |
|---|---|
| AWS Accounts | |
| AWS Regions | |
| Investigation Period | |
| Business Units | |
| Critical Workloads | |
| Identity Types | |
| Primary Data Sources | |
| Excluded Systems | |
| Known Limitations |
Confirm that the hunt includes all enabled AWS Regions.
Attackers may intentionally operate in less frequently monitored Regions.
Step 2 — Develop Threat-Hunting Hypotheses
Section titled “Step 2 — Develop Threat-Hunting Hypotheses”Create clear, testable hypotheses.
Hypothesis 1 — Compromised IAM Credential
Section titled “Hypothesis 1 — Compromised IAM Credential”An attacker may be using a compromised IAM access key from an unusual IP address or Region to perform resource discovery and privilege escalation.Hypothesis 2 — Unauthorised Privilege Escalation
Section titled “Hypothesis 2 — Unauthorised Privilege Escalation”An identity may have attached an administrative policy, modified an IAM role, or changed a trust policy without an approved change record.Hypothesis 3 — Persistence
Section titled “Hypothesis 3 — Persistence”An attacker may have created an access key, IAM user, role, Lambda function, or scheduled event to maintain access.Hypothesis 4 — Defence Evasion
Section titled “Hypothesis 4 — Defence Evasion”An attacker may have modified or disabled CloudTrail, AWS Config, GuardDuty, Security Hub, or logging controls.Hypothesis 5 — Data Exfiltration
Section titled “Hypothesis 5 — Data Exfiltration”An identity may have accessed or downloaded an unusual volume of sensitive S3 data before communicating with an external destination.Hypothesis Register
Section titled “Hypothesis Register”| Hypothesis ID | Description | Data Sources | Priority |
|---|---|---|---|
| H-001 | Compromised IAM credential | CloudTrail, Detective | High |
| H-002 | Privilege escalation | CloudTrail, IAM | Critical |
| H-003 | Persistence mechanism | CloudTrail, Config | High |
| H-004 | Defence evasion | CloudTrail, Security Hub | Critical |
| H-005 | Data exfiltration | S3 data events, Flow Logs | Critical |
Step 3 — Confirm Data Availability
Section titled “Step 3 — Confirm Data Availability”Verify that required telemetry is available.
Review:
- Organization CloudTrail
- Multi-Region logging
- Global service events
- S3 data events
- Lambda data events
- CloudTrail Lake event data stores
- CloudWatch Logs retention
- VPC Flow Logs
- Route 53 Resolver query logs
- Security Lake sources
- AWS Config recorder status
- GuardDuty coverage
- Security Hub aggregation
AWS CLI
Section titled “AWS CLI”aws cloudtrail describe-trailsaws cloudtrail get-trail-status \ --name YOUR_TRAIL_NAMEaws cloudtrail list-event-data-storesaws configservice describe-configuration-recorder-statusDocument any visibility gaps.
| Data Source | Enabled | Retention | Accounts Covered | Regions Covered | Gap |
|---|---|---|---|---|---|
| CloudTrail | |||||
| VPC Flow Logs | |||||
| AWS Config | |||||
| Security Lake | |||||
| GuardDuty |
Step 4 — Identify High-Risk Identities
Section titled “Step 4 — Identify High-Risk Identities”Create a list of identities requiring priority monitoring.
Include:
- Root accounts
- IAM users
- Administrative roles
- Break-glass roles
- CI/CD roles
- Security roles
- Cross-account roles
- Service roles
- Third-party integration roles
- Dormant identities
AWS CLI
Section titled “AWS CLI”List IAM users:
aws iam list-usersList roles:
aws iam list-rolesGenerate credential report:
aws iam generate-credential-reportRetrieve the report:
aws iam get-credential-reportIdentify:
- Users without MFA
- Old access keys
- Multiple active access keys
- Unused identities
- Users with direct administrative permissions
- Roles with external trust relationships
Step 5 — Hunt for Unusual Console Logins
Section titled “Step 5 — Hunt for Unusual Console Logins”Search for:
- Root-account logins
- Console logins without MFA
- Failed console logins
- Logins from unusual countries
- Logins outside normal working hours
- First-time source IP addresses
- Multiple identities using the same IP address
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, userIdentity.type, userIdentity.arn, sourceIPAddress, userAgent, additionalEventData.MFAUsed, responseElements.ConsoleLogin| filter eventName = "ConsoleLogin"| sort @timestamp descSearch for successful logins without MFA:
fields @timestamp, userIdentity.arn, sourceIPAddress, additionalEventData.MFAUsed, responseElements.ConsoleLogin| filter eventName = "ConsoleLogin"| filter responseElements.ConsoleLogin = "Success"| filter additionalEventData.MFAUsed = "No"| sort @timestamp descSearch for root logins:
fields @timestamp, userIdentity.type, sourceIPAddress, userAgent, responseElements.ConsoleLogin| filter eventName = "ConsoleLogin"| filter userIdentity.type = "Root"| sort @timestamp descAWS CLI
Section titled “AWS CLI”aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLoginStep 6 — Hunt for Newly Created Access Keys
Section titled “Step 6 — Hunt for Newly Created Access Keys”Search for:
CreateAccessKeyUpdateAccessKeyDeleteAccessKey- Access-key usage shortly after creation
- Keys created outside normal working hours
- Keys created by unusual administrators
- Keys used from unfamiliar IP addresses
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, userIdentity.arn, sourceIPAddress, requestParameters.userName, responseElements.accessKey.accessKeyId| filter eventName = "CreateAccessKey"| sort @timestamp descReview every newly created key.
Determine:
- Who created it
- For which user
- Whether a change request exists
- When it was first used
- Which services it accessed
- Whether it was used from a new Region
Step 7 — Hunt for Privilege-Escalation Activity
Section titled “Step 7 — Hunt for Privilege-Escalation Activity”Search for:
AttachUserPolicyAttachRolePolicyPutUserPolicyPutRolePolicyCreatePolicyVersionSetDefaultPolicyVersionAddUserToGroupUpdateAssumeRolePolicyPassRoleCreateRoleCreateLoginProfile
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, eventName, userIdentity.arn, sourceIPAddress, requestParameters| filter eventName in [ "AttachUserPolicy", "AttachRolePolicy", "PutUserPolicy", "PutRolePolicy", "CreatePolicyVersion", "SetDefaultPolicyVersion", "AddUserToGroup", "UpdateAssumeRolePolicy", "PassRole", "CreateRole"]| sort @timestamp descPrioritise events involving:
AdministratorAccessiam:PassRolests:AssumeRoleAction: "*"Resource: "*"Step 8 — Hunt for IAM Role-Trust Modifications
Section titled “Step 8 — Hunt for IAM Role-Trust Modifications”Search for:
UpdateAssumeRolePolicy- Newly trusted external AWS accounts
- Federated identity-provider changes
- Wildcard principals
- New service principals
- Cross-account role assumptions
AWS CLI
Section titled “AWS CLI”aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateAssumeRolePolicyReview role trust policies:
aws iam get-role \ --role-name YOUR_ROLE_NAMELook for:
{ "Principal": { "AWS": "*" }}or unexpected external account IDs.
Document:
| Role | Previous Trust | Current Trust | Changed By | Approved |
|---|---|---|---|---|
Step 9 — Hunt for Suspicious AssumeRole Activity
Section titled “Step 9 — Hunt for Suspicious AssumeRole Activity”Search for:
AssumeRole- Unusual source identities
- Role assumptions from external accounts
- Role chaining
- High-privilege role assumptions
- Assumptions outside normal working hours
- Short bursts of activity across multiple Regions
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, userIdentity.arn, sourceIPAddress, requestParameters.roleArn, requestParameters.roleSessionName, responseElements.credentials.accessKeyId| filter eventName = "AssumeRole"| sort @timestamp descQuestions:
- Is the source identity authorised to assume the role?
- Is the role session name expected?
- Was MFA required?
- Was the role used in a new AWS Region?
- Did sensitive API calls follow the role assumption?
Step 10 — Hunt for Persistence Mechanisms
Section titled “Step 10 — Hunt for Persistence Mechanisms”Search for:
- New IAM users
- New IAM roles
- New access keys
- New login profiles
- New Lambda functions
- Modified Lambda functions
- New EventBridge scheduled rules
- New CloudWatch Events rules
- New EC2 key pairs
- New EC2 instances
- New Secrets Manager secrets
- Changes to Systems Manager documents
Events of Interest
Section titled “Events of Interest”CreateUser
CreateAccessKey
CreateLoginProfile
CreateRole
UpdateAssumeRolePolicy
CreateFunction
UpdateFunctionCode
PutRule
PutTargets
CreateKeyPair
RunInstances
CreateSecret
CreateDocumentCloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, eventName, userIdentity.arn, sourceIPAddress, requestParameters| filter eventName in [ "CreateUser", "CreateAccessKey", "CreateLoginProfile", "CreateRole", "CreateFunction", "UpdateFunctionCode", "PutRule", "PutTargets", "CreateKeyPair", "RunInstances", "CreateSecret"]| sort @timestamp descStep 11 — Hunt for Security-Control Tampering
Section titled “Step 11 — Hunt for Security-Control Tampering”Search for activity involving:
CloudTrail
Section titled “CloudTrail”StopLoggingDeleteTrailUpdateTrailPutEventSelectorsDeleteEventDataStore
GuardDuty
Section titled “GuardDuty”DeleteDetectorUpdateDetectorDisassociateMembersStopMonitoringMembers
Security Hub
Section titled “Security Hub”DisableSecurityHubBatchDisableStandardsUpdateStandardsControl
AWS Config
Section titled “AWS Config”StopConfigurationRecorderDeleteConfigurationRecorderDeleteDeliveryChannel
CloudWatch
Section titled “CloudWatch”DeleteLogGroupDeleteMetricFilterDeleteAlarms
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, eventName, eventSource, userIdentity.arn, sourceIPAddress, requestParameters| filter eventName in [ "StopLogging", "DeleteTrail", "DeleteEventDataStore", "DeleteDetector", "DisableSecurityHub", "StopConfigurationRecorder", "DeleteConfigurationRecorder", "DeleteLogGroup", "DeleteMetricFilter", "DeleteAlarms"]| sort @timestamp descAny confirmed unauthorised tampering should be escalated immediately.
Step 12 — Hunt for Suspicious Region Activity
Section titled “Step 12 — Hunt for Suspicious Region Activity”Attackers may use Regions that are rarely monitored.
Search for:
- API calls in unused Regions
- EC2 instances launched in unusual Regions
- New VPCs
- New access keys used in unexpected Regions
- Security services disabled in specific Regions
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, awsRegion, eventName, eventSource, userIdentity.arn, sourceIPAddress| stats count(*) as EventCount by awsRegion, eventName| sort EventCount descCompare observed Regions with CloudNova’s approved Region list.
Approved Region Register
Section titled “Approved Region Register”| Region | Business Use | Approved |
|---|---|---|
Step 13 — Hunt for Suspicious EC2 Activity
Section titled “Step 13 — Hunt for Suspicious EC2 Activity”Search for:
RunInstancesModifyInstanceAttributeAssociateIamInstanceProfileCreateImageCreateSnapshotAuthorizeSecurityGroupIngressModifyNetworkInterfaceAttribute- Instances launched from unusual AMIs
- Instances with public IP addresses
- Unusual CPU or network utilisation
AWS CLI
Section titled “AWS CLI”aws ec2 describe-instancesaws ec2 describe-security-groupsLook for:
- Ports 22 or 3389 open to the Internet
- Database ports open publicly
- New administrative instance profiles
- Cryptocurrency-mining behaviour
- Internal scanning activity
- Unexpected outbound communication
Step 14 — Hunt for Network Reconnaissance
Section titled “Step 14 — Hunt for Network Reconnaissance”Use VPC Flow Logs to identify:
- Port scanning
- Repeated rejected connections
- Internal subnet scanning
- Connections to many destination ports
- Connections to many internal IP addresses
- Unusual east-west traffic
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, action| filter action = "REJECT"| stats count(*) as Attempts by srcAddr, dstAddr, dstPort| sort Attempts descSearch for sources contacting multiple ports:
fields srcAddr, dstAddr, dstPort| stats count_distinct(dstPort) as UniquePorts by srcAddr, dstAddr| filter UniquePorts > 20| sort UniquePorts descRecord suspicious sources for further investigation.
Step 15 — Hunt for Suspicious Outbound Connections
Section titled “Step 15 — Hunt for Suspicious Outbound Connections”Search for:
- Connections to known malicious IP addresses
- Unusual destination countries
- High-volume outbound traffic
- Traffic to uncommon ports
- Repeated beacon-like communication
- Outbound traffic from sensitive subnets
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, packets, bytes, action| filter action = "ACCEPT"| sort bytes desc| limit 100Correlate suspicious source addresses with:
- EC2 instance IDs
- Security groups
- IAM instance profiles
- GuardDuty findings
- Inspector findings
Step 16 — Hunt for S3 Data Discovery and Collection
Section titled “Step 16 — Hunt for S3 Data Discovery and Collection”Search for:
ListAllMyBucketsListBucketsListObjectsGetBucketPolicyGetBucketAclGetObjectGetObjectAttributesSelectObjectContent
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, eventName, userIdentity.arn, sourceIPAddress, requestParameters.bucketName, requestParameters.key| filter eventSource = "s3.amazonaws.com"| filter eventName in [ "ListAllMyBuckets", "ListBuckets", "ListObjects", "GetBucketPolicy", "GetBucketAcl", "GetObject", "SelectObjectContent"]| sort @timestamp descLook for:
- Large numbers of
GetObjectcalls - Access to sensitive prefixes
- Access from unusual identities
- Access from new IP addresses
- Access outside normal business hours
Step 17 — Hunt for S3 Security-Control Changes
Section titled “Step 17 — Hunt for S3 Security-Control Changes”Search for:
PutBucketPolicyDeleteBucketPolicyPutBucketAclPutPublicAccessBlockDeletePublicAccessBlockPutBucketEncryptionDeleteBucketEncryptionPutBucketLogging
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, eventName, userIdentity.arn, sourceIPAddress, requestParameters.bucketName| filter eventName in [ "PutBucketPolicy", "DeleteBucketPolicy", "PutBucketAcl", "DeletePublicAccessBlock", "PutBucketEncryption", "DeleteBucketEncryption"]| sort @timestamp descReview whether any bucket became publicly or externally accessible.
Step 18 — Hunt for Secrets and KMS Activity
Section titled “Step 18 — Hunt for Secrets and KMS Activity”Search for:
GetSecretValuePutSecretValueUpdateSecretDeleteSecretDecryptCreateGrantPutKeyPolicyDisableKeyScheduleKeyDeletion
CloudWatch Logs Insights
Section titled “CloudWatch Logs Insights”fields @timestamp, eventName, eventSource, userIdentity.arn, sourceIPAddress, requestParameters| filter eventName in [ "GetSecretValue", "PutSecretValue", "UpdateSecret", "DeleteSecret", "Decrypt", "CreateGrant", "PutKeyPolicy", "DisableKey", "ScheduleKeyDeletion"]| sort @timestamp descPrioritise:
- Bulk secret access
- Secret access by new identities
- KMS decrypt activity from unusual Regions
- Key-policy modifications
- Scheduled key deletion
Step 19 — Hunt for Lambda and Serverless Persistence
Section titled “Step 19 — Hunt for Lambda and Serverless Persistence”Search for:
CreateFunctionUpdateFunctionCodeUpdateFunctionConfigurationAddPermissionCreateEventSourceMappingPutFunctionEventInvokeConfig
Review:
- Function code source
- Execution role
- Environment variables
- Triggers
- External network access
- Recent modifications
AWS CLI
Section titled “AWS CLI”aws lambda list-functionsaws lambda get-function \ --function-name YOUR_FUNCTION_NAMEIdentify functions created or modified outside approved deployment pipelines.
Step 20 — Hunt for EventBridge Scheduled Persistence
Section titled “Step 20 — Hunt for EventBridge Scheduled Persistence”Search for:
PutRulePutTargetsEnableRuleCreateApiDestinationCreateConnection
AWS CLI
Section titled “AWS CLI”aws events list-rulesaws events list-targets-by-rule \ --rule YOUR_RULE_NAMELook for:
- Unusual schedules
- Lambda targets
- Step Functions targets
- API destinations
- Rules created by unexpected identities
- Rules invoking administrative automation
Step 21 — Review GuardDuty and Security Hub for Supporting Evidence
Section titled “Step 21 — Review GuardDuty and Security Hub for Supporting Evidence”Review:
- High and critical findings
- Findings previously archived
- Repeated findings
- Findings affecting the same identity
- Findings affecting the same EC2 instance
- Findings associated with the same source IP
AWS CLI
Section titled “AWS CLI”aws guardduty list-findings \ --detector-id YOUR_DETECTOR_IDaws securityhub get-findingsDo not limit the hunt to active alerts.
Archived or previously resolved findings may reveal a longer attack pattern.
Step 22 — Investigate Entities with Amazon Detective
Section titled “Step 22 — Investigate Entities with Amazon Detective”Use Amazon Detective to review:
- IAM identities
- EC2 instances
- Kubernetes entities, where applicable
- Source IP addresses
- API-call volume
- Network connections
- Associated findings
- Historical behaviour
Determine whether:
- Behaviour is new
- Activity exceeds the normal baseline
- Multiple entities share the same infrastructure
- Suspicious activity began before the initial alert
- The entity accessed unusual services
Step 23 — Query Amazon Security Lake with Athena
Section titled “Step 23 — Query Amazon Security Lake with Athena”Use Security Lake to perform cross-source correlation.
Example investigation workflow:
Source IP Address
↓
CloudTrail Identity Activity
↓
VPC Network Connections
↓
GuardDuty Finding
↓
Affected Resource
↓
S3 or Secrets AccessExample query structure:
SELECT time, account_uid, region, actor_user_name, src_endpoint_ip, api_operation, api_service_nameFROM security_lake_tableWHERE src_endpoint_ip = 'SUSPICIOUS_IP'ORDER BY time ASC;Adapt table and field names to the Security Lake schema available in the environment.
Record:
- Query name
- Query execution ID
- Data source
- Date range
- Results
- Evidence location
Step 24 — Query AWS CloudTrail Lake
Section titled “Step 24 — Query AWS CloudTrail Lake”Create queries for suspicious IAM behaviour.
Example:
SELECT eventTime, eventName, eventSource, awsRegion, sourceIPAddress, userIdentity.arnFROM YOUR_EVENT_DATA_STORE_IDWHERE eventName IN ( 'CreateAccessKey', 'AttachUserPolicy', 'AttachRolePolicy', 'UpdateAssumeRolePolicy', 'StopLogging' )ORDER BY eventTime ASC;Search by suspicious IP:
SELECT eventTime, eventName, eventSource, awsRegion, userIdentity.arn, sourceIPAddressFROM YOUR_EVENT_DATA_STORE_IDWHERE sourceIPAddress = 'SUSPICIOUS_IP'ORDER BY eventTime ASC;Step 25 — Correlate Related Activity
Section titled “Step 25 — Correlate Related Activity”Build an entity-correlation matrix.
| Entity | Related Identity | Related IP | Related Resource | Related Finding |
|---|---|---|---|---|
Correlate using:
- Access key ID
- IAM ARN
- Source IP address
- User agent
- Role session name
- AWS account
- Region
- Resource ID
- Event time
Identify whether apparently separate alerts form part of the same attack sequence.
Step 26 — Map Findings to MITRE ATT&CK
Section titled “Step 26 — Map Findings to MITRE ATT&CK”Map confirmed and suspected behaviour.
| ATT&CK Tactic | Technique or Behaviour | AWS Evidence |
|---|---|---|
| Initial Access | Valid account abuse | ConsoleLogin, access-key use |
| Persistence | New access key or Lambda function | CreateAccessKey, CreateFunction |
| Privilege Escalation | Administrative policy attachment | AttachUserPolicy |
| Defence Evasion | CloudTrail disabled | StopLogging |
| Discovery | AWS resource enumeration | ListBuckets, DescribeInstances |
| Credential Access | Secrets retrieval | GetSecretValue |
| Collection | S3 object access | GetObject |
| Command and Control | Suspicious outbound traffic | VPC Flow Logs |
| Exfiltration | Large outbound transfer | Flow Logs, S3 events |
| Impact | Security-control modification | Security-group or policy changes |
Step 27 — Document Threat-Hunting Findings
Section titled “Step 27 — Document Threat-Hunting Findings”Classify each hunt result.
- Confirmed Malicious
- Suspicious
- Policy Violation
- Misconfiguration
- Benign Administrative Activity
- False Positive
- Inconclusive
Threat-Hunting Findings Register
Section titled “Threat-Hunting Findings Register”| Finding ID | Description | Evidence | Classification | Severity |
|---|---|---|---|---|
| TH-001 | ||||
| TH-002 | ||||
| TH-003 |
For every finding, document:
- Detection time
- Event time
- Affected identity
- Affected account
- Affected Region
- Affected resource
- Business impact
- Recommended action
Step 28 — Identify Detection Gaps
Section titled “Step 28 — Identify Detection Gaps”Review why each suspicious activity was or was not detected.
Questions:
- Was the required telemetry available?
- Was the event logged?
- Did an existing rule monitor the event?
- Did the alert severity reflect business risk?
- Was the alert routed to the correct team?
- Did the analyst have sufficient context?
- Did the detection produce excessive false positives?
- Was the response playbook complete?
Detection Gap Register
Section titled “Detection Gap Register”| Gap ID | Behaviour | Existing Detection | Gap | Recommendation |
|---|---|---|---|---|
| DG-001 | ||||
| DG-002 | ||||
| DG-003 |
Step 29 — Design New Detection Use Cases
Section titled “Step 29 — Design New Detection Use Cases”Create detection use cases for identified gaps.
Detection Use Case 1 — Access Key Creation Followed by Sensitive Activity
Section titled “Detection Use Case 1 — Access Key Creation Followed by Sensitive Activity”Trigger when:
- A new access key is created
- The key is used within a short period
- Sensitive API activity follows
- The source IP is new or unusual
Detection Use Case 2 — Administrative Policy Attachment
Section titled “Detection Use Case 2 — Administrative Policy Attachment”Trigger when:
AdministratorAccessis attached- The change occurs outside approved automation
- The affected identity is not in the approved administrator group
Detection Use Case 3 — Role Trust Policy Modified
Section titled “Detection Use Case 3 — Role Trust Policy Modified”Trigger when:
UpdateAssumeRolePolicyoccurs- A new external account is trusted
- A wildcard principal is introduced
Detection Use Case 4 — Security Service Disabled
Section titled “Detection Use Case 4 — Security Service Disabled”Trigger when:
- CloudTrail, GuardDuty, Security Hub, Config, or logging is disabled
Detection Use Case 5 — Unusual Region Activity
Section titled “Detection Use Case 5 — Unusual Region Activity”Trigger when:
- Activity occurs in an unapproved AWS Region
- EC2 resources are launched
- Access keys are used
- IAM or logging services are modified
Step 30 — Create EventBridge Detection Rules
Section titled “Step 30 — Create EventBridge Detection Rules”Example detection targets:
- CloudTrail logging stopped
- IAM administrative policy attached
- IAM role trust policy changed
- Security group opened publicly
- S3 public-access protection removed
- GuardDuty high-severity finding
- Security Hub critical finding
Example event pattern:
{ "source": [ "aws.iam" ], "detail-type": [ "AWS API Call via CloudTrail" ], "detail": { "eventSource": [ "iam.amazonaws.com" ], "eventName": [ "AttachUserPolicy", "AttachRolePolicy", "PutUserPolicy", "PutRolePolicy", "UpdateAssumeRolePolicy" ] }}AWS CLI
Section titled “AWS CLI”aws events put-rule \ --name CN-SOC-IAM-Privilege-Changes \ --event-pattern file://iam-privilege-event-pattern.jsonStep 31 — Create CloudWatch Metric Filters
Section titled “Step 31 — Create CloudWatch Metric Filters”Create metric filters for:
- Root-account usage
- Console login without MFA
- Administrative policy attachment
- Access-key creation
- CloudTrail changes
- Security-group changes
- S3 public-access changes
- KMS key changes
Example filter:
{ ($.eventName = "StopLogging") || ($.eventName = "DeleteTrail") }Example AWS CLI:
aws logs put-metric-filter \ --log-group-name CloudTrail-Logs \ --filter-name CloudTrail-Tampering \ --filter-pattern '{ ($.eventName = "StopLogging") || ($.eventName = "DeleteTrail") }' \ --metric-transformations \ metricName=CloudTrailTamperingCount,metricNamespace=CloudNova/SOC,metricValue=1Step 32 — Define Detection Severity
Section titled “Step 32 — Define Detection Severity”Assign detection severity based on:
- Identity privilege
- Resource criticality
- Data sensitivity
- Confidence
- Number of affected accounts
- Persistence
- Defence evasion
- Business impact
Detection Severity Matrix
Section titled “Detection Severity Matrix”| Detection | Default Severity | Escalation Condition |
|---|---|---|
| Root login | High | Critical if no MFA |
| Access key created | Medium | High if followed by privileged activity |
| Admin policy attached | High | Critical if unauthorised |
| CloudTrail stopped | Critical | Immediate incident |
| Security group opened publicly | High | Critical for sensitive workload |
| S3 public access enabled | High | Critical for regulated data |
| Unusual Region activity | Medium | High if resources are launched |
Step 33 — Develop Detection Response Playbooks
Section titled “Step 33 — Develop Detection Response Playbooks”For each new detection, define:
- Alert title
- Alert description
- Severity
- Required data
- Triage questions
- Validation steps
- Escalation criteria
- Containment options
- False-positive conditions
- Closure criteria
Detection Playbook Template
Section titled “Detection Playbook Template”| Field | Analyst Entry |
|---|---|
| Detection Name | |
| Alert Source | |
| Severity | |
| Required Evidence | |
| Triage Questions | |
| Escalation Criteria | |
| Containment Actions | |
| False-Positive Conditions | |
| Closure Criteria |
Step 34 — Test New Detections Safely
Section titled “Step 34 — Test New Detections Safely”Use a dedicated test account or sandbox.
Approved test activities may include:
- Create a temporary IAM user
- Create a test access key
- Attach a controlled test policy
- Modify a test security group
- Generate a GuardDuty sample finding
- Create a temporary EventBridge rule
- Generate a test CloudTrail event
Do not:
- Expose production workloads
- Use real malware
- Disable production logging
- Make sensitive data public
- Test destructive automation in production
Record:
| Test ID | Detection | Test Action | Expected Result | Actual Result |
|---|---|---|---|---|
| T-001 | ||||
| T-002 |
Step 35 — Tune Detection Logic
Section titled “Step 35 — Tune Detection Logic”Review:
- False positives
- Duplicate alerts
- Missing context
- Alert volume
- Detection delay
- Severity
- Routing
- Suppression conditions
- Approved automation
Tune using:
- Known service accounts
- Approved IP addresses
- Approved Regions
- Change windows
- Resource tags
- Identity groups
- Business-unit context
- Asset criticality
Do not suppress security activity only to reduce alert volume.
Every suppression must have:
- Business justification
- Risk approval
- Expiry date
- Documented owner
Step 36 — Measure Detection Effectiveness
Section titled “Step 36 — Measure Detection Effectiveness”Measure:
- Detection coverage
- Detection latency
- True-positive rate
- False-positive rate
- Mean Time to Detect
- Mean Time to Triage
- Mean Time to Escalate
- Number of detection gaps
- Number of new detections created
- Number of outdated detections retired
- Percentage of detections with playbooks
Detection Metrics
Section titled “Detection Metrics”| Metric | Current Value | Target |
|---|---|---|
| Detection Coverage | ||
| Mean Time to Detect | ||
| False-Positive Rate | ||
| Detections with Playbooks | ||
| Critical Techniques Covered | ||
| Multi-Region Coverage |
Step 37 — Create the Threat-Hunting Report
Section titled “Step 37 — Create the Threat-Hunting Report”The technical report should include:
- Executive Summary
- Hunt Objective
- Scope
- Hypotheses
- Data Sources
- Queries Used
- Findings
- Evidence
- ATT&CK Mapping
- Detection Gaps
- New Detection Use Cases
- Detection Testing
- Detection Tuning
- Risk Assessment
- Recommended Actions
- Continuous Improvement Plan
Step 38 — Create the Detection Engineering Backlog
Section titled “Step 38 — Create the Detection Engineering Backlog”Prioritise new detections.
| Priority | Detection Use Case | Business Risk | Owner | Target Date |
|---|---|---|---|---|
| P1 | CloudTrail disabled | Critical | Detection Engineering | |
| P1 | Administrator policy attached | Critical | Detection Engineering | |
| P2 | New access key followed by API activity | High | SOC Engineering | |
| P2 | Role trust-policy modification | High | IAM Security | |
| P3 | Unusual Region activity | Medium | Cloud Security |
Step 39 — Develop the Continuous Improvement Roadmap
Section titled “Step 39 — Develop the Continuous Improvement Roadmap”Immediate — Within 24 Hours
Section titled “Immediate — Within 24 Hours”- Escalate confirmed malicious findings
- Restore missing telemetry
- Enable monitoring in uncovered Regions
- Implement detection for logging tampering
- Review high-risk identities
- Disable unauthorised persistence mechanisms
Short-Term — Within 30 Days
Section titled “Short-Term — Within 30 Days”- Implement new identity-abuse detections
- Improve role-trust monitoring
- Expand S3 data-event logging
- Improve alert enrichment
- Develop missing playbooks
- Tune high-volume detections
- Establish weekly threat-hunting sessions
Medium-Term — Within 90 Days
Section titled “Medium-Term — Within 90 Days”- Expand Amazon Security Lake coverage
- Implement detection-as-code
- Create automated detection testing
- Integrate threat intelligence
- Map detections to MITRE ATT&CK
- Establish purple-team exercises
- Create a formal detection lifecycle
Long-Term — Within 180 Days
Section titled “Long-Term — Within 180 Days”- Implement user and entity behaviour analytics
- Establish 24×7 threat-hunting capability
- Integrate cloud and endpoint telemetry
- Build multi-cloud detection engineering
- Measure detection maturity
- Conduct regular adversary-emulation exercises
Step 40 — Present Findings to SOC Leadership
Section titled “Step 40 — Present Findings to SOC Leadership”Present:
- Threats identified
- Confirmed security incidents
- Visibility gaps
- Detection weaknesses
- High-priority risks
- New detections created
- Required investment
- Staffing and training needs
- Continuous-improvement roadmap
Use business-focused language.
Explain:
- What risk exists
- Why it matters
- What must be fixed
- Who owns the action
- When it should be completed
🧪 Enterprise Challenge
Section titled “🧪 Enterprise Challenge”CloudNova Technologies’ SOC has received no critical alerts during the previous seven days.
However, the threat-hunting team identifies several suspicious activities:
- An IAM access key was created at 01:20 UTC
- The key was used from an unfamiliar IP address
- The identity enumerated IAM roles and S3 buckets
AdministratorAccesswas attached temporarily- A role trust policy was modified
- A Lambda function was created in an unused Region
- An EventBridge rule invoked the Lambda function every hour
- Several Secrets Manager secrets were accessed
- CloudTrail logging was stopped for 12 minutes
- An EC2 instance communicated with an unusual external IP
- Sensitive S3 objects were downloaded
- The administrative policy was detached before the working day began
No single security service generated a complete incident alert.
As the Tier-3 SOC Analyst, you must:
- Validate the threat-hunting hypothesis
- Identify the affected identity
- Correlate activity across all Regions
- Identify privilege escalation
- Identify persistence
- Identify defence evasion
- Determine whether data was accessed or exfiltrated
- Determine the incident scope
- Preserve evidence
- Escalate the activity as a security incident
- Create new detection logic
- Test the new detections
- Document detection gaps
- Present an improvement plan to SOC leadership
📄 Deliverables
Section titled “📄 Deliverables”Capture evidence of:
- Threat-hunting scope
- CloudTrail queries
- CloudTrail Lake queries
- CloudWatch Logs Insights queries
- Security Lake or Athena queries
- IAM identity activity
- Access-key creation
- Privilege-escalation events
- Role-trust changes
- Persistence mechanisms
- Security-control tampering
- EC2 and network activity
- S3 data access
- GuardDuty and Security Hub findings
- Detection rules
- EventBridge rules
- CloudWatch metric filters
- Detection testing
Submit:
- Threat-Hunting Plan
- Threat-Hunting Hypothesis Register
- Security Telemetry Coverage Matrix
- Threat-Hunting Query Catalogue
- Threat-Hunting Findings Register
- Entity Correlation Matrix
- Incident Timeline
- MITRE ATT&CK Mapping
- Detection Gap Register
- Detection Use-Case Catalogue
- Detection Response Playbooks
- Detection Testing Report
- Detection Engineering Backlog
- Continuous Improvement Roadmap
- Executive Threat-Hunting Report
🧹 Cleanup
Section titled “🧹 Cleanup”Delete only temporary test resources:
- Test IAM users
- Test access keys
- Test IAM policies
- Test EventBridge rules
- Test Lambda functions
- Test CloudWatch alarms
- Test metric filters
- Test EC2 instances
- Temporary test security groups
- Temporary Athena query results
Retain:
- CloudTrail logs
- Security Lake data
- VPC Flow Logs
- GuardDuty findings
- Security Hub findings
- Threat-hunting queries
- Detection rules
- Detection playbooks
- Findings register
- Detection gap register
- Threat-hunting report
- Continuous-improvement roadmap
Confirm that all temporary test credentials are disabled or deleted.
Review AWS Cost Explorer and remove unnecessary resources.
✅ Runbook Checklist
Section titled “✅ Runbook Checklist”- Threat-Hunting Scope Defined
- Threat-Hunting Hypotheses Created
- Data Availability Confirmed
- High-Risk Identities Identified
- Console Login Activity Reviewed
- New Access Keys Investigated
- Privilege-Escalation Activity Reviewed
- Role-Trust Changes Reviewed
- AssumeRole Activity Reviewed
- Persistence Mechanisms Investigated
- Security-Control Tampering Investigated
- Unusual Region Activity Reviewed
- EC2 Activity Reviewed
- Network Reconnaissance Investigated
- Suspicious Outbound Traffic Reviewed
- S3 Data Access Investigated
- S3 Security Changes Reviewed
- Secrets and KMS Activity Reviewed
- Lambda Persistence Reviewed
- EventBridge Persistence Reviewed
- GuardDuty Findings Correlated
- Security Hub Findings Correlated
- Amazon Detective Investigation Completed
- Security Lake Queries Completed
- CloudTrail Lake Queries Completed
- Entity Correlation Completed
- MITRE ATT&CK Mapping Completed
- Threat-Hunting Findings Documented
- Detection Gaps Identified
- New Detection Use Cases Designed
- EventBridge Rules Created
- CloudWatch Metric Filters Created
- Detection Severity Defined
- Detection Playbooks Created
- Detection Testing Completed
- Detection Logic Tuned
- Detection Effectiveness Measured
- Threat-Hunting Report Completed
- Detection Engineering Backlog Created
- Continuous Improvement Roadmap Completed
- Findings Presented to SOC Leadership
💡 Lessons Learned
Section titled “💡 Lessons Learned”How is proactive threat hunting different from alert-driven SOC monitoring?
Section titled “How is proactive threat hunting different from alert-driven SOC monitoring?”Which threat-hunting hypothesis produced the most valuable findings?
Section titled “Which threat-hunting hypothesis produced the most valuable findings?”Which attacker activities were not detected by the existing SOC controls?
Section titled “Which attacker activities were not detected by the existing SOC controls?”How did CloudTrail, Security Lake, VPC Flow Logs, GuardDuty, Security Hub, and Detective complement one another?
Section titled “How did CloudTrail, Security Lake, VPC Flow Logs, GuardDuty, Security Hub, and Detective complement one another?”Which indicators provided the strongest evidence of identity compromise?
Section titled “Which indicators provided the strongest evidence of identity compromise?”Which actions represented privilege escalation, persistence, defence evasion, discovery, and collection?
Section titled “Which actions represented privilege escalation, persistence, defence evasion, discovery, and collection?”What telemetry gaps limited the investigation?
Section titled “What telemetry gaps limited the investigation?”Which new detection should CloudNova Technologies implement first, and why?
Section titled “Which new detection should CloudNova Technologies implement first, and why?”How can detection-as-code improve the quality and consistency of SOC monitoring?
Section titled “How can detection-as-code improve the quality and consistency of SOC monitoring?”How should threat hunting support CloudNova Technologies’ wider incident-response programme?
Section titled “How should threat hunting support CloudNova Technologies’ wider incident-response programme?”🚀 Next Module
Section titled “🚀 Next Module”**14 — Enterprise Cloud Security Projects