Responding to security events

Searching logs: CloudTrail Lake, Athena and Logs Insights

When an alert fires, the first question is always “what exactly happened?”. Skill 2.2.2 asks you to search and correlate logs across services. In practice that means asking the same questions of CloudTrail (who called which API), VPC Flow Logs (which IPs talked), application logs and service logs, and lining the answers up by time, user and IP address.

AWS gives you three main query tools. Which one is right depends mostly on where the logs already live. The exam likes the cheapest tool that works without moving data.

CloudTrail Lake and CloudWatch Logs Insights

CloudTrail Lake stores events in event data stores that you query with SQL (Trino SELECT statements), and it has a query generator that writes SQL from an English prompt. It can also hold AWS Config items and events from outside AWS. Important for 2026: CloudTrail Lake is no longer open to new customers from 31 May 2026; existing customers keep using it, and AWS recommends migrating to Amazon CloudWatch (organization-wide CloudTrail ingestion, OCSF pipelines, S3 Tables for Athena). CloudWatch Logs Insights queries log groups with the Logs Insights query language, OpenSearch PPL or OpenSearch SQL. Queries time out after 60 minutes, results stay available for 7 days, and you pay per GB of data scanned. It is the natural choice for application logs, Lambda logs, VPC Flow Logs sent to CloudWatch, and a trail delivered to CloudWatch Logs.

-- Athena (table created from the CloudTrail console)
SELECT eventtime, eventsource, eventname, sourceipaddress, errorcode
FROM cloudtrail_logs_org_trail
WHERE useridentity.accesskeyid = 'AKIAIOSFODNN7EXAMPLE'
  AND timestamp >= '2026/10/08'
ORDER BY eventtime;

# CloudWatch Logs Insights (log group: aws-cloudtrail-logs-org)
fields @timestamp, eventName, sourceIPAddress, errorCode
| filter userIdentity.accessKeyId = "AKIAIOSFODNN7EXAMPLE"
| stats count(*) as calls by eventName, sourceIPAddress
| sort calls desc

Correlating across sources: Security Lake and OpenSearch

A real attack crosses many log types: a phishing login (identity logs), API calls (CloudTrail), DNS lookups (Route 53 Resolver), network traffic (VPC Flow Logs) and maybe a firewall outside AWS. Each tool writes its own format. To correlate them you want one schema and one place to query.

Amazon OpenSearch Service for investigations

Amazon OpenSearch Service is a search and dashboard engine. Security teams use it as a log analytics platform: near real-time search, dashboards, alerting, and the Security Analytics plugin with detection rules. It can ingest logs (for example through OpenSearch Ingestion) or query data in place, including Security Lake data, through direct query integrations. Choose it when the question says interactive dashboards, full-text search or near real-time alerting over large log volumes. Choose Athena for occasional SQL over S3 at the lowest cost.

SELECT time_dt, api.service.name, api.operation,
       actor.user.uid, src_endpoint.ip
FROM amazon_security_lake_glue_db_us_east_1
     .amazon_security_lake_table_us_east_1_cloud_trail_mgmt_2_0
WHERE src_endpoint.ip = '203.0.113.9'
  AND time_dt > TIMESTAMP '2026-10-08 00:00:00'
ORDER BY time_dt;

Validating findings: scope and impact

A finding is a claim, not a proof. Skill 2.2.3 asks you to validate findings from AWS security services and work out scope (which accounts, identities and resources are involved) and impact (what data, money or availability was affected). Only then do you choose how hard to contain. A false positive treated as a real attack can cause an outage; a real attack treated as noise can cause a breach.

IAM Access Analyzer and Macie help measure scope

IAM Access Analyzer answers “who could reach this?”. External access findings show buckets, roles, keys and other resources shared outside your account or organization; unused access findings show idle roles, keys and permissions; and policy generation can build a least-privilege policy from CloudTrail activity after the incident. The new AWS Security Hub correlates findings from GuardDuty, Inspector, Macie and CSPM checks into exposure findings with an attack path graph, which helps you see how far an attacker could go. Amazon Macie answers “was the data sensitive?”: run a sensitive data discovery job on the buckets the attacker read to see if they held personal or financial data.

$ aws guardduty get-findings --detector-id 12abc34d567e8fa901bc2d34e56789f0 \
    --finding-ids 6eb2d7b1example --query "Findings[0]"
{ "Type": "UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS",
  "Severity": 8,
  "Resource": { "AccessKeyDetails": {
      "AccessKeyId": "ASIAEXAMPLE7Q", "UserName": "web-app-role",
      "UserType": "AssumedRole" } },
  "Service": { "Count": 14, "Archived": false,
      "EventFirstSeen": "2026-10-09T02:11:40Z",
      "EventLastSeen": "2026-10-09T03:58:02Z",
      "Action": { "AwsApiCallAction": { "Api": "ListBuckets",
          "RemoteIpDetails": { "IpAddressV4": "198.51.100.23" } } } } }

Containing compromised IAM credentials

Stolen credentials are the most common AWS incident. Skill 2.2.4 asks you to contain, eradicate and recover. For credentials the order is: stop the credentials working, then remove anything the attacker created, then give the real owner new, safer credentials. The exam tests one detail again and again: long-term keys and temporary sessions are stopped in different ways.

The full credential runbook

1. Identify the entity from the finding (user type, user name, access key ID) and the API calls. 2. Contain: set the IAM user’s key to Inactive; for a role, revoke active sessions; for a console user, also reset the password and attach a deny-all policy if needed. 3. Eradicate: search CloudTrail for everything that identity did, then delete attacker-created users, keys, roles, EC2 instances and Lambda functions; delete the old key; issue a new one (or better, move the workload to a role). 4. Recover and learn: check the billing for crypto mining, use Access Analyzer’s unused access and Access Advisor to tighten permissions, and add Secrets Manager or roles so long-term keys are not needed. If AWS detects your key in public, it may attach the AWSCompromisedKeyQuarantineV3 managed policy to the user; do not simply remove it, finish the runbook first.

$ aws iam update-access-key --user-name ci-deployer \
    --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive

$ cat revoke.json
{ "Version": "2012-10-17",
  "Statement": [{ "Effect": "Deny", "Action": "*", "Resource": "*",
    "Condition": { "DateLessThan":
      { "aws:TokenIssueTime": "2026-10-10T09:30:00Z" } } }] }

$ aws iam put-role-policy --role-name web-app-role \
    --policy-name AWSRevokeOlderSessions --policy-document file://revoke.json

Containing a compromised EC2 instance and the network

For a compromised instance the goal is to stop the damage but keep the evidence. Do not terminate first. Take it out of service, cut its network, capture its memory and disks, and only then decide whether to analyze or rebuild. The exam’s favorite trap is connection tracking: changing security groups does not end sessions that are already open.

Auto Scaling, load balancers and evidence

Putting an instance into Standby deregisters it from attached target groups (after connection draining) and stops Auto Scaling health checks on it. If you do not decrement desired capacity, Auto Scaling launches a clean replacement, so the service stays up. You can also detach the instance from the group, or suspend the health check and replace processes. Then capture volatile data first (memory, process list, network connections) with a tool run through Systems Manager, take EBS snapshots of every volume, tag everything with the case ID, and copy the snapshots to the forensic account (share with a KMS key the forensic account can use). This is the work the Day 2 automated forensics tools do for you.

$ aws ec2 modify-instance-attribute --instance-id i-0abc12345def67890 \
    --disable-api-termination
$ aws autoscaling enter-standby --auto-scaling-group-name web-asg \
    --instance-ids i-0abc12345def67890 --no-should-decrement-desired-capacity
$ aws ec2 modify-instance-attribute --instance-id i-0abc12345def67890 \
    --groups sg-0isolation11111111
$ aws ec2 create-network-acl-entry --network-acl-id acl-0aa11bb22 \
    --ingress --rule-number 50 --protocol -1 --rule-action deny \
    --cidr-block 198.51.100.23/32
$ aws ec2 create-snapshot --volume-id vol-0123456789abcdef0 \
    --description "IR-2026-042 root volume" \
    --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Case,Value=IR-2026-042}]'

S3, Lambda, containers and recovery

The same contain, eradicate, recover pattern applies to every resource type, but the buttons are different. For S3 you close access; for Lambda you stop invocations; for containers you isolate the pod or task and the node. Then you recover: restore clean data from backups and rebuild compute from known-good images instead of trying to clean a hacked server.

S3, Lambda and containers

S3: turn on Block Public Access (account and bucket level), add a bucket policy that denies the compromised principal or allows only your VPC endpoint, revoke the principal’s credentials, then use versioning to restore overwritten or deleted objects (Object Lock prevents deletion if it was set up earlier). Use CloudTrail data events or server access logs to see which objects were read, and Macie to judge sensitivity. Lambda: set reserved concurrency to 0 to stop every invocation, remove or narrow triggers, revoke the execution role’s sessions, rotate secrets in its environment, and redeploy a known-good version or alias. Containers: on EKS, isolate the pod with a deny-all network policy, label it for forensics, cordon the node so nothing new is scheduled there, and revoke the pod or node role credentials; on ECS, attach an isolation security group to the task’s ENI. GuardDuty Runtime Monitoring and EKS Protection give the findings; Inspector scans the rebuilt image.

$ aws s3api put-public-access-block --bucket customer-exports \
    --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

$ aws lambda put-function-concurrency --function-name orders-api \
    --reserved-concurrent-executions 0

$ aws backup start-restore-job \
    --recovery-point-arn arn:aws:ec2:eu-west-1::snapshot/snap-0aa11bb22cc33dd44 \
    --iam-role-arn arn:aws:iam::111122223333:role/BackupRestoreRole \
    --resource-type EBS --metadata file://restore-ebs.json

Root cause analysis with Amazon Detective

After the fire is out, you must know how it started so it does not happen again. Skill 2.2.5 asks you to describe methods for root cause analysis, and it names Amazon Detective. Root cause means the first weakness: a leaked key, an unpatched app, an open security group, a too-wide trust policy. You find it by building a timeline backwards from the first finding.

Integrations and the timeline method

From a GuardDuty finding or a Security Hub finding, choose Investigate in Detective to open the right entity profile with the time window already set. In the profile, look at new behavior: first-time API calls, new geolocations, new user agents, unusual data volume. Then build the timeline: initial access (for example the first call from a new IP with the leaked key) → persistence (new user or key) → actions on objectives (data read, instances launched). The earliest step tells you the root cause. Finish with lessons learned: fix the weakness, add a detection, update the runbook and test it (Day 1).

$ aws detective start-investigation \
    --graph-arn arn:aws:detective:eu-west-1:111122223333:graph:abc123 \
    --entity-arn arn:aws:iam::111122223333:role/web-app-role \
    --scope-start-time 2026-10-08T00:00:00Z --scope-end-time 2026-10-10T00:00:00Z
{ "InvestigationId": "000000000000000000001" }

$ aws detective get-investigation --graph-arn arn:aws:detective:eu-west-1:111122223333:graph:abc123 \
    --investigation-id 000000000000000000001 --query "[Status,Severity]"
[ "SUCCESSFUL", "HIGH" ]

Cheatsheet extras: account-compromise checklist and KMS key emergencies

Two cheat-sheet lists are worth knowing by heart: what to do when the whole account may be compromised, and what to do with a KMS key during an incident. One old list needs updating too.

The account-compromise checklist

When AWS Support or Trust & Safety says your account may be compromised, AWS’s own checklist is: change the root user password and turn on MFA for root and console users; rotate and delete all root and IAM access keys; delete IAM users (and roles) you didn’t create and change the passwords of the others; check your bill and every Region for resources you didn’t create (EC2 instances, Spot requests, Lambda functions, snapshots, AMIs, S3 buckets) and delete them after you have kept evidence; check your account contact details; and reply to the notice in the Support Center. With an organization, also check for new accounts or SCP changes the attacker made.

{ "source": ["aws.kms"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": { "eventName": ["ScheduleKeyDeletion", "DisableKey"] } }

$ aws kms describe-key --key-id 1234abcd-12ab-34cd-56ef-1234567890ab \
    --query "KeyMetadata.[KeyState,DeletionDate]"
[ "PendingDeletion", "2026-11-19T10:02:11Z" ]

$ aws kms cancel-key-deletion --key-id 1234abcd-12ab-34cd-56ef-1234567890ab
$ aws kms enable-key --key-id 1234abcd-12ab-34cd-56ef-1234567890ab

Lecture summary map

Day 3 on one screen.

1. Mind map

2. Match the requirement to the tool

3. “If the question says… pick…”

Must-remember cheat sheet

Core quiz

Check the basics first. Pick an answer, then press Check.

Hard quiz

Harder than most real exam questions on purpose, with one “choose two” and one matching question. Aim for 9 out of 12.

Domain 2 review quiz (Days 1–3)

Questions from every Incident Response day. If you miss one, reopen that day’s lecture and its “Must remember” boxes.

Practice test questions (from your mock exams)

8 questions taken from your own SCS-003 mock exams and matched to today’s topics. Each answer was checked against current AWS behaviour; where a mock’s answer was outdated or wrong, the explanation says so. The source is shown at the end of each explanation.

Hands-on lab (30 minutes, low cost)

1. In the CloudTrail console, choose Create Athena table for your trail’s bucket. Run a query that lists your own eventname and sourceipaddress for today (filter on the date to keep the scan tiny).
2. In GuardDuty, choose Generate sample findings. Open a sample UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS finding and note the severity, Count, resource and remote IP. Then create a suppression rule for one sample type and see it move to Archived.
3. Create a test IAM user with an access key. Deactivate the key with update-access-key --status Inactive and confirm a call with it fails. Create a test role, assume it, then press Revoke active sessions and confirm the old session gets AccessDenied while a fresh assume works.
4. Launch a t3.micro (Free Tier eligible) and SSH in. Replace its security group with an empty isolation group and notice the SSH session stays up; then add a NACL deny for your IP and watch it drop. Take an EBS snapshot with a Case tag.
5. Optional: enable Amazon Detective (30-day free trial in a Region) and open a sample finding with Investigate in Detective. Real data appears only after it has collected some activity.
6. Clean up: terminate the instance, delete the snapshot, the NACL entry and isolation group, the test user, key and role, the suppression rule and archived samples. Disable Detective before the trial ends, and drop the Athena table (the query results bucket keeps costing a little).