The last Detection day. In Day 3 you collected logs into one place. Today you use them: query with CloudWatch Logs Insights, Athena and CloudTrail Lake, parse and correlate with Firehose, Lambda, OpenSearch and Managed Grafana, then fix the usual reasons logs go missing. It ends with a review quiz across all four days.
Days 1 to 3 were about collecting logs. Today is about using them: asking questions, joining data from many places, and fixing things when logs go missing. Skill 1.2.4 names three tools for analysis. The first one is CloudWatch Logs Insights, the query screen built into CloudWatch Logs.
Use it when the logs are already in CloudWatch Logs and you want an answer in seconds: which IP failed to sign in most, which Lambda calls were slow, which ports were rejected. You don’t build tables or move data.
Concept profile
CloudWatch Logs Insights
often confused with CloudWatch metric filters
?What is it?
An interactive query engine for log groups in CloudWatch Logs. You pick log groups and a time range, write a query, and get a table or chart back.
⚙How does it work?
Three query languages: Logs Insights QL, OpenSearch PPL and OpenSearch SQL (SQL supports JOINs across log groups).
Common commands: fields, filter, stats, sort, limit, parse, dedup, plus helpers like bin() and isIpv4InSubnet().
JSON logs such as CloudTrail are parsed automatically, so you can use fields like userIdentity.arn.
Billed per GB of log data scanned. Queries time out after 60 minutes; results are kept for 7 days.
In a monitoring account with cross-account observability, one query can cover log groups in many source accounts.
{ }Structure
Pick log groups, run a query, keep or chart the result.
⇄Logs Insights vs CloudWatch metric filters
Logs Insights
Metric filter
Question type
Ad hoc, any time, after the fact
Fixed pattern, counted as it arrives
Output
Table or chart now
A CloudWatch metric (then an alarm)
Past data
Yes, any retained logs
No, only new events after creation
Cost
Per GB scanned per query
Metric charges
Exam hint
“investigate”, “find which IP”
“alert whenever”, “alarm on”
!Common mistakes the exam tests
Choosing Logs Insights for continuous alerting: alerts come from metric filters and alarms (Day 1), or from EventBridge.
Querying 30 days across every log group when one hour of one group would do: cost grows with data scanned.
Expecting to query logs that live only in S3: that’s Athena.
Forgetting that a metric filter only counts events that arrive after you create it, while Logs Insights can look back.
Thinking you must copy logs to a central account to query them: cross-account observability lets a monitoring account query source accounts.
✓Benefits and use cases
Incident triage: “what did this access key do in the last hour?”
Top talkers or rejected flows in VPC flow logs.
Slow or failing Lambda invocations; dashboards from saved queries.
Queries worth knowing by heart
The exam won’t ask you to write syntax from memory, but it will show a query or describe one. Know that filter narrows events, stats … by groups and counts, bin(1h) makes time buckets, and parse pulls fields out of plain text lines (for example application logs). For VPC flow logs, fields like srcAddr, dstPort and action are discovered automatically in the default format.
Think of it like · a school’s office records
Finding answers in a big pile of attendance sheets.
AWS concept
Familiar thing
Log group
≈
One filing cabinet of attendance sheets
Logs Insights query
≈
Asking the secretary: ‘which students were late most this week?’
filter / stats
≈
Pull only the late slips, then count them per student
Metric filter
≈
A tally counter at the door that clicks every time a late student walks in
Data scanned
≈
How many sheets the secretary had to read, which is what you pay for
Monitoring account
≈
The head office that can search every school’s cabinets
Real AWS example · two investigation queries
First: failed console sign-ins per source IP from a CloudTrail log group. Second: the most rejected ports in a VPC flow log group.
fields @timestamp, userIdentity.arn, sourceIPAddress
| filter eventName = "ConsoleLogin" and responseElements.ConsoleLogin = "Failure"
| stats count(*) as fails by sourceIPAddress
| sort fails desc
| limit 10
filter action = "REJECT"
| stats count(*) as rejects by srcAddr, dstPort
| sort rejects desc
| limit 20
A quick win: filter isIpv4InSubnet(srcAddr, "10.0.0.0/8") keeps only internal sources.
Exam-style scenario
A security engineer gets a GuardDuty finding about an access key. CloudTrail already sends events to a CloudWatch Logs log group. The engineer needs, within minutes, a list of every API call that key made in the last 2 hours, grouped by event name. What is the fastest approach?
A. Create a metric filter for the access key and wait for the metric.
B. Run a CloudWatch Logs Insights query on the CloudTrail log group: filter on userIdentity.accessKeyId, stats count(*) by eventName, time range 2 hours.
C. Create an Athena table over the trail’s S3 bucket and add partitions.
D. Export the log group to S3 and download it.
Show the worked answer
Answer: B
The data is already in CloudWatch Logs; Logs Insights answers ad hoc questions over past events immediately.
Why not the others
A: A metric filter only counts new events and gives a number, not a list.
C: Works, but takes longer to set up than a query on data already in CloudWatch.
D: Slow and manual.
How the exam tests this
“Investigate logs already in CloudWatch Logs”→ Logs Insights.
“Alert every time a pattern appears”→ Metric filter + alarm (not Insights).
“Query many accounts’ log groups at once”→ Monitoring account (cross-account observability).
“JOIN two log groups in one query”→ Logs Insights with OpenSearch SQL.
“Reduce query cost”→ Shorter time range, fewer log groups (less data scanned).
Long-term logs usually live in S3: the organization trail, VPC flow logs, ALB and WAF logs. Keeping them in CloudWatch Logs for years is expensive. Amazon Athena runs SQL directly on those S3 files, with no servers and no loading step. CloudTrail Lake is a different option: CloudTrail keeps the events in its own managed store and you query them with SQL.
Concept profile
Amazon Athena with partition projection
often confused with CloudTrail Lake
?What is it?
Serverless SQL over files in S3. You create an external table that describes the log format and where the files are; Athena reads only what the query needs.
⚙How does it work?
Create the table from the CloudTrail console (simplest) or with CREATE EXTERNAL TABLE in Athena.
Logs are stored in folders by account, Region and date. Partition projection tells Athena how to calculate those folders from table properties (projection.enabled, storage.location.template).
With projection, new days are found automatically: no ALTER TABLE ADD PARTITION, no crawler runs.
Filter on the partition column in WHERE so Athena reads fewer files. You pay per data scanned.
The same idea works for VPC flow logs (choose Parquet and Hive-compatible prefixes for cheaper queries), ALB, CloudFront and WAF logs.
{ }Structure
Your files plus a table, or CloudTrail’s own managed store.
Up to 3,653 days (one-year extendable) or 2,557 days (seven-year)
Availability
Everyone
Closed to new customers from 31 May 2026
!Common mistakes the exam tests
Creating a table without partitions over years of logs: every query scans everything, slow and costly.
Using manual partitions and forgetting to add new ones: queries silently miss recent days. Projection fixes this.
Wrong storage.location.template (for example missing the o-orgid part of an organization trail path): the query returns zero rows.
Recommending CloudTrail Lake to a new account in a question set after May 2026: new customers can’t enable it; use Athena or Logs Insights.
Thinking a CloudTrail Lake KMS key can be swapped later: once associated it can’t be removed or changed.
✓Benefits and use cases
Who did what across a year of organization trail logs.
Flow log forensics months after an incident.
Cheap long-term log analytics for compliance.
CloudTrail Lake in one paragraph
CloudTrail Lake converts events into Apache ORC in an immutable event data store. You query it with SQL (any valid Trino SELECT statement), or let the query generator write SQL from an English prompt. It has managed and custom dashboards, and you can federate an event data store to the AWS Glue Data Catalog to query it from Athena. Important for 2026: AWS says CloudTrail Lake is no longer open to new customers from 31 May 2026; existing customers can keep using it. AWS’s own migration advice for CloudTrail Lake users is Amazon CloudWatch: since December 2025 CloudWatch can ingest CloudTrail organization-wide (also VPC and WAF logs), normalize to OCSF with pipelines, and expose the data as Apache Iceberg tables in S3 Tables for Athena; existing event data stores can be exported with aws logs create-import-task.
Think of it like · a library archive
Old newspapers, two ways to search them.
AWS concept
Familiar thing
S3 log files
≈
Boxes of old newspapers in your own basement
Athena table
≈
A catalogue card that says what is in each box
Partition projection
≈
Boxes labelled by year/month/day, so you open only the right box
Scanning without partitions
≈
Reading every newspaper to find one article
CloudTrail Lake
≈
A paid archive service that stores the papers and lets you search by question
Closed to new customers
≈
The archive no longer takes new members, but current members keep their cards
Real AWS example · Athena query on a projected CloudTrail table
Who got AccessDenied most in October? The timestamp partition keeps the scan small.
SELECT useridentity.arn, eventsource, eventname, count(*) AS calls
FROM cloudtrail_logs
WHERE "timestamp" BETWEEN '2026/10/01' AND '2026/10/10'
AND errorcode = 'AccessDenied'
GROUP BY 1, 2, 3
ORDER BY calls DESC
LIMIT 20;
-- table properties that made this work
'projection.enabled'='true',
'projection.timestamp.type'='date',
'projection.timestamp.format'='yyyy/MM/dd',
'projection.timestamp.range'='2020/01/01,NOW',
'storage.location.template'='s3://amzn-s3-demo-bucket/AWSLogs/111122223333/CloudTrail/eu-west-1/${timestamp}'
In CloudTrail Lake the same question is a SELECT … FROM <event-data-store-id> WHERE errorCode = 'AccessDenied'.
Exam-style scenario
A company keeps 3 years of organization trail logs in S3. Analysts run Athena queries every day, but costs are high and queries are slow. Data from the last few days is often missing from results. What should the security engineer do?
A. Move all logs into CloudWatch Logs and use Logs Insights.
B. Recreate the Athena table with partition projection that matches the trail’s account/Region/date prefix, and require date filters in queries.
C. Run a Glue crawler every hour over the whole bucket.
D. Enable S3 Transfer Acceleration.
Show the worked answer
Answer: B
Projection finds new partitions automatically and lets queries read only the needed days, which lowers cost and fixes the missing recent data.
Why not the others
A: Expensive for 3 years of data and a big migration.
C: Crawlers add cost and delay; projection is the documented fix.
D: Speeds uploads, not queries.
How the exam tests this
“SQL over trail logs in S3”→ Athena.
“New partitions without ALTER TABLE”→ Partition projection.
Parse and normalize: Firehose and Lambda transforms
Skill 1.2.5 is about turning raw logs into something you can correlate. Different sources use different formats: JSON from CloudTrail, space-separated flow logs, plain text from applications. Before you can join them, you parse (split into fields), normalize (same field names and types, for example OCSF), and often enrich or mask them. The standard AWS pipeline is a CloudWatch Logs subscription filter into Amazon Data Firehose with a Lambda transform.
Concept profile
Firehose with a Lambda transform
often confused with CloudWatch Logs subscription straight to Lambda
?What is it?
A managed delivery stream that buffers incoming records, calls your Lambda function on each batch to change them, and delivers the results to S3, OpenSearch, Splunk, other partners or an HTTP endpoint.
⚙How does it work?
A subscription filter on a log group sends matching events (compressed) to Firehose, Kinesis Data Streams, Lambda or OpenSearch.
Firehose buffers by size or time. For the Lambda transform the buffer hint is 0.2 to 3 MB (default 1 MB) and 0 to 900 seconds (default 60 s).
The function returns every record with recordId, result (Ok, Dropped or ProcessingFailed) and new base64 data.
Failed records go to the S3 backup under a processing-failed/ prefix, so nothing is lost silently.
Firehose can also decompress CloudWatch Logs records and convert JSON to Parquet or ORC for cheaper Athena queries.
{ }Structure
Filter, buffer, transform, deliver; failures are kept.
⇄Firehose + Lambda vs CloudWatch Logs subscription straight to Lambda
Firehose + Lambda
Subscription straight to Lambda
Buffering and retries
Built in
You write it
Delivery to S3 / OpenSearch / Splunk
Built in
Your code
Failed records
Backed up to S3
Lost unless you handle them
Latency
Seconds to minutes (buffer)
Near real time
Exam hint
“least operational overhead”, “deliver to”
“custom real-time action per event”
!Common mistakes the exam tests
Setting the transform function timeout above 5 minutes: Firehose supports at most 5 minutes per invocation.
Returning fewer records than received, or changing recordId: Firehose treats them as failures.
Writing a custom Lambda that loads S3 and OpenSearch itself when Firehose delivers natively (more code to run).
Forgetting that a subscription filter needs a role (for Firehose or Kinesis destinations) that CloudWatch Logs can assume.
Masking secrets only after storage: transform in the stream so raw values never reach the destination.
✓Benefits and use cases
Mask emails or tokens before logs reach a SIEM.
Convert app text logs into JSON fields.
Map fields to OCSF before sending to Security Lake as a custom source.
Think of it like · a post office sorting room
Letters arrive messy; they leave sorted.
AWS concept
Familiar thing
Subscription filter
≈
A clerk who picks only the letters for the security department
Firehose buffer
≈
A sack that is sent when it is full or every hour
Lambda transform
≈
A worker who opens each letter, blacks out private details and stamps a standard form
Ok / Dropped / ProcessingFailed
≈
Delivered, thrown away on purpose, or put aside for checking
processing-failed/ prefix
≈
The shelf for letters that couldn’t be read
OCSF
≈
Every letter rewritten on the same official form
Real AWS example · what the transform function must return
One output entry per input record, same recordId, a result, and base64 data.
The first record’s data decodes to {"time":"2026-10-10T09:15:00Z","user":"*****"}: the user was masked in flight.
Exam-style scenario
A company must stream application logs from CloudWatch Logs to a third-party SIEM. Email addresses must be masked before the data leaves AWS, records that can’t be parsed must be kept for review, and the team wants the least operational overhead. What should it build?
A. A subscription filter to a Lambda function that masks data and calls the SIEM API with custom retry code.
B. A subscription filter to Amazon Data Firehose with a Lambda transform that masks emails, an HTTP endpoint (or partner) destination, and S3 backup for failed records.
C. A daily export of the log group to S3 and a script that uploads files.
D. A metric filter for email addresses.
Show the worked answer
Answer: B
Firehose gives buffering, retries, partner delivery and failed-record backup; the Lambda transform only masks.
Why not the others
A: Works, but you must build retries and failure handling yourself.
C: Not streaming, and data leaves unmasked unless the script masks it.
D: Metric filters count events; they don’t change or deliver logs.
How the exam tests this
“Transform logs in flight, least effort”→ Firehose + Lambda transform.
“Send log group events onward”→ Subscription filter.
“Keep records that failed parsing”→ S3 backup processing-failed/.
“Cheaper Athena queries on streamed logs”→ Firehose record format conversion to Parquet.
“Same field names across sources”→ Normalize (for example OCSF).
Correlate: OpenSearch, Security Hub findings and Managed Grafana
Correlation means putting events from different sources side by side to tell one story: a GuardDuty finding, the CloudTrail calls of the same role, and the flow logs of the same instance. Skill 1.2.5 names Amazon OpenSearch Service and Amazon Managed Grafana; skill 1.2.4 names Security Hub findings. Each plays a different role.
Concept profile
OpenSearch Service with Security Lake direct query
often confused with Amazon Athena
?What is it?
A search and analytics service with dashboards. Its direct query integration with Amazon Security Lake (a zero-ETL integration) lets you search and visualize Security Lake’s OCSF data without building an ingestion pipeline.
⚙How does it work?
Create a Security Lake data source in OpenSearch Service and query from OpenSearch Dashboards or the OpenSearch UI.
Interactive queries read Security Lake data in place. Indexed views (materialized views) copy selected data into an index for fast dashboards, and cost extra for indexing and storage.
Security Lake tables are managed by Lake Formation; from another account you need a Security Lake subscriber with query access.
Comes with query templates and prebuilt dashboards. Filter on accountid, region and time_dt for speed.
Classic ingestion still works: Firehose or OpenSearch Ingestion pipelines push logs into an OpenSearch domain for full-text search and alerting.
{ }Structure
Data stays in Security Lake; OpenSearch searches it, Grafana shows everything together.
⇄OpenSearch direct query vs Amazon Athena
OpenSearch direct query
Athena
Interface
Dashboards, search, visual exploration
SQL console and API
Data movement
None (or indexed views for speed)
None
Best for
Interactive hunting, dashboards, alerting
Ad hoc SQL and reports
Pricing
OpenSearch compute units per hour
Per data scanned
Exam hint
“visualize”, “search across”, “dashboards”
“SQL query”, “ad hoc”, “serverless”
!Common mistakes the exam tests
Building Firehose pipelines to copy all Security Lake data into OpenSearch when direct query can read it in place.
Querying from another account without a Security Lake subscriber with query access (Lake Formation permissions).
Using Managed Grafana as a log store: it only visualizes data from its sources.
Treating Security Hub as a log search tool: it aggregates findings, not raw logs.
Forgetting Security Hub only receives findings in Regions where it is enabled; use a home (aggregation) Region.
✓Benefits and use cases
Threat hunting across OCSF data from all accounts.
Security operations dashboards with alerting.
Correlating a finding with the logs around it.
Security Hub findings and Amazon Managed Grafana
Security Hub (the posture part is now called Security Hub CSPM, which uses the AWS Security Finding Format, ASFF) collects findings from GuardDuty, Inspector, Macie, Config and partners, and correlates them. To analyse findings, use filters and insights: an insight is a saved filter plus a group-by attribute, for example “active critical findings grouped by resource ID”, which shows the resources that need help first. Send findings to EventBridge to act on them. The newer Security Hub experience uses OCSF for findings, the same schema as Security Lake.
Amazon Managed Grafana gives you managed Grafana workspaces. Users sign in with IAM Identity Center or a SAML 2.0 identity provider. It reads data sources such as CloudWatch, OpenSearch, X-Ray, Timestream and Managed Service for Prometheus (and more through plugins), so one dashboard can show metrics, logs and findings from many accounts.
Think of it like · a police investigation room
Detectives connect clues from many places.
AWS concept
Familiar thing
Security Lake
≈
The evidence store, every item labelled in one format
OpenSearch direct query
≈
Searching the evidence store from your desk without moving the boxes
Indexed view
≈
Photocopies of the most important files kept on your desk for speed
Security Hub insight
≈
A board listing which suspects appear in the most reports
Managed Grafana
≈
The big wall screen showing maps, timelines and charts together
Correlation
≈
Linking the phone record, the camera image and the witness statement to one person
Real AWS example · a Security Hub insight to find the worst resources
Fix the S3 bucket first: it has the most critical findings.
Exam-style scenario
A security team uses Amazon Security Lake in a delegated administrator account. Analysts want to search and build dashboards over the OCSF data from all accounts, with the least data movement and no ingestion pipelines to run. What should they use?
A. Firehose streams that copy every Security Lake source into an OpenSearch domain.
B. An Amazon OpenSearch Service direct query data source for Security Lake, using indexed views only for the dashboards that need speed.
C. Amazon Managed Grafana as the log store.
D. Security Hub insights.
Show the worked answer
Answer: B
Direct query is the zero-ETL integration: data stays in Security Lake, and indexed views are optional for performance.
Why not the others
A: Copies data and adds pipelines to run.
C: Grafana visualizes; it doesn’t store or search logs itself.
D: Insights group findings, not raw OCSF logs.
How the exam tests this
“Search Security Lake without ETL”→ OpenSearch direct query.
“Faster dashboards over Security Lake”→ Indexed views (materialized views).
“Which resources have the most findings”→ Security Hub insight with group-by.
“Shared dashboards, SSO, many data sources”→ Amazon Managed Grafana.
“Query Security Lake from another account”→ Subscriber with query access.
Must remember
Direct query = zero-ETL, data stays in Security Lake.
Indexed views copy data and cost extra.
Insight = filter + group-by attribute.
Security Hub CSPM uses ASFF; Security Lake uses OCSF.
Grafana: visualize, sign in via Identity Center / SAML.
Analyze resource logging: Lambda, API Gateway, health checks, CloudFront
Task 1.3 is troubleshooting. Skill 1.3.1 gives four examples where logging or monitoring depends on the right setting and the right permission. In every case, ask two questions: is logging turned on in the right place? and who writes the logs, and are they allowed to?
Concept profile
API Gateway CloudWatch logging
often confused with Lambda function logging
?What is it?
Two independent kinds of logs for REST APIs: execution logs (what API Gateway did with the request, errors, traces) and access logs (who called what, in a format you define with $context variables).
⚙How does it work?
API Gateway writes the logs itself, so it needs an IAM role it can assume (trust apigateway.amazonaws.com, policy AmazonAPIGatewayPushToCloudWatchLogs).
Set that role once per account per Region in the API Gateway account settings (cloudwatchRoleArn).
Then turn on logging per stage: execution log level ERROR or INFO, optional data trace, and an access log destination (log group or Firehose stream whose name starts with amazon-apigateway-).
Execution logs go to API-Gateway-Execution-Logs_{rest-api-id}/{stage}.
HTTP APIs support access logging only; use $context error variables to see integration errors.
{ }Structure
Four resources, four places where logging silently fails.
⇄API Gateway logging vs Lambda function logging
API Gateway logging
Lambda logging
Who writes the logs
API Gateway service, using the account CloudWatch role
Lambda, using the function’s execution role
Where the role is set
Account settings, per Region
On each function
Default log group
API-Gateway-Execution-Logs_{id}/{stage}
/aws/lambda/<function-name>
Turn on
Per stage (log level, access format)
On by default if the role allows it
Exam hint
“no logs from any API in this Region”
“function runs but no logs”
!Common mistakes the exam tests
Setting the stage log level to INFO but never setting the account-level CloudWatch role: nothing is written.
Setting the role in us-east-1 and expecting APIs in eu-west-1 to log: the setting is per Region.
Leaving data trace on in production: it can log full request and response bodies, including sensitive data.
Removing logs permissions from a Lambda execution role: the function still runs, but no logs appear.
Pointing Lambda at a custom log group that is KMS-encrypted without the logs service in the key policy.
✓Benefits and use cases
Prove who called an API and from which IP (access logs).
Debug 5xx errors between API Gateway and the backend (execution logs).
Audit Lambda behaviour with JSON logs and log levels.
Lambda, health checks and CloudFront
Lambda sends logs to /aws/lambda/<function-name> by default if the execution role allows logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents (the managed policy AWSLambdaBasicExecutionRole). Advanced logging controls let you choose text or JSON, filter by log level, and pick a custom log group. Logs can take 5 to 10 minutes to appear. Route 53 health checks call your endpoint from checkers around the world; an endpoint is healthy only if more than 18% of checkers say so. If a security group or NACL blocks the Route 53 health checker IP ranges, the check fails even though users can reach the app. Health check metrics and alarms are in us-east-1. CloudFront standard logging (v2) can deliver to CloudWatch Logs, Firehose or S3 (Parquet only for S3); the legacy version delivers to S3 only. Real-time logs go to Kinesis Data Streams within seconds. Standard logs are best effort, so a few entries can arrive late or be missing.
Think of it like · a shop with CCTV
Cameras only help if they are switched on and can record.
AWS concept
Familiar thing
Stage log level
≈
The camera’s on/off switch for one shop entrance
Account CloudWatch role
≈
The recorder in the back office; without it, every camera shows live but saves nothing
Per Region
≈
Each branch of the shop needs its own recorder
Lambda execution role
≈
The staff badge that lets a worker write in the shop diary
Route 53 health checker IPs
≈
The inspector who must be let in the door to check the shop is open
CloudFront best-effort logs
≈
A visitor book that a few people forget to sign
Real AWS example · fix API Gateway logging in one Region
Set the account role, then turn on execution logs for every method on the stage.
The role trusts apigateway.amazonaws.com and has AmazonAPIGatewayPushToCloudWatchLogs. AWS STS must be active in the Region.
Exam-style scenario
A team enabled INFO execution logging on all stages of three REST APIs in eu-west-1. No API Gateway log groups appear. Lambda logs for the backend functions appear normally. What is the most likely cause?
A. The Lambda execution roles lack logs permissions.
B. No CloudWatch Logs role ARN is set in the API Gateway account settings for eu-west-1.
C. Access logging must be enabled before execution logging.
D. The log groups must be created manually with a KMS key.
Show the worked answer
Answer: B
API Gateway writes execution logs using the account-level role, which is set per Region. Without it, nothing is written for any API.
Why not the others
A: Lambda logs are working, and they don’t affect API Gateway logs.
C: The two are independent.
D: API Gateway creates the log group itself; KMS isn’t required.
How the exam tests this
“No logs from any REST API in a Region”→ Set cloudwatchRoleArn in account settings.
“Lambda runs, no logs”→ Execution role missing logs:PutLogEvents (etc.).
“Health check fails but site works”→ Allow Route 53 health checker IPs.
“Can’t find the health check alarm”→ Look in us-east-1.
“CloudFront logs to CloudWatch Logs or Firehose”→ Standard logging v2.
Must remember
API GW: account role per Region + stage log level.
Skill 1.3.2 asks you to fix broken logging. Most exam stories are one of five problems: logging was never turned on, the writer has no permission, the KMS key policy blocks encryption, the configuration points to the wrong place, or the instance can’t reach the AWS endpoint. Work through them in that order.
Concept profile
Troubleshooting the CloudWatch agent
often confused with Logs missing because of a resource policy
?What is it?
Finding why the CloudWatch agent on EC2 or on-premises servers doesn’t send logs or metrics, and fixing the agent, its configuration, its credentials or its network path.
⚙How does it work?
Check it runs: amazon-cloudwatch-agent-ctl -m ec2 -a status. A "status": "stopped" means start it, or keep it running with Systems Manager (State Manager, Day 2).
Read the agent log (/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log) and, if it won’t start, configuration-validation.log. Set "debug": true for more detail.
After editing the config, reload it with -a fetch-config; otherwise the old config stays active.
Credentials: attach an instance profile with CloudWatchAgentServerPolicy (on-premises: a profile with credentials).
Network: private subnets need a NAT gateway or interface VPC endpoints for CloudWatch Logs and CloudWatch. The AWSSupport-TroubleshootCloudWatchAgent runbook checks role and connectivity.
{ }Structure
Five questions, in order, for any missing-log problem.
⇄Agent troubleshooting vs Logs missing because of a resource policy
Agent problem
Resource policy problem
Who writes
The agent, with the instance role
An AWS service (CloudTrail, flow logs, S3, CloudFront)
Permission lives in
IAM role (identity policy)
Bucket policy or KMS key policy
Where to look
Agent log on the instance
Trail status, flow log status, CloudTrail errors
Typical fix
Policy, config, fetch-config, endpoint
Add service principal + SourceArn
Exam hint
“agent installed but no logs”
“trail stopped delivering after policy change”
!Common mistakes the exam tests
Editing amazon-cloudwatch-agent.json and not running fetch-config.
Wrong file_path (or the agent user can’t read the file): no error in CloudWatch, just no data.
Instance role without CloudWatchAgentServerPolicy, or an access key on the instance that overrides the role.
Private subnet with no NAT and no logs interface endpoint.
Looking for agent metrics in the wrong namespace: the default is CWAgent.
✓Benefits and use cases
OS security logs (/var/log/secure, Windows Security) in CloudWatch.
Metrics like memory and disk that EC2 doesn’t send by default.
Fleet-wide config from Parameter Store.
Missing logs caused by resource policies and keys
S3 delivery: CloudTrail needs a bucket policy that allows cloudtrail.amazonaws.coms3:GetBucketAcl on the bucket and s3:PutObject on AWSLogs/<account-id>/* (plus AWSLogs/<org-id>/* for an organization trail), with aws:SourceArn of the trail. A wrong prefix, a wrong SourceArn or a Requester Pays bucket stops delivery. Flow logs, CloudFront v2 and other vended logs use delivery.logs.amazonaws.com; S3 server access logs use logging.s3.amazonaws.com. KMS: an encrypted log group needs the key policy to allow logs.<region>.amazonaws.com (symmetric keys only), and a trail with SSE-KMS needs cloudtrail.amazonaws.com allowed to kms:GenerateDataKey*. If you disable the key, CloudWatch Logs can’t read data encrypted with it. Flow logs to CloudWatch Logs need an IAM role that trusts vpc-flow-logs.amazonaws.com and allows logs:CreateLogStream, logs:PutLogEvents and the describe actions.
Think of it like · a delivery to a locked warehouse
The parcel never arrives. Why?
AWS concept
Familiar thing
Agent not running
≈
The courier didn’t come to work today
Missing IAM permission
≈
The courier has no delivery badge
Bucket policy
≈
The warehouse guard’s list of couriers allowed in
KMS key policy
≈
The parcel must go in a locked box, but the courier wasn’t given the key
No VPC endpoint or NAT
≈
The road out of the village is closed
fetch-config
≈
Giving the courier the new address list; until then he drives the old route
Real AWS example · agent stopped, plus the key policy line people forget
Status check, then the statement an encrypted log group needs in its KMS key policy.
-s restarts the agent after loading the config from Parameter Store.
Exam-style scenario
After a security review, an organization trail stopped delivering logs to its S3 bucket in the logging account. The trail uses SSE-KMS. The review replaced the KMS key policy with one that only allows the security team’s role. What should the engineer do?
A. Turn off SSE-KMS on the trail.
B. Add a key policy statement allowing cloudtrail.amazonaws.com to use kms:GenerateDataKey* (with the trail’s SourceArn / encryption context condition).
C. Add s3:PutObject for the security team role to the bucket policy.
D. Recreate the trail in every member account.
Show the worked answer
Answer: B
CloudTrail must be allowed to generate data keys to encrypt log files; the new key policy removed that permission.
Why not the others
A: Lowers security instead of fixing the policy.
C: The writer is CloudTrail, not the security team role.
D: Not needed; the organization trail is fine apart from the key.
How the exam tests this
“Agent installed, no logs, role correct”→ Check status, agent log, fetch-config, endpoint.
“Private subnet, no internet”→ Interface VPC endpoint for CloudWatch Logs.
Cheatsheet extras: session logging, exports and other log paths
The cheat sheet’s “EC2 internal logging” and troubleshooting pages ask one question many ways: how do you get the logs you need off a workload, and why don’t they arrive? Section 06 covered the CloudWatch agent. This section adds Session Manager session logging (what an admin typed), the export task, container logging on Fargate, and log groups as an access boundary.
Concept profile
Session Manager session logging
often confused with CloudTrail StartSession events
?What is it?
A Session Manager preference that saves the full input and output of shell sessions on managed instances to an S3 bucket, a CloudWatch Logs log group, or both. It is how you audit what an administrator actually did on a server without SSH.
⚙How does it work?
Set once per account and Region in the Session Manager preferences (the SSM-SessionManagerRunShell document).
CloudWatch Logs can stream session data as it happens or upload it when the session ends; S3 gets the log when the session ends.
The instance profile writes the logs, so it needs s3:PutObject or logs:CreateLogStream/logs:PutLogEvents (and kms:Decrypt/kms:GenerateDataKey if a KMS key encrypts the bucket, log group or session).
A kmsKeyId encrypts the session data itself in transit, on top of TLS.
Session logging doesn’t cover port forwarding or SSH over Session Manager sessions; there the data is encrypted end to end.
{ }Structure
Two records: CloudTrail for the who, session logs for the what.
⇄Session logging vs CloudTrail StartSession events
Session Manager session logging
CloudTrail StartSession events
Records
Commands and output inside the session
That a session started or ended
Where
S3 and/or CloudWatch Logs
CloudTrail trail / event history
Turned on by
Session Manager preferences
Always (management event)
Written by
The instance profile role
CloudTrail
Exam hint
“record every command an admin runs”
“who connected to the instance and when”
!Common mistakes the exam tests
Expecting CloudTrail to show the commands typed in a session: it only shows the API calls.
Choosing SSH with key pairs and bastion hosts when the question says no inbound ports and full session auditing.
Forgetting that the instance profile, not the user, needs write access to the log bucket or group.
Expecting port forwarding sessions to be logged.
Encrypting the log group with a KMS key that the instance role can’t use: session logging fails, and the session itself may fail to start.
✓Benefits and use cases
Auditable admin access to EC2 and on-premises servers with no open ports.
Evidence for compliance (who did what on which server).
Real-time review of commands in CloudWatch Logs with metric filters.
Think of it like · a hotel room with a visitor book
The front desk logs who got a key card; a recorder in the room logs what happened.
AWS concept
Familiar thing
CloudTrail StartSession
≈
The front desk book: “Mr X took the key to room 12 at 10:04”
Session logging
≈
The room’s voice recorder
S3 vs CloudWatch Logs
≈
Tapes filed after checkout vs a live feed to security
Instance profile permissions
≈
The room needs a working cable to send the recording
Port forwarding not logged
≈
A sealed parcel passed through the room: you know it passed, not what was inside
No inbound ports
≈
No back door; everyone comes through the front desk
Real AWS example · session logging preferences and the instance role statement
Part of the Session Manager preferences document, then the extra permission the instance profile needs.
In production, scope Resource to the bucket, log group and key ARNs. With streaming on, a metric filter can alert on a command such as sudo su within seconds.
Exam-style scenario
Administrators must reach Linux instances in private subnets with no inbound ports open. Auditors want a record of every command typed, kept in S3 for one year, and the security team wants to know who started each session. Which solution meets the requirements?
A. A bastion host with SSH, and CloudTrail to record the commands.
B. Session Manager with session logging to an S3 bucket (with a lifecycle rule), an instance profile that can write to it, and CloudTrail for StartSession events.
C. EC2 Instance Connect with VPC flow logs.
D. Session Manager port forwarding to SSH, with session logging on.
Show the worked answer
Answer: B
Session Manager needs no inbound ports; session logging captures the commands; CloudTrail records who started each session.
Why not the others
A: Needs an open SSH port, and CloudTrail never records commands typed in a shell.
C: Instance Connect needs a path to port 22, and flow logs show traffic, not commands.
D: Port forwarding and SSH sessions are not captured by session logging.
Other ways logs leave (or fail to leave) a workload
Need
Use
Watch out for
Copy old log data to S3 once in a while
CloudWatch Logs export task (CreateExportTask)
Batch, not real time; data can take up to 12 hours to be exportable; one running task per account; bucket in the same Region with a policy for logs.<region>.amazonaws.com
Stream logs as they arrive
Subscription filter to Firehose, Kinesis, Lambda or OpenSearch (cross-account through a destination)
Covered in Day 3; the right answer whenever the question says “real time”
Containers on Fargate
The awslogs (or FireLens) log driver in the task definition
You can’t install the CloudWatch agent on Fargate hosts; the task execution role needs logs permissions
Keep sensitive logs apart
A separate log group with its own IAM permissions, retention and KMS key
Log groups never expire by default; set retention (1 day to 10 years) or pay to keep everything
The cheat sheet’s old “CloudWatch Logs agent” (awslogs, /var/log/awslogs.log) is replaced by the unified CloudWatch agent. The checks are the same as in section 06: installed, running, config loaded, role permissions, and a network path to the endpoint.
Pick the export task only for occasional archiving. Any question that says real time, near real time or stream wants a subscription filter, not CreateExportTask.
How the exam tests this
“Record every command administrators run”→ Session Manager session logging to S3 / CloudWatch Logs.
“Who connected to the instance”→ CloudTrail StartSession events.
Tap a branch to jump back to that part of the lecture. Try redrawing this map from memory on paper.
2. Match the requirement to the tool or fix
The left column is the exam wording; the right column is the answer.
3. “If the question says… pick…”
Find which IP failed sign-in most, logs in CloudWatchLogs Insights
Alert every time the pattern appearsMetric filter + alarm, not Insights
SQL over years of trail logs in S3Athena with partition projection
Recent days missing from Athena resultsPartition projection (no manual partitions)
Managed SQL store of events (already enrolled)CloudTrail Lake
Mask or reformat logs before a SIEMFirehose + Lambda transform
Search Security Lake without pipelinesOpenSearch direct query
Which resources have the most findingsSecurity Hub insight (group by)
API Gateway writes no logs in a RegionAccount-level CloudWatch role ARN
Health check fails, site worksAllow Route 53 health checker IPs
Agent installed, no dataStatus, agent log, fetch-config, role, endpoint
Trail or log group stopped after a policy changeBucket policy / KMS key policy for the service
Day 4 in one sentence
Query logs where they live, reshape them in flight, correlate them on one screen, and when they vanish, check switch, permission, key, config and network.
09
Must-remember cheat sheet
1. Logs Insights: QL, PPL, SQL; pay per GB scanned.
2. Insights timeout 60 min; results 7 days.
3. Insights = investigate; metric filter = alert.
4. Athena: partition projection, no ADD PARTITION.
5. Easiest Athena table: from the CloudTrail console.
Check the basics first. Pick an answer, then press Check.
11
Hard quiz
Harder than most real exam questions on purpose, with one “choose two” and one matching question. Aim for 9 out of 12.
12
Domain 1 review quiz (Days 1–4)
Questions from every Detection day. If you miss one, reopen that day’s lecture and its “Must remember” boxes.
13
Practice test questions (from your mock exams)
These 9 questions come from your own SCS-C03 mock exams, matched to today’s topics. Wording was modernized where services were renamed (for example CloudWatch Events is now Amazon EventBridge), and each answer was checked against current AWS behavior; where the mock’s answer key was wrong, the explanation says so. The source (mock and question number) is at the end of each explanation.
14
Hands-on lab (30 minutes, low cost)
Create a small Lambda function with a new execution role that has no logs permissions. Invoke it and confirm that no /aws/lambda/… log group appears. Attach AWSLambdaBasicExecutionRole, invoke again and wait a few minutes for the logs.
If your trail sends events to CloudWatch Logs, open Logs Insights, choose that log group and the last hour, and run stats count(*) by eventSource, eventName | sort count(*) desc | limit 20. Note the “data scanned” figure under the results.
In the CloudTrail console, open Event history and choose Create Athena table for your trail’s bucket. Run a query with a narrow date filter and check the data scanned.
Create a test REST API (mock integration). Turn on ERROR execution logging on a stage before setting the account CloudWatch role, then see what happens. Set cloudwatchRoleArn for the Region and call the API again.
Optional: create a log group with a new KMS key whose policy lacks the logs principal, and see associate-kms-key fail. Add the statement from section 06 and try again.
Clean up: delete the Lambda function, test API, IAM roles and the test log groups; schedule the KMS key for deletion (keys cost money each month); drop the Athena table and delete query result files in S3. Athena and Logs Insights charge per data scanned, so always use short time ranges.