AWS Security Monitoring in 2023: Untangle the chaos

Michael Wittig – 04 Aug 2023

AWS security monitoring is a set of practices, tools, and processes designed to detect and respond to security threats and vulnerabilities within the Amazon Web Services (AWS) cloud environment. Sounds easy? In this blog post, I share how I use a variety (but not all) of AWS services to detect issues quickly and alert someone in charge to solve them.

Untangle the chaos

Available AWS services

AWS is confusing. Multiple services are competing for your attention when you try to understand the basics of security monitoring. Let me untangle the chaos by grouping the related AWS services/capabilities into three groups (visually and textually). After that, I will present you my selection of AWS services.

I came up with the following three groups:

  1. Sources of information: Provide raw security events or data that we can analyze. Usually, the volume is very high and overwhelming. Low-level findings also fall in this category.
  2. Best practices and anomaly detection: Create higher-level findings by analyzing raw security events and data using predefined best practices or machine learning to detect unusual patterns.
  3. Aggregation: Correlate and aggregate findings with all kinds of information and present them in a human-friendly way.

Some AWS services belong to multiple groups. E.g., AWS Security Hub checks for best practices and aggregates events in a human-friendly way.

Visual grouping of AWS security services and capabilities

The following sources of information can be used to monitor security-related activity in your AWS Account:

  • AWS API (current configuration of each resource can be requested)
  • AWS Config configuration item
  • AWS CloudTrail management and data event
  • Amazon VPC Flow Logs
  • Amazon Route 53 DNS query logs
  • AWS Health event
  • Amazon Inspector vulnerability finding
  • Amazon Macie sensitive data finding
  • Amazon ECR Basic scanning finding
  • AWS IoT Device Defender Detect finding

The following AWS services/capabilities check the above sources against best practices or to detect anomalies. The following table lists the services and the sources that they use.

AWS service/capability AWS API Config item Config rule CloudTrail event VPC Flow Logs Route 53 DNS query logs
AWS Config rule no yes x no no no
Amazon Macie policy finding yes no no no no no
AWS Trusted Advisor check yes no no no no no
AWS Security Hub control no no yes no no no
AWS CloudTrail Insights event no no no yes no no
Amazon GuardDuty no no no yes yes yes
AWS IAM Access Analyzer finding yes no no no no no
AWS Firewall Manager finding yes no no no no no
AWS IoT Device Defender Audit finding yes no no no no no

Last but not least, AWS aggregates all the information in different services. The following table lists the services and the sources that they use.

AWS service/capability Config rule CloudTrail event VPC Flow Logs Route 53 DNS query logs Inspector findings Macie finding Security Hub finding GuardDuty finding Trusted Advisor check Health event Access Analyzer finding Firewall Manager finding IoT Device Defender finding
AWS Security Hub yes no no no yes yes x yes no yes yes yes yes
Amazon Security Lake no yes yes yes no no yes no no no no no no
Amazon Detective no yes yes no no no yes yes no no no no no
AWS Audit Manager yes yes no no no no yes no no no no no no
AWS Config Conformance Pack yes no no no no no no no no no no no no
AWS Trusted Advisor no no no no no no yes no yes no no no no

Selecting the right AWS services

Many AWS services/capabilities overlap in what information they analyze and the checks they perform. Examples:

  • AWS Trusted Advisor, AWS Config Rules / Conformance Packs, and AWS Security Hub Security standards all check if you follow predefined best practices in your AWS account.
  • Amazon Detective, AWS CloudTrail Insight, and Amazon GuardDuty analyze your CloudTrail data to find suspicious activity.

If you mindlessly enable all the services, you pay for the same check multiple times. But the issue worsens: Each service creates a finding, but all refer to the same problem. For example, if you make AWS Trusted Advisor, AWS Config Rules / Conformance Packs, AWS Security Hub Security standards, and Amazon Macie create an S3 bucket public, a firework of findings. They all tell you that the bucket is now public. But how do you manage this information overload? Amazon invented another service to aggregate this information again: Amazon Detective.

I suggest that you enable the following AWS services/capabilities in your a delegated admin AWS Account (aka security account, a feature of AWS organizations) to check all your AWS accounts in one place:

  • AWS Config with a retention period of 1 year in each region you use.
  • AWS Security Hub with the AWS Foundational Security Best Practices (FSBP) standard enabled in each region you use (AWS Security Hub requires AWS Config).
  • Optional: Amazon GuardDuty in each region you use.
  • Optional: Amazon Inspector in each region you use.

By default, GuardDuty and Inspector send findings to Security Hub. Therefore, Security Hub is your central place to work with findings. A finding can be NEW, acknowledged (NOTIFIED), SUPPRESSED, or RESOLVED. To end goal is to keep the number of findings low.

You can disable AWS Security Hub controls if you disagree with the “best practice.”

Last, we must alert someone in charge to handle the finding.

Incident Response

A finding is the beginning, not the end, of the security incident response process. Once a finding is created, the process starts:

  1. Alert the right person.
  2. Analyse finding (research additional data).
  3. Fix issue.
  4. Resolve finding.

To alert the right person, you have to understand how Security Hub publishes information about findings. Security Hub publishes an event to EventBridge whenever a finding is created or updated. If you followed my advice to use a delegated admin AWS Account, the event is published in the source (aka member) and delegated admin accounts. I recommend creating an EventBridge rule to listen to new findings in each AWS member account (not the delegated admin). Assuming that you use AWS accounts to isolate workloads, there should be a relationship between the AWS account and a team in charge that can be alerted when a new finding arrives. Remember to create an EventBridge rule in each region you use.

The following Terraform snippet creates an EventBridge rule connected to an SNS topic:

resource "aws_cloudwatch_event_rule" "security_hub_finding" {
name = "security-hub-finding"
description = "Findings (severity >= high) from AWS SecurityHub."
event_pattern = <<JSON
{
"source": [
"aws.securityhub"
],
"detail-type": [
"Security Hub Findings - Imported"
],
"detail": {
"findings": {
"Severity": {
"Normalized": [{"numeric": [">=", 70]}]
},
"Workflow": {
"Status": [
"NEW"
]
},
"RecordState": [
"ACTIVE"
]
}
}
}
JSON
}

resource "aws_cloudwatch_event_target" "security_hub_finding" {
rule = aws_cloudwatch_event_rule.security_hub_finding.name
target_id = "sns"
arn = "YOUR SNS TOPIC ARN"
}

There is one issue with this approach. As long as the status is NEW, you will receive an EventBridge event per finding every day. If this feels too spammy, write a Lambda function to set the status to NOTIFIED. Alternatively, you can use marbot, our AWS Monitoring chatbot. marbot creates the EventBridge rule for you and sets the status to NOTIFIED after you receive an alert in Slack or Microsoft Teams about the new finding.

Summary

Many AWS services and capabilities are relevant when talking about AWS Security Monitoring. They overlap in what information they analyze and the checks they perform. If you carefully select the right AWS services, you can avoid duplicate findings and costs while observing all the security-relevant sources. Use AWS Security Hub as your central place for AWS Security Monitoring. Optionally, use GuardDuty and Inspector to feed additional insights into Security Hub. Last, use EventBridge to forward Security Hub findings to the right team.

Michael Wittig

Michael Wittig

I’ve been building on AWS since 2012 together with my brother Andreas. We are sharing our insights into all things AWS on cloudonaut and have written the book AWS in Action. Besides that, we’re currently working on bucketAV, HyperEnv for GitHub Actions, and marbot.

Here are the contact options for feedback and questions.