AWS CloudTrail: your audit log is incomplete

Andreas Wittig – 26 Jun 2019

Recently, I was investigating the size of a security breach caused by leaked AWS credentials. The first place to go in such a scenario is the audit log recorded by CloudTrail. When configured correctly, CloudTrail captures the requests to the AWS API and stores them on S3 or forwards them to CloudWatch Logs. Analyzing the audit log allows you to answer questions like:

  • Did IAM user xyz change any parts of the cloud infrastructure within the last seven days?
  • Who has made changes to the security groups within the AWS account?
  • Did anyone start an EC2 instance in us-east-1?

AWS CloudTrail: your audit log is incomplete

So far, so good. Based on the CloudTrail logs, no one was using the leaked AWS credentials to modify the cloud infrastructure. But unfortunately, I could not answer the question, whether someone was using the AWS credentials to download sensitive data from S3.

Why not? Because of poor defaults and an unfortunate price model. By default, CloudTrail does not capture read and write requests to S3 - so-called data events. I assume that is because AWS is charging $0.10 per 100,000 data events.

I highly encourage you to check your CloudTrail configuration and enable capturing data events for all S3 buckets. Or at least for the S3 buckets, that may contain sensitive data.

Read on to learn how to configure CloudTrail with the AWS Management Console or AWS CloudFormation.


Looking for a new challenge?

  • tecRacer

    Cloud Consultant

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Spain, and Switzerland
    AWS only Infrastructure as Code EC2 Containers Serverless
  • tecRacer

    Cloud Migration Specialist

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Spain, and Switzerland
    Lift&Shift Transformation EC2 RDS VPC

AWS Management Console: enable S3 data events

  1. Open the AWS Management Console.
  2. Go to the CloudTrail service.
  3. Edit an existing trail or create a new one.
  4. Enable data events for S3 buckets, as shown in the following screenshot.

AWS Management Console: enable S3 data events

AWS CloudFormation: enable S3 data events

The following code snippet illustrates how to create a trail which stores audit logs and S3 and captures data events from all S3 buckets.

Resources:
TrailBucket:
Type: 'AWS::S3::Bucket'
Properties: {}
TrailBucketPolicy:
Type: 'AWS::S3::BucketPolicy'
Properties:
Bucket: !Ref TrailBucket
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: 'cloudtrail.amazonaws.com'
Action: 's3:GetBucketAcl'
Resource: !Sub 'arn:aws:s3:::${TrailBucket}'
- Effect: Allow
Principal:
Service: 'cloudtrail.amazonaws.com'
Action: 's3:PutObject'
Resource: !Sub 'arn:aws:s3:::${TrailBucket}/AWSLogs/${AWS::AccountId}/*'
Condition:
StringEquals:
's3:x-amz-acl': 'bucket-owner-full-control'
Trail:
DependsOn:
- TrailBucketPolicy
Type: 'AWS::CloudTrail::Trail'
Properties:
IncludeGlobalServiceEvents: true
IsLogging: true
IsMultiRegionTrail: true
EventSelectors: [
{
DataResources: [{Type: 'AWS::S3::Object', Values: ['arn:aws:s3:::']}],
IncludeManagementEvents: true,
ReadWriteType: All
}
]
S3BucketName: !Ref TrailBucket

However, keep in mind that recording data events add some extra costs.

  • CloudTrail: $0.10 per 100,000 data events
  • S3: additional write requests and storage
  • CloudWatch: additional data ingestion and storage

So please do the math, before enabling data events for high-throughput S3 buckets. And keep an eye on your AWS costs.

Summary

I highly recommend enabling capturing S3 data events with CloudTrail. Doing so allows you to find out who was reading or writing data to S3. However, keep in mind that data events add substantial extra costs for high-throughput S3 buckets.

Become a cloudonaut supporter

Andreas Wittig

Andreas Wittig ( Email, Twitter, or LinkedIn )

We launched the cloudonaut blog in 2015. Since then, we have published 345 articles, 45 podcast episodes, and 37 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Jaap-Jan Frans, Jason Yorty, Jeff Finley, Jens Gehring, jhoadley, Johannes Grumböck, John Culkin, Jonas Mellquist, Juraj Martinka, Kamil Oboril, Ken Snyder, Ross Mohan, Ross Mohan, sam onaga, Shawn Tolidano, Thorsten Hoeger, Todd Valentine, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.