cloudonaut
Home Blog Podcast YouTube Games Books Logout
Subscribe Logout

A Deep Dive into AWS CloudTrail

Andreas Wittig – 18 Mar 2021

Who made changes to sensitive parts of your cloud infrastructure? Capture audit logs with AWS CloudTrail. Learn how to analyze the audit logs with the help of CloudWatch Logs Insights or Athena. On top of that, we discuss how to rollout CloudTrail to all AWS accounts belonging to your organization. Last but not least, you will learn about the blind spots and how to avoid extensive costs.

A Deep Dive into AWS CloudTrail

What to expect from the video?

  • Demo: Querying audit logs with CloudWatch
  • Demo: Querying audit logs with Athena
  • Best practices for configuring CloudTrail (multi-account)
  • About blind spots: S3, DynamoDB, SQS, SNS, …
  • About extensive costs: data events are expensive
  • Demo: Real-time alerts (CIS AWS Foundations)

Enjoy the video!

Capturing and storing audit logs is only half of the job. It would be best if you were capable of analyzing the logs as well.

Querying audit logs with CloudWatch

Here are some examples of queries for CloudWatch Logs Insights.

Which regions are used within account?

fields @timestamp, @message
| stats count() by awsRegion
| sort awsRegion asc

Are IAM users used to access resources?

fields @timestamp, @message
| filter userIdentity.type = "IAMUser"
| stats count() by userIdentity.arn
| sort awsRegion asc

Who signed in through the AWS Management Console?

fields @timestamp, @message
| filter eventSource = "signin.amazonaws.com"
| stats count() by userIdentity.arn

Which IAM policies have been changed?

fields @timestamp, @message
| filter eventSource = "iam.amazonaws.com" and (eventName = "AttachGroupPolicy" or eventName = "AttachRolePolicy" or eventName = "AttachUserPolicy" or eventName = "CreatePolicy" or eventName = "CreatePolicyVersion" or eventName = "DeleteGroupPolicy" or eventName = "DeletePolicy" or eventName = "DeletePolicyVersion" or eventName = "DeleteRolePolicy" or eventName = "DeleteUserPolicy" or eventName = "DetachGroupPolicy" or eventName = "DetachRolePolicy" or eventName = "DetachUserPolicy" or eventName = "PutGroupPolicy" or eventName = "PutRolePolicy" or eventName = "PutUserPolicy" or eventName = "UpdateAssumeRolePolicy")

Detect changes to Security Groups.

fields @timestamp, @message
| filter eventSource = "ec2.amazonaws.com" and (eventName = "AuthorizeSecurityGroupEgress" or eventName = "AuthorizeSecurityGroupIngress" or eventName = "RevokeSecurityGroupEgress" or eventName = "RevokeSecurityGroupIngress")

Did anyone make use of leaked AWS credentials?

fields @timestamp, @message
| filter userIdentity.accessKeyId = 'AKIA36A2NNHBPCARNKJG'

Querying audit logs with Athena

Besides that, Athena offers a powerful way to search through audit logs captured by CloudTrail as well.

Creating the table to analyze CloudTrail logs

Make sure to replace $BUCKETNAME with the name of the S3 bucket that you are storing CloudTrail logs in. Also you need to update the list of AWS accounts near projection.account.values.

CREATE EXTERNAL TABLE cloudtrail_logs_pp(
    eventVersion STRING,
    userIdentity STRUCT<
        type: STRING,
        principalId: STRING,
        arn: STRING,
        accountId: STRING,
        invokedBy: STRING,
        accessKeyId: STRING,
        userName: STRING,
        sessionContext: STRUCT<
            attributes: STRUCT<
                mfaAuthenticated: STRING,
                creationDate: STRING>,
            sessionIssuer: STRUCT<
                type: STRING,
                principalId: STRING,
                arn: STRING,
                accountId: STRING,
                userName: STRING>>>,
    eventTime STRING,
    eventSource STRING,
    eventName STRING,
    awsRegion STRING,
    sourceIpAddress STRING,
    userAgent STRING,
    errorCode STRING,
    errorMessage STRING,
    requestParameters STRING,
    responseElements STRING,
    additionalEventData STRING,
    requestId STRING,
    eventId STRING,
    readOnly STRING,
    resources ARRAY<STRUCT<
        arn: STRING,
        accountId: STRING,
        type: STRING>>,
    eventType STRING,
    apiVersion STRING,
    recipientAccountId STRING,
    serviceEventDetails STRING,
    sharedEventID STRING,
    vpcEndpointId STRING
  )
PARTITIONED BY (
   `timestamp` string,
   `account` string,
   `region` string)
ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  's3://$BUCKETNAME/AWSLogs/'
TBLPROPERTIES (
  'projection.enabled'='true', 
  'projection.timestamp.format'='yyyy/MM/dd', 
  'projection.timestamp.interval'='1', 
  'projection.timestamp.interval.unit'='DAYS', 
  'projection.timestamp.range'='2020/01/01,NOW', 
  'projection.timestamp.type'='date',
  'projection.account.type'='enum',
  'projection.account.values'='111111111111,222222222222',
  'projection.region.type'='enum',
  'projection.region.values'='eu-north-1,ap-south-1,eu-west-3,eu-west-2,eu-west-1,ap-northeast-2,ap-northeast-1,sa-east-1,ca-central-1,ap-southeast-1,ap-southeast-2,eu-central-1,us-east-1,us-east-2,us-west-1,us-west-2',
  'storage.location.template'='s3://$BUCKETNAME/AWSLogs/${account}/CloudTrail/${region}/${timestamp}')

Detect changes to Security Groups.

SELECT * FROM "default"."cloudtrail_logs_pp" WHERE eventsource = 'ec2.amazonaws.com' AND eventname IN ('AuthorizeSecurityGroupEgress', 'AuthorizeSecurityGroupIngress', 'RevokeSecurityGroupEgress', 'RevokeSecurityGroupIngress') AND account = '068189904525' and timestamp > '2021/01/01' limit 50;

CloudTrail Data Events

Copy our spreadsheet to calculate costs for CloudTrail Data Events.

Realtime Alerts

Our CloudFormation template cloudtrail.yaml includes the realtime alerts as defined in the CIS AWS Foundations Benchmark.

Andreas Wittig

Andreas Wittig

I’ve been building on AWS since 2012 together with my brother Michael. We are sharing our insights into all things AWS on cloudonaut and have written the book AWS in Action. Besides that, we’re currently working on bucketAV,HyperEnv for GitHub Actions, and marbot.

Here are the contact options for feedback and questions.

Further reading