A Deep Dive into AWS CloudTrail

Andreas Wittig – 18 Mar 2021

Who made changes to sensitive parts of your cloud infrastructure? Capture audit logs with AWS CloudTrail. Learn how to analyze the audit logs with the help of CloudWatch Logs Insights or Athena. On top of that, we discuss how to rollout CloudTrail to all AWS accounts belonging to your organization. Last but not least, you will learn about the blind spots and how to avoid extensive costs.

A Deep Dive into AWS CloudTrail

What to expect from the video?

  • Demo: Querying audit logs with CloudWatch
  • Demo: Querying audit logs with Athena
  • Best practices for configuring CloudTrail (multi-account)
  • About blind spots: S3, DynamoDB, SQS, SNS, …
  • About extensive costs: data events are expensive
  • Demo: Real-time alerts (CIS AWS Foundations)

Enjoy the video!

Capturing and storing audit logs is only half of the job. It would be best if you were capable of analyzing the logs as well.


Looking for a new challenge?

  • tecRacer

    Cloud Consultant

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Spain, and Switzerland
    AWS only Infrastructure as Code EC2 Containers Serverless
  • tecRacer

    Cloud Migration Specialist

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Spain, and Switzerland
    Lift&Shift Transformation EC2 RDS VPC

Querying audit logs with CloudWatch

Here are some examples of queries for CloudWatch Logs Insights.

Which regions are used within account?

fields @timestamp, @message
| stats count() by awsRegion
| sort awsRegion asc

Are IAM users used to access resources?

fields @timestamp, @message
| filter userIdentity.type = "IAMUser"
| stats count() by userIdentity.arn
| sort awsRegion asc

Who signed in through the AWS Management Console?

fields @timestamp, @message
| filter eventSource = "signin.amazonaws.com"
| stats count() by userIdentity.arn

Which IAM policies have been changed?

fields @timestamp, @message
| filter eventSource = "iam.amazonaws.com" and (eventName = "AttachGroupPolicy" or eventName = "AttachRolePolicy" or eventName = "AttachUserPolicy" or eventName = "CreatePolicy" or eventName = "CreatePolicyVersion" or eventName = "DeleteGroupPolicy" or eventName = "DeletePolicy" or eventName = "DeletePolicyVersion" or eventName = "DeleteRolePolicy" or eventName = "DeleteUserPolicy" or eventName = "DetachGroupPolicy" or eventName = "DetachRolePolicy" or eventName = "DetachUserPolicy" or eventName = "PutGroupPolicy" or eventName = "PutRolePolicy" or eventName = "PutUserPolicy" or eventName = "UpdateAssumeRolePolicy")

Detect changes to Security Groups.

fields @timestamp, @message
| filter eventSource = "ec2.amazonaws.com" and (eventName = "AuthorizeSecurityGroupEgress" or eventName = "AuthorizeSecurityGroupIngress" or eventName = "RevokeSecurityGroupEgress" or eventName = "RevokeSecurityGroupIngress")

Did anyone make use of leaked AWS credentials?

fields @timestamp, @message
| filter userIdentity.accessKeyId = 'AKIA36A2NNHBPCARNKJG'

Querying audit logs with Athena

Besides that, Athena offers a powerful way to search through audit logs captured by CloudTrail as well.

Creating the table to analyze CloudTrail logs

Make sure to replace $BUCKETNAME with the name of the S3 bucket that you are storing CloudTrail logs in. Also you need to update the list of AWS accounts near projection.account.values.

CREATE EXTERNAL TABLE cloudtrail_logs_pp(
eventVersion STRING,
userIdentity STRUCT<
type: STRING,
principalId: STRING,
arn: STRING,
accountId: STRING,
invokedBy: STRING,
accessKeyId: STRING,
userName: STRING,
sessionContext: STRUCT<
attributes: STRUCT<
mfaAuthenticated: STRING,
creationDate: STRING>,
sessionIssuer: STRUCT<
type: STRING,
principalId: STRING,
arn: STRING,
accountId: STRING,
userName: STRING>>>,
eventTime STRING,
eventSource STRING,
eventName STRING,
awsRegion STRING,
sourceIpAddress STRING,
userAgent STRING,
errorCode STRING,
errorMessage STRING,
requestParameters STRING,
responseElements STRING,
additionalEventData STRING,
requestId STRING,
eventId STRING,
readOnly STRING,
resources ARRAY<STRUCT<
arn: STRING,
accountId: STRING,
type: STRING>>,
eventType STRING,
apiVersion STRING,
recipientAccountId STRING,
serviceEventDetails STRING,
sharedEventID STRING,
vpcEndpointId STRING
)
PARTITIONED BY (
`timestamp` string,
`account` string,
`region` string)
ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://$BUCKETNAME/AWSLogs/'
TBLPROPERTIES (
'projection.enabled'='true',
'projection.timestamp.format'='yyyy/MM/dd',
'projection.timestamp.interval'='1',
'projection.timestamp.interval.unit'='DAYS',
'projection.timestamp.range'='2020/01/01,NOW',
'projection.timestamp.type'='date',
'projection.account.type'='enum',
'projection.account.values'='111111111111,222222222222',
'projection.region.type'='enum',
'projection.region.values'='eu-north-1,ap-south-1,eu-west-3,eu-west-2,eu-west-1,ap-northeast-2,ap-northeast-1,sa-east-1,ca-central-1,ap-southeast-1,ap-southeast-2,eu-central-1,us-east-1,us-east-2,us-west-1,us-west-2',
'storage.location.template'='s3://$BUCKETNAME/AWSLogs/${account}/CloudTrail/${region}/${timestamp}')

Detect changes to Security Groups.

SELECT * FROM "default"."cloudtrail_logs_pp" WHERE eventsource = 'ec2.amazonaws.com' AND eventname IN ('AuthorizeSecurityGroupEgress', 'AuthorizeSecurityGroupIngress', 'RevokeSecurityGroupEgress', 'RevokeSecurityGroupIngress') AND account = '068189904525' and timestamp > '2021/01/01' limit 50;

CloudTrail Data Events

Copy our spreadsheet to calculate costs for CloudTrail Data Events.

Realtime Alerts

Our CloudFormation template cloudtrail.yaml includes the realtime alerts as defined in the CIS AWS Foundations Benchmark.

Become a cloudonaut supporter

Andreas Wittig

Andreas Wittig ( Email, Twitter, or LinkedIn )

We launched the cloudonaut blog in 2015. Since then, we have published 347 articles, 45 podcast episodes, and 37 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Christopher Hipwell, Jaap-Jan Frans, Jason Yorty, Jeff Finley, Jens Gehring, jhoadley, Johannes Grumböck, John Culkin, Jonas Mellquist, Juraj Martinka, Kamil Oboril, Ken Snyder, Ross Mohan, Ross Mohan, sam onaga, Satyendra Sharma, Shawn Tolidano, Thorsten Hoeger, Todd Valentine, Victor Grenu, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.