A Deep Dive into AWS CloudTrail

Andreas Wittig – 18 Mar 2021

Who made changes to sensitive parts of your cloud infrastructure? Capture audit logs with AWS CloudTrail. Learn how to analyze the audit logs with the help of CloudWatch Logs Insights or Athena. On top of that, we discuss how to rollout CloudTrail to all AWS accounts belonging to your organization. Last but not least, you will learn about the blind spots and how to avoid extensive costs.

A Deep Dive into AWS CloudTrail

What to expect from the video?

  • Demo: Querying audit logs with CloudWatch
  • Demo: Querying audit logs with Athena
  • Best practices for configuring CloudTrail (multi-account)
  • About blind spots: S3, DynamoDB, SQS, SNS, …
  • About extensive costs: data events are expensive
  • Demo: Real-time alerts (CIS AWS Foundations)

Enjoy the video!

Capturing and storing audit logs is only half of the job. It would be best if you were capable of analyzing the logs as well.

Andreas and Michael Wittig

Please support our work!

We have published 331 articles, 42 podcast episodes, and 15 videos. It's all free and means a lot of work in our spare time.

$6,000.00 will help us to write in-depth service reviews and comparisons in 2022! With your help, we can continue to produce independent & high-quality content focused on AWS.

$1,454.00 raised

Thanks to Alan Leech, Alex DeBrie, e9e4e5f0faef, Jaap-Jan Frans, Jens Gehring, jhoadley, Kamil Oboril, Ken Snyder, Thorsten Hoeger, Todd Valentine, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.

Support us

Querying audit logs with CloudWatch

Here are some examples of queries for CloudWatch Logs Insights.

Which regions are used within account?

fields @timestamp, @message
| stats count() by awsRegion
| sort awsRegion asc

Are IAM users used to access resources?

fields @timestamp, @message
| filter userIdentity.type = "IAMUser"
| stats count() by userIdentity.arn
| sort awsRegion asc

Who signed in through the AWS Management Console?

fields @timestamp, @message
| filter eventSource = "signin.amazonaws.com"
| stats count() by userIdentity.arn

Which IAM policies have been changed?

fields @timestamp, @message
| filter eventSource = "iam.amazonaws.com" and (eventName = "AttachGroupPolicy" or eventName = "AttachRolePolicy" or eventName = "AttachUserPolicy" or eventName = "CreatePolicy" or eventName = "CreatePolicyVersion" or eventName = "DeleteGroupPolicy" or eventName = "DeletePolicy" or eventName = "DeletePolicyVersion" or eventName = "DeleteRolePolicy" or eventName = "DeleteUserPolicy" or eventName = "DetachGroupPolicy" or eventName = "DetachRolePolicy" or eventName = "DetachUserPolicy" or eventName = "PutGroupPolicy" or eventName = "PutRolePolicy" or eventName = "PutUserPolicy" or eventName = "UpdateAssumeRolePolicy")

Detect changes to Security Groups.

fields @timestamp, @message
| filter eventSource = "ec2.amazonaws.com" and (eventName = "AuthorizeSecurityGroupEgress" or eventName = "AuthorizeSecurityGroupIngress" or eventName = "RevokeSecurityGroupEgress" or eventName = "RevokeSecurityGroupIngress")

Did anyone make use of leaked AWS credentials?

fields @timestamp, @message
| filter userIdentity.accessKeyId = 'AKIA36A2NNHBPCARNKJG'

Querying audit logs with Athena

Besides that, Athena offers a powerful way to search through audit logs captured by CloudTrail as well.

Creating the table to analyze CloudTrail logs

Make sure to replace $BUCKETNAME with the name of the S3 bucket that you are storing CloudTrail logs in. Also you need to update the list of AWS accounts near projection.account.values.

CREATE EXTERNAL TABLE cloudtrail_logs_pp(
eventVersion STRING,
userIdentity STRUCT<
type: STRING,
principalId: STRING,
arn: STRING,
accountId: STRING,
invokedBy: STRING,
accessKeyId: STRING,
userName: STRING,
sessionContext: STRUCT<
attributes: STRUCT<
mfaAuthenticated: STRING,
creationDate: STRING>,
sessionIssuer: STRUCT<
type: STRING,
principalId: STRING,
arn: STRING,
accountId: STRING,
userName: STRING>>>,
eventTime STRING,
eventSource STRING,
eventName STRING,
awsRegion STRING,
sourceIpAddress STRING,
userAgent STRING,
errorCode STRING,
errorMessage STRING,
requestParameters STRING,
responseElements STRING,
additionalEventData STRING,
requestId STRING,
eventId STRING,
readOnly STRING,
resources ARRAY<STRUCT<
arn: STRING,
accountId: STRING,
type: STRING>>,
eventType STRING,
apiVersion STRING,
recipientAccountId STRING,
serviceEventDetails STRING,
sharedEventID STRING,
vpcEndpointId STRING
)
PARTITIONED BY (
`timestamp` string,
`account` string,
`region` string)
ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://$BUCKETNAME/AWSLogs/'
TBLPROPERTIES (
'projection.enabled'='true',
'projection.timestamp.format'='yyyy/MM/dd',
'projection.timestamp.interval'='1',
'projection.timestamp.interval.unit'='DAYS',
'projection.timestamp.range'='2020/01/01,NOW',
'projection.timestamp.type'='date',
'projection.account.type'='enum',
'projection.account.values'='111111111111,222222222222',
'projection.region.type'='enum',
'projection.region.values'='eu-north-1,ap-south-1,eu-west-3,eu-west-2,eu-west-1,ap-northeast-2,ap-northeast-1,sa-east-1,ca-central-1,ap-southeast-1,ap-southeast-2,eu-central-1,us-east-1,us-east-2,us-west-1,us-west-2',
'storage.location.template'='s3://$BUCKETNAME/AWSLogs/${account}/CloudTrail/${region}/${timestamp}')

Detect changes to Security Groups.

SELECT * FROM "default"."cloudtrail_logs_pp" WHERE eventsource = 'ec2.amazonaws.com' AND eventname IN ('AuthorizeSecurityGroupEgress', 'AuthorizeSecurityGroupIngress', 'RevokeSecurityGroupEgress', 'RevokeSecurityGroupIngress') AND account = '068189904525' and timestamp > '2021/01/01' limit 50;

CloudTrail Data Events

Copy our spreadsheet to calculate costs for CloudTrail Data Events.

Realtime Alerts

Our CloudFormation template cloudtrail.yaml includes the realtime alerts as defined in the CIS AWS Foundations Benchmark.

Andreas Wittig

Andreas Wittig

I launched cloudonaut.io in 2015 with my brother Michael. Since then, we have published hundreds of articles, podcast episodes, and videos. It’s all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.
Have you learned something new by reading, listening, or watching our content? If so, we kindly ask you to support us in producing high-quality & independent AWS content. We look forward to sharing our AWS knowledge with you.

Support us

Feedback? Questions? You can reach me via Email, Twitter, or LinkedIn.