Record AWS API calls to improve IAM Policies

Michael Wittig – 11 Sep 2020

Have you ever looked at an IAM policy and wondered: Is it really necessary to grant access to this specific action? Or do you need to know which API calls a legacy or 3rd party application is actually sending to come up with a secure IAM policy? CloudTrail can help here, but there is something better: Record API calls with the AWS SDKs and CLI (including the stuff that is not visible in CloudTrail).

Record AWS API calls

Do you prefer listening to a podcast episode over reading a blog post? Here you go!

In this blog post, you learn to capture the data without touching source code. You also analyze the data and use the results to improve your IAM policies.

Turning Client Side Monitoring on

All AWS SDKs and the AWS CLI support “Client Side Monitoring (CSM)”. Luckily, most applications use AWS SDKs to integrate with AWS. If you enable CSM, each API request is reported via UDP on port 31000. You can turn on CSM by setting the environment variable AWS_CSM_ENABLED to true or via the shared config file ~/.aws/config:

[default]
csm_enabled = true

[profile profile1]
csm_enabled = true

Warning Keep in mind that you need to add the csm_enabled setting for each Linux user, e.g.:
/root/.aws/config
/home/ec2-user/.aws/config

Warning Keep in mind that you have to restart the process that uses an AWS SDK after changing the config!

You can check if API calls are reported with this command:

tcpdump -i lo -n udp port 31000 -A

To debug a process where no data shows up, get the PID of the process with ps -ef, and inspect the environment variables:

xargs -0 -L1 -a /proc/PID/environ

If HOME or AWS_CSM_ENABLED are not defined, CSM will not work. If AWS_CONFIG_FILE is defined, you have to edit that file and append csm_enabled = true.

If the process is started by systemd, edit the unit file (e.g., /usr/lib/systemd/system/amazon-ssm-agent.service) and turn on AWS_CSM_ENABLED in the [Service] section:

[Service]
Environment=AWS_CSM_ENABLED=true

Next, you will learn how to capture and archive the CSM data to S3. Doing so allows you to analyze the data with the help of Athena later.

Capturing the data

First, create an S3 bucket for storing the data.

fluentd can listen on a UDP port, transform and buffer the data, and upload it to S3.

On Amazon Linux 2 EC2 instance, installing fluentd is a one-liner (other distros are supported as well):

curl -L https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent4.sh | sh

Allow your EC2 instance to interact with the newly created bucket (replace BUCKET_NAME with the name of your bucket) by adding the following IAM policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::BUCKET_NAME/awscsm/*"
},
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::BUCKET_NAME"
}
]
}

Last but not least, configure fluentd by replacing the file /etc/td-agent/td-agent.conf with the following content (replace BUCKET_NAME with the name of your bucket, and REGION with your AWS region (e.g., us-east-1)):

<source>
@type udp
tag awscsm
<parse>
@type json
</parse>
port 31000
bind 127.0.0.1
</source>
<filter awscsm>
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
</record>
</filter>
<match awscsm>
@type s3
s3_region REGION
s3_bucket BUCKET_NAME
check_apikey_on_start false
check_bucket false
path ${tag}/year=%Y/month=%m/day=%d/
<buffer tag,time>
@type memory
timekey 10m
timekey_use_utc true
timekey_wait 1m
chunk_limit_size 256m
</buffer>
<format>
@type json
</format>
</match>

Activate the new fluentd configuration with this command:

systemctl start td-agent.service

If data comes in, fluentd uploads a file to S3 every 10 minutes (or every 256 MB) with a 1-minute delay.

Cover of Amazon Web Services in Action

Level up, strengthen your AWS skills.

Our book Amazon Web Services in Action is a comprehensive introduction to computing, storing, and networking in the AWS cloud. You'll find clear, relevant coverage of all the essential AWS services, emphasizing best practices for security, high availability, and scalability. Get the first chapter for free!

I recommend waiting for a couple of days to capture enough data.

Analyzing the data

Create an AWS Glue Crawler that looks into s3://BUCKET_NAME/awscsm/ every hour. Run the crawler manually to speed up table creation for the first time.

After the crawler finished, switch to Athena. There you will find a table awscsm that you can query.

To get an idea of how the data looks like, run:

SELECT * FROM awscsm LIMIT 25

Only get API calls, but not the attempts (one call can have multiple attempts):

SELECT * FROM awscsm WHERE type='ApiCall' LIMIT 25

Get the most popular API calls:

SELECT service, api, COUNT(*) as count FROM awscsm WHERE type='ApiCall' GROUP by service, api ORDER BY count DESC LIMIT 25

Comparing with CloudTrail

I use the following CloudWatch Insights query to get the most popular API calls (replace IAM_ROLE_NAME with the name of the IAM role attached to your EC2 instance):

fields @timestamp, @message
| filter userIdentity.arn like "IAM_ROLE_NAME"
| stats count() as count by eventSource, eventName
| sort count desc
| limit 25

The top calls look entirely different. From CSM, I get:

API calls captured with CSM

And from CloudTrail, I get:

API calls captured with CloudTrail

As you can see, CloudTrail does not capture most of the “data” events. Unfortunately, most calls of a typical application are categorized as “data” events.

Summary

Securing IAM policies of running systems is hard. You need all available data to reduce the risk of accidentally removing permissions required by the system. CloudTrail provides a good foundation. Unfortunately, not all API calls are visible in CloudTrail. E.g., SQS “data events” are not captured by CloudTrail. Client Side Monitoring (CSM) can be used to capture the calls that are made with AWS SDKs and the AWS CLI. Both sources combined can help you to detect IAM permissions that are not needed anymore.

Thanks, Scott Piper, for bringing CSM to my attention.

Tags: aws iam csm
Michael Wittig

Michael Wittig

I’m an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I’m writing this blog and all other projects together with my brother Andreas.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

You can contact me via Email, Twitter, and LinkedIn.

Briefcase icon
Hire me