Record AWS API calls to improve IAM Policies
Have you ever looked at an IAM policy and wondered: Is it really necessary to grant access to this specific action? Or do you need to know which API calls a legacy or 3rd party application is actually sending to come up with a secure IAM policy? CloudTrail can help here, but there is something better: Record API calls with the AWS SDKs and CLI (including the stuff that is not visible in CloudTrail).
Do you prefer listening to a podcast episode over reading a blog post? Here you go!
In this blog post, you learn to capture the data without touching source code. You also analyze the data and use the results to improve your IAM policies.
All AWS SDKs and the AWS CLI support “Client Side Monitoring (CSM)”. Luckily, most applications use AWS SDKs to integrate with AWS. If you enable CSM, each API request is reported via UDP on port 31000. You can turn on CSM by setting the environment variable
true or via the shared config file
Warning Keep in mind that you need to add the
csm_enabledsetting for each Linux user, e.g.:
Warning Keep in mind that you have to restart the process that uses an AWS SDK after changing the config!
You can check if API calls are reported with this command:
tcpdump -i lo -n udp port 31000 -A
To debug a process where no data shows up, get the PID of the process with
ps -ef, and inspect the environment variables:
xargs -0 -L1 -a /proc/PID/environ
AWS_CSM_ENABLED are not defined, CSM will not work. If
AWS_CONFIG_FILE is defined, you have to edit that file and append
csm_enabled = true.
If the process is started by
systemd, edit the unit file (e.g.,
/usr/lib/systemd/system/amazon-ssm-agent.service) and turn on
AWS_CSM_ENABLED in the
Next, you will learn how to capture and archive the CSM data to S3. Doing so allows you to analyze the data with the help of Athena later.
First, create an S3 bucket for storing the data.
fluentd can listen on a UDP port, transform and buffer the data, and upload it to S3.
On Amazon Linux 2 EC2 instance, installing
fluentd is a one-liner (other distros are supported as well):
curl -L https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent4.sh | sh
Allow your EC2 instance to interact with the newly created bucket (replace
BUCKET_NAME with the name of your bucket) by adding the following IAM policy:
Last but not least, configure
fluentd by replacing the file
/etc/td-agent/td-agent.conf with the following content (replace
BUCKET_NAME with the name of your bucket, and
REGION with your AWS region (e.g., us-east-1)):
Activate the new
fluentd configuration with this command:
systemctl start td-agent.service
If data comes in,
fluentd uploads a file to S3 every 10 minutes (or every 256 MB) with a 1-minute delay.
Level up, strengthen your AWS skills.
Our book Amazon Web Services in Action is a comprehensive introduction to computing, storing, and networking in the AWS cloud. You'll find clear, relevant coverage of all the essential AWS services, emphasizing best practices for security, high availability, and scalability. Get the first chapter for free!
I recommend waiting for a couple of days to capture enough data.
Create an AWS Glue Crawler that looks into
s3://BUCKET_NAME/awscsm/ every hour. Run the crawler manually to speed up table creation for the first time.
After the crawler finished, switch to Athena. There you will find a table awscsm that you can query.
To get an idea of how the data looks like, run:
SELECT * FROM awscsm LIMIT 25
Only get API calls, but not the attempts (one call can have multiple attempts):
SELECT * FROM awscsm WHERE type='ApiCall' LIMIT 25
Get the most popular API calls:
SELECT service, api, COUNT(*) as count FROM awscsm WHERE type='ApiCall' GROUP by service, api ORDER BY count DESC LIMIT 25
I use the following CloudWatch Insights query to get the most popular API calls (replace
IAM_ROLE_NAME with the name of the IAM role attached to your EC2 instance):
fields @timestamp, @message
The top calls look entirely different. From CSM, I get:
And from CloudTrail, I get:
As you can see, CloudTrail does not capture most of the “data” events. Unfortunately, most calls of a typical application are categorized as “data” events.
Securing IAM policies of running systems is hard. You need all available data to reduce the risk of accidentally removing permissions required by the system. CloudTrail provides a good foundation. Unfortunately, not all API calls are visible in CloudTrail. E.g., SQS “data events” are not captured by CloudTrail. Client Side Monitoring (CSM) can be used to capture the calls that are made with AWS SDKs and the AWS CLI. Both sources combined can help you to detect IAM permissions that are not needed anymore.
Thanks, Scott Piper, for bringing CSM to my attention.