Record AWS API calls to improve IAM Policies
Have you ever looked at an IAM policy and wondered: Is it really necessary to grant access to this specific action? Or do you need to know which API calls a legacy or 3rd party application is actually sending to come up with a secure IAM policy? CloudTrail can help here, but there is something better: Record API calls with the AWS SDKs and CLI (including the stuff that is not visible in CloudTrail).
Do you prefer listening to a podcast episode over reading a blog post? Here you go!
In this blog post, you learn to capture the data without touching source code. You also analyze the data and use the results to improve your IAM policies.
Record AWS API calls to improve IAM Policies in Action
Watch the following video to learn how to record AWS API calls to improve your IAM Policies.
Turning Client Side Monitoring on
All AWS SDKs and the AWS CLI support “Client Side Monitoring (CSM)”. Luckily, most applications use AWS SDKs to integrate with AWS. If you enable CSM, each API request is reported via UDP on port 31000. You can turn on CSM by setting the environment variable AWS_CSM_ENABLED
to true
or via the shared config file ~/.aws/config
:
[default] |
Warning Keep in mind that you need to add the
csm_enabled
setting for each Linux user, e.g.:/root/.aws/config
/home/ec2-user/.aws/config
Warning Keep in mind that you have to restart the process that uses an AWS SDK after changing the config!
You can check if API calls are reported with this command:
tcpdump -i lo -n udp port 31000 -A |
To debug a process where no data shows up, get the PID of the process with ps -ef
, and inspect the environment variables:
xargs -0 -L1 -a /proc/PID/environ |
If HOME
or AWS_CSM_ENABLED
are not defined, CSM will not work. If AWS_CONFIG_FILE
is defined, you have to edit that file and append csm_enabled = true
.
If the process is started by systemd
, edit the unit file (e.g., /usr/lib/systemd/system/amazon-ssm-agent.service
) and turn on AWS_CSM_ENABLED
in the [Service]
section:
[Service] |
Next, you will learn how to capture and archive the CSM data to S3. Doing so allows you to analyze the data with the help of Athena later.
Capturing the data
First, create an S3 bucket for storing the data.
fluentd can listen on a UDP port, transform and buffer the data, and upload it to S3.
On Amazon Linux 2 EC2 instance, installing fluentd
is a one-liner (other distros are supported as well):
curl -L https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent4.sh | sh |
Allow your EC2 instance to interact with the newly created bucket (replace BUCKET_NAME
with the name of your bucket) by adding the following IAM policy:
{ |
Last but not least, configure fluentd
by replacing the file /etc/td-agent/td-agent.conf
with the following content (replace BUCKET_NAME
with the name of your bucket, and REGION
with your AWS region (e.g., us-east-1)):
<source> |
Activate the new fluentd
configuration with this command:
systemctl start td-agent.service |
If data comes in, fluentd
uploads a file to S3 every 10 minutes (or every 256 MB) with a 1-minute delay.
I recommend waiting for a couple of days to capture enough data.
Analyzing the data
Create an AWS Glue Crawler that looks into s3://BUCKET_NAME/awscsm/
every hour. Run the crawler manually to speed up table creation for the first time.
After the crawler finished, switch to Athena. There you will find a table awscsm that you can query.
To get an idea of how the data looks like, run:
SELECT * FROM awscsm LIMIT 25 |
Only get API calls, but not the attempts (one call can have multiple attempts):
SELECT * FROM awscsm WHERE type='ApiCall' LIMIT 25 |
Get the most popular API calls:
SELECT service, api, COUNT(*) as count FROM awscsm WHERE type='ApiCall' GROUP by service, api ORDER BY count DESC LIMIT 25 |
Comparing with CloudTrail
I use the following CloudWatch Insights query to get the most popular API calls (replace IAM_ROLE_NAME
with the name of the IAM role attached to your EC2 instance):
fields @timestamp, @message |
The top calls look entirely different. From CSM, I get:
And from CloudTrail, I get:
As you can see, CloudTrail does not capture most of the “data” events. Unfortunately, most calls of a typical application are categorized as “data” events.
Summary
Securing IAM policies of running systems is hard. You need all available data to reduce the risk of accidentally removing permissions required by the system. CloudTrail provides a good foundation. Unfortunately, not all API calls are visible in CloudTrail. E.g., SQS “data events” are not captured by CloudTrail. Client Side Monitoring (CSM) can be used to capture the calls that are made with AWS SDKs and the AWS CLI. Both sources combined can help you to detect IAM permissions that are not needed anymore.
Thanks, Scott Piper, for bringing CSM to my attention.
Further reading
- Article Review: AWS App Mesh – A service mesh for EC2, ECS, and EKS
- Article Run the AWS CLI v2 inside Docker
- Article Messaging on AWS
- Tag iam
- Tag highlight