Monitor your AWS account to detect suspicious behavior in real time

You can track every change made to your AWS account with CloudTrail. Did you know that you can also monitor your AWS account in near real time with custom rules specific to your use case?

By combining CloudTrail, S3, SNS, and Lambda, you can run a piece of code to check the API activity in your account. Because of the reporting frequency of CloudTrail, this will happen approximately every 5 minutes. This post explains how to deploy a solution to monitor your EC2 instance tags for suspicious behavior.

The following figure shows how this works on a high level.

Let’s look at a concrete example.

What is suspicious behavior?

CloudTrail is recording a lot of API activity. Your job is to determine which activities are suspicious. Here are a few ideas:

• A security group was changed to open a port to the outside world (0.0.0.0/0).
• An IAM user was created outside of regular business hours.
• An EC2 instance was started without following your company’s tag schema (for example, you may mark technical ownership, cost ownership, and so on).

The example that follows implements the idea of EC2 instance tag monitoring.

Monitoring EC2 instance tags

Each time CloudTrail has new data for you, a Lambda function is triggered. The Lambda function needs to do the following:

1. Understand the input data generated from SNS.
2. Download the compressed CloudTrail files from an S3 bucket.
3. Uncompress the files.
4. Iterate through the API activities, looking for EC2 tag-related events: RunInstances, CreateTags, and DeleteTags.
5. Alert violations of the tag schema.

Fortunately, the code is implemented already so that we won’t dive into Node.js code this time. Instead, we’ll focus on deploying this solution.

Deploying the solution

Deploying Lambda is possible almost entirely with CloudFormation. A few steps are required to prepare everything you need:

1. Choose an AWS region you want to monitor (referenced as $region in the following).
2. Create an SNS topic in $region, and subscribe to the topic via email. The system will send alerts to this endpoint.
3. Download the code by running wget https://github.com/widdix/aws-tag-watch/archive/master.zip in your terminal.
4. Run unzip master.zip in your terminal.
5. Change dir by running cd aws-tag-watch-master/.
6. Run npm install in your terminal to install Node.js dependencies.
7. Edit config.json, and set region to $region and alertTopicArn to the ARN of your SNS topic from step 1.
8. Execute ./bundle.sh in your console.
9. Upload aws-tag-watch.zip to S3 (the bucket must be in $region).
10. Create a CloudFormation stack based on template.json.

Now your AWS account in $region is monitored. Whenever you run a new EC2 instance or change the tags of an existing EC2 instance, the Lambda function will check whether you’re sticking to the tag schema.

Room for improvement

Raising an alert via email isn’t that helpful if you are working on a team. You may want to look at OpsGenie, which integrates nicely with SNS.

This blog post has been translated into German: Beobachte deinen AWS Account in echtzeit um verdächtige Aktivitäten aufzuspüren.