📣 Limited offer: subscribe to cloudonaut plus, get a t-shirt for free

📣 Limited offer: free cloudonaut t-shirt

Monitor your AWS account to detect suspicious behavior in real time

Michael Wittig – 23 Aug 2015 (updated 30 Aug 2016)

You can track every change made to your AWS account with CloudTrail. Did you know that you can also monitor your AWS account in near real time with custom rules specific to your use case?

By combining CloudTrail, S3, SNS, and Lambda, you can run a piece of code to check the API activity in your account. Because of the reporting frequency of CloudTrail, this will happen approximately every 5 minutes. This post explains how to deploy a solution to monitor your EC2 instance tags for suspicious behavior.

The following figure shows how this works on a high level.

Monitor CloudTrail with Lambda

Let’s look at a concrete example.

What is suspicious behavior?

CloudTrail is recording a lot of API activity. Your job is to determine which activities are suspicious. Here are a few ideas:

  • A security group was changed to open a port to the outside world (0.0.0.0/0).
  • An IAM user was created outside of regular business hours.
  • An EC2 instance was started without following your company’s tag schema (for example, you may mark technical ownership, cost ownership, and so on).

The example that follows implements the idea of EC2 instance tag monitoring.

Special offer: cloudonaut t-shirt

Do you love our blog posts and podcast episodes? Unlock our weekly videos and online events by subscribing to cloudonaut plus.

Special offer: Join cloudonaut plus before November 30th, and we will send you a cloudonaut t-shirt for free.

Subscribe now!

Monitoring EC2 instance tags

Each time CloudTrail has new data for you, a Lambda function is triggered. The Lambda function needs to do the following:

  1. Understand the input data generated from SNS.
  2. Download the compressed CloudTrail files from an S3 bucket.
  3. Uncompress the files.
  4. Iterate through the API activities, looking for EC2 tag-related events: RunInstances, CreateTags, and DeleteTags.
  5. Alert violations of the tag schema.

Fortunately, the code is implemented already so that we won’t dive into Node.js code this time. Instead, we’ll focus on deploying this solution.

Deploying the solution

Deploying Lambda is possible almost entirely with CloudFormation. A few steps are required to prepare everything you need:

  1. Choose an AWS region you want to monitor (referenced as $region in the following).
  2. Create an SNS topic in $region, and subscribe to the topic via email. The system will send alerts to this endpoint.
  3. Download the code by running wget https://github.com/widdix/aws-tag-watch/archive/master.zip in your terminal.
  4. Run unzip master.zip in your terminal.
  5. Change dir by running cd aws-tag-watch-master/.
  6. Run npm install in your terminal to install Node.js dependencies.
  7. Edit config.json, and set region to $region and alertTopicArn to the ARN of your SNS topic from step 1.
  8. Execute ./bundle.sh in your console.
  9. Upload aws-tag-watch.zip to S3 (the bucket must be in $region).
  10. Create a CloudFormation stack based on template.json.

Now your AWS account in $region is monitored. Whenever you run a new EC2 instance or change the tags of an existing EC2 instance, the Lambda function will check whether you’re sticking to the tag schema.

Room for improvement

Raising an alert via email isn’t that helpful if you are working on a team. You may want to look at OpsGenie, which integrates nicely with SNS.

This blog post has been translated into German: Beobachte deinen AWS Account in echtzeit um verdächtige Aktivitäten aufzuspüren.

Michael Wittig

Michael Wittig

I’m an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I’m writing this blog and all other projects together with my brother Andreas.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

You can contact me via Email, Twitter, and LinkedIn.

Briefcase icon
Hire me