👉 AWS Debug Games (Beta) - Prove your AWS expertise by solving tricky challenges.

👉 AWS Debug Games - Prove your AWS expertise.

Event Driven Security Automation on AWS

Michael Wittig – 04 Apr 2016

If your developers create their own infrastructure in minutes, who cares about security? In some organizations the security teams become the new bottleneck if the company wants to deploy every commit to production. Because manual reviews can’t be done for every commit. Teri Radichel’s paper Balancing Security and Innovation With Event Driven Automation shows how you can use AWS to tackle the security bottleneck problem.

I will shortly summarize her work and encourage you to read the whole paper and have a look at the running example on GitHub to get a better understanding of the idea in practice.

Integration of Security Automation

Teri mentions three integration points:

Blocking Before Deployment will prevent a “known bad” from executing in the environment. To automate blocking non-compliant actions requires a mechanism for tracking deployment events and injection of security responses into the deployment process.

Audit and Respond to What Was Deployed and automatically roll back changes that were not approved or compliant. This approach will reduce the amount of time the problem exists in the environment. The problem with this approach is the window of time in which the noncompliant changes exist in the environment as it can be used to exploit systems. It would be better to prohibit a “known bad” from entering the environment in the first place.

The Unknowns: Automated Intrusion Detection and Response Unknown security problems can be addressed by a security review that may delay projects and put security teams at odds with innovators. Rather than attempting to block unknowns from entering the environment, security teams can consider focusing on automated intrusion detection and response.

AWS Support for Security Automation

Teri says:

Amazon’s AWS cloud platform offers many of the requirements for event driven
security. Although a security team can automate compliance, intrusion detection, and response in any environment, AWS provides a platform where much of the heavy lifting is already done freeing security teams to focus on implementing security rules instead of generic system and platform components. Almost every type of resource that can be created on AWS can be instantiated using web service calls via code, instead of a human manually taking some action.

Looking for a new challenge?


    Cloud Operations Lead

    DEMICON • AWS Advanced Consulting Partner • Remote (Europe)
    service-delivery-management hiring devops platform

In my own words: AWS records Account activity in multiple ways:

  • CloudTrail records every API call.
  • VPC Flow Logs track if network traffic was allowed or denied by Security Groups or NACLs.
  • S3, CloudFront, and ELB Access Logs contain every call made to this services from the public

You can analyze this data in near real-time with Lambda and respond based on rules you define to audit “known bad”.

The response should both prevent further damage but also maintain evidence to determine the root cause of the problem.

The following example demonstrates Event Driven Security Automation on AWS.

An example: Bad network activity results in termination of the EC2 instance

Teri’s example demonstrates three different rules:

  • A REJECT in the logs will trigger the code to snapshot the instance and terminate it. A REJECT indicates that the instance-initiated traffic is not expected or acceptable.
  • Outbound traffic on unexpected ports is logged. This traffic is not currently terminated because more analysis would be needed to determine which traffic is invalid or valid. Once this termination was made, instances exhibiting invalid traffic patterns could also be terminated.
  • If an instance contacts a bad IP address in the Alert Logic top 10 list, a snapshot will be made and it will be terminated. The code uses the top 10 IP addresses sent
    to the Alert Logic mailing list in February 2016 to flag bad IP addresses. This could be handled differently by pulling it from a database, a configuration file, or some other means of matching traffic against a known bad IP list.

Her implementation uses a honeypot EC2 instance as the victim of the attack. The EC2 instance runs in one VPC with Flow Logs turned on. Flow Logs send traffic to CloudWatch Logs. A Lambda function is connected to the Logs by using a subscription. If the Lambda function find’s malicious traffic it takes a snapshot of the EC2 instance and terminates it.

Thinking ahead: Automated Blocking Before Deployment

I wrote a small tool called aws-cf-checker to do compliance checks on CloudFormation templates before they are deployed. Combined with the idea that I introduced in a previous post CloudFormation vs Engineers: How to protect your CloudFormation managed AWS account from human intervention you can even enforce this process inside AWS.

Become a cloudonaut supporter

Michael Wittig

Michael Wittig ( Email Twitter LinkedIn Mastodon )

We launched the cloudonaut blog in 2015. Since then, we have published 365 articles, 67 podcast episodes, and 67 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, Christopher Hipwell, e9e4e5f0faef, Jason Yorty, Jeff Finley, jhoadley, Johannes Konings, John Culkin, Jonathan Deamer, Juraj Martinka, Ken Snyder, Markus Ellers, Oriol Rodriguez, Ross Mohan, sam onaga, Satyendra Sharma, Simon Devlin, Todd Valentine, Victor Grenu, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.