Event Driven Security Automation on AWS

If your developers create their own infrastructure in minutes, who cares about security? In some organizations the security teams become the new bottleneck if the company wants to deploy every commit to production. Because manual reviews can’t be done for every commit. Teri Radichel’s paper Balancing Security and Innovation With Event Driven Automation shows how you can use AWS to tackle the security bottleneck problem.

I will shortly summarize her work and encourage you to read the whole paper and have a look at the running example on GitHub to get a better understanding of the idea in practice.

Integration of Security Automation

Teri mentions three integration points:

Blocking Before Deployment will prevent a “known bad” from executing in the environment. To automate blocking non-compliant actions requires a mechanism for tracking deployment events and injection of security responses into the deployment process.

Audit and Respond to What Was Deployed and automatically roll back changes that were not approved or compliant. This approach will reduce the amount of time the problem exists in the environment. The problem with this approach is the window of time in which the noncompliant changes exist in the environment as it can be used to exploit systems. It would be better to prohibit a “known bad” from entering the environment in the first place.

The Unknowns: Automated Intrusion Detection and Response Unknown security problems can be addressed by a security review that may delay projects and put security teams at odds with innovators. Rather than attempting to block unknowns from entering the environment, security teams can consider focusing on automated intrusion detection and response.

AWS Support for Security Automation

Teri says:

Amazon’s AWS cloud platform offers many of the requirements for event driven
security. Although a security team can automate compliance, intrusion detection, and response in any environment, AWS provides a platform where much of the heavy lifting is already done freeing security teams to focus on implementing security rules instead of generic system and platform components. Almost every type of resource that can be created on AWS can be instantiated using web service calls via code, instead of a human manually taking some action.

In my own words: AWS records Account activity in multiple ways:

• CloudTrail records every API call.
• VPC Flow Logs track if network traffic was allowed or denied by Security Groups or NACLs.
• S3, CloudFront, and ELB Access Logs contain every call made to this services from the public

You can analyze this data in near real-time with Lambda and respond based on rules you define to audit “known bad”.

The response should both prevent further damage but also maintain evidence to determine the root cause of the problem.

The following example demonstrates Event Driven Security Automation on AWS.

An example: Bad network activity results in termination of the EC2 instance

Teri’s example demonstrates three different rules:

• A REJECT in the logs will trigger the code to snapshot the instance and terminate it. A REJECT indicates that the instance-initiated traffic is not expected or acceptable.

• Outbound traffic on unexpected ports is logged. This traffic is not currently terminated because more analysis would be needed to determine which traffic is invalid or valid. Once this termination was made, instances exhibiting invalid traffic patterns could also be terminated.
• If an instance contacts a bad IP address in the Alert Logic top 10 list, a snapshot will be made and it will be terminated. The code uses the top 10 IP addresses sent
  to the Alert Logic mailing list in February 2016 to flag bad IP addresses. This could be handled differently by pulling it from a database, a configuration file, or some other means of matching traffic against a known bad IP list.

Her implementation uses a honeypot EC2 instance as the victim of the attack. The EC2 instance runs in one VPC with Flow Logs turned on. Flow Logs send traffic to CloudWatch Logs. A Lambda function is connected to the Logs by using a subscription. If the Lambda function find’s malicious traffic it takes a snapshot of the EC2 instance and terminates it.

Thinking ahead: Automated Blocking Before Deployment

I wrote a small tool called aws-cf-checker to do compliance checks on CloudFormation templates before they are deployed. Combined with the idea that I introduced in a previous post CloudFormation vs Engineers: How to protect your CloudFormation managed AWS account from human intervention you can even enforce this process inside AWS.