cloudonaut
Home Blog Podcast YouTube Books Logout
Subscribe Logout

AWS Velocity Series: Containerized ECS based app infrastructure

Michael Wittig – 27 Mar 2017

EC2 Container Service (ECS) is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. To run an application on ECS, you need the following components:

  • Docker image
  • ECS cluster: EC2 instances running Docker and the ECS agent
  • ECS service: Managed Docker containers on the ECS cluster

ECS provides the ability to run a production-ready application on EC2 with reduced responsibilities and increased deployment speed compared to the EC2 based approach. By production-ready, I mean:

  • Highly available: no single point of failure
  • Scalable: increase or decrease the number of instances/containers based on load
  • Frictionless deployment: deliver new versions of your application automatically without downtime
  • Secure: patching operating systems and libraries frequently, follow the least privilege principle in all areas
  • Operations: provide tools like logging, monitoring and alerting to recognize and debug problems

The overall architecture will consist of a load balancer, forwarding requests to containers running on multiple EC2 instances, distributed among different availability zones (data centers).

ECS based app architecture

The diagram was created with Cloudcraft - Visualize your cloud architecture like a pro.

AWS Velocity Cover

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Let’s start with the needed infrastructure. The ECS cluster and the service.

ECS cluster

The ECS cluster is a fleet of EC2 instances with the ECS agent and Docker installed. The ECS cluster is responsible for scheduling the work (containers) to the EC2 instances.

ECS cluster architecture

The diagram was created with Cloudcraft - Visualize your cloud architecture like a pro.

Your Docker containers will run on those EC2 instances. You don’t need to care about the ECS cluster too much. We provide you a free and production-ready CloudFormation template. Please setup the ECS cluster now if you want to setup the scenario. AWS charges will likely occur! The ECS cluster need to run in a VPC, so if you don’t have a VPC stack based of our Free Templates for AWS CloudFormation (https://github.com/widdix/aws-cf-templates/tree/master/vpc) create a VPC stack first.

This first step was easy. Now you learn to use the ECS cluster with ECS services.

ECS service

The ECS service is responsible for launching Docker containers in the cluster. The service also makes sure that failed containers are replaced, and it also takes care about performing rolling updates for you. You also need a load balancer to route traffic to the containers. The following ECS service template is based on free and production-ready CloudFormation template.

Load balancer

You can follow step by step or get the full source code here: https://github.com/widdix/aws-velocity

Create a file infrastructure/ecs.yml. The first part of the file contains the load balancer. To fully describe an Application Load Balancer, you need:

  • A Security Group that allows traffic on port 80
  • The lApplication Load Balancer itself
  • A Target Group, which is a group of EC2 instances. Each of them runs containers that can receive traffic from the load balancer
  • A Listener, which wires the load balancer together with the target group and defines the listening port

Watch out for comments with more detailed information in the code.

infrastructure/ecs.ymlGitHub
---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'ECS: service that runs on an ECS cluster based on ecs/cluster.yaml and uses a dedicated ALB, a cloudonaut.io template'
Parameters:
  # You can reuse a VPC for multiple applications. In this case, we use one of our Free Templates for AWS CloudFormation (https://github.com/widdix/aws-cf-templates/tree/master/vpc).
  ParentVPCStack:
    Description: 'Stack name of parent VPC stack based on vpc/vpc-*azs.yaml template.'
    Type: String
  ParentClusterStack:
    Description: 'Stack name of parent Cluster stack based on ecs/cluster.yaml template.'
    Type: String
Resources:
  # Allow traffic from the load balancer to the EC2 instances in the cluster. This is only necessary because we reuse the cluster template in the way it is!
  SecurityGroupInALB:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      GroupId:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-SecurityGroup'
      IpProtocol: tcp
      FromPort: 0
      ToPort: 65535
      SourceSecurityGroupId: !Ref ALBSecurityGroup
  # The load balancer accepts HTTP traffic. Therefore the firewall must allow incoming traffic on port 80.
  ALBSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: 'ecs-cluster-alb'
      VpcId:
        'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'
      SecurityGroupIngress:
      - CidrIp: '0.0.0.0/0'
        FromPort: 80
        ToPort: 80
        IpProtocol: tcp
  # The load balancer needs to run in public subnets because our users should be able to access the app from the Internet.
  LoadBalancer:
    Type: 'AWS::ElasticLoadBalancingV2::LoadBalancer'
    Properties:
      Scheme: 'internet-facing'
      SecurityGroups:
      - !Ref ALBSecurityGroup
      Subnets: 
      - 'Fn::ImportValue': !Sub '${ParentVPCStack}-SubnetAPublic'
      - 'Fn::ImportValue': !Sub '${ParentVPCStack}-SubnetBPublic'
  # A target group groups a bunch of backend instances that receive traffic from the load balancer. the health check ensures that only working backends are used.
  DefaultTargetGroup:
    Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
    Properties:
      HealthCheckIntervalSeconds: 15
      HealthCheckPath: '/1'
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 10
      HealthyThresholdCount: 2
      Matcher:
        HttpCode: '200'
      Port: 80
      Protocol: HTTP
      UnhealthyThresholdCount: 4
      VpcId:
        'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'
  # The load balancer should listen on port 80 for HTTP traffic
  HttpListener:
    Type: 'AWS::ElasticLoadBalancingV2::Listener'
    Properties:
      DefaultActions:
      - TargetGroupArn: !Ref DefaultTargetGroup
        Type: forward
      LoadBalancerArn: !Ref LoadBalancer
      Port: 80
      Protocol: HTTP
# A CloudFormation stack can return information that is needed by other stacks or scripts.
Outputs:
  DNSName:
    Description: 'The DNS name for the ECS cluster/service load balancer.'
    Value: !GetAtt 'LoadBalancer.DNSName'
    Export:
      Name: !Sub '${AWS::StackName}-DNSName'
  URL:
    Description: 'URL to the ECS service.'
    Value: !Sub 'http://${LoadBalancer.DNSName}'
    Export:
      Name: !Sub '${AWS::StackName}-URL'

But how do you get notified if something goes wrong? Let’s add a parameter to the Parameters section to make the receiver configurable:

infrastructure/ecs.ymlGitHub
AdminEmail:
  Description: 'The email address of the admin who receives alerts.'
  Type: String

Alerts are triggered by a CloudWatch Alarm which can send an alert to an SNS topic. You can subscribe to this topic via an email address to receive the alerts. Let’s create an SNS topic and two alarms in the Resources section:

infrastructure/ecs.ymlGitHub
# A SNS topic is used to send alerts via Email to the value of the AdminEmail parameter 
  Alerts:
    Type: 'AWS::SNS::Topic'
    Properties:
      Subscription:
      - Endpoint: !Ref AdminEmail
        Protocol: email
# This alarm is triggered, if the load balancer responds with 5XX status codes
  LoadBalancer5XXAlarm:
    Type: 'AWS::CloudWatch::Alarm'
    Properties:
      EvaluationPeriods: 1
      Statistic: Sum
      Threshold: 0
      AlarmDescription: 'Load balancer responds with 5XX status codes.'
      Period: 60
      AlarmActions:
      - !Ref Alerts
      Namespace: 'AWS/ApplicationELB'
      Dimensions:
      - Name: LoadBalancer
        Value: !GetAtt 'LoadBalancer.LoadBalancerFullName'
      ComparisonOperator: GreaterThanThreshold
      MetricName: HTTPCode_ELB_5XX_Count
  # This alarm is triggered, if the backend responds with 5XX status codes
  LoadBalancerTargetGroup5XXAlarm:
    Type: 'AWS::CloudWatch::Alarm'
    Properties:
      EvaluationPeriods: 1
      Statistic: Sum
      Threshold: 0
      AlarmDescription: 'Load balancer target responds with 5XX status codes.'
      Period: 60
      AlarmActions:
      - !Ref Alerts
      Namespace: 'AWS/ApplicationELB'
      Dimensions:
      - Name: LoadBalancer
        Value: !GetAtt 'LoadBalancer.LoadBalancerFullName'
      ComparisonOperator: GreaterThanThreshold
      MetricName: HTTPCode_Target_5XX_Count

Let’s recap what you implemented: A load balancer with a firewall rule that allows traffic on port 80. In the case of 5XX status codes, you will receive an Email. But the load balancer alone is not enough. Now it’s time to add the ECS service.

ECS service

I already talked about the ECS service. It will take care of your containers. To be more precise, it will take care of your tasks that run in the ECS cluster. One task can contain one or multiple Docker containers. Three resources are needed:

  • A Task Definition that describes the Docker containers (similar to a Docker Compose file, or a Kubernetes Deployment)
  • An IAM Role for your container, so you don’t need to pass in static credentials when you want to interact with AWS from within your container. If you don’t want to make AWS API calls from your container the role is not needed.
  • The ECS Service itself.

To make things parameterizable, you also need to add a few parameters to the Parameters section:

infrastructure/ecs.ymlGitHub
# Where does this Docker image comes from? It will be created in the pipeline!
Image:
  Description: 'The image to use for a container, which is passed directly to the Docker daemon. You can use images in the Docker Hub registry or specify other repositories (repository-url/image:tag).'
  Type: String
DesiredCount:
  Description: 'The number of simultaneous tasks, which you specify by using the TaskDefinition property, that you want to run on the cluster.'
  Type: Number
  Default: 2
  ConstraintDescription: 'Must be >= 1'
  MinValue: 1
MaxCapacity:
  Description: 'The maximum number of simultaneous tasks, that you want to run on the cluster.'
  Type: Number
  Default: 4
  ConstraintDescription: 'Must be >= 1'
  MinValue: 1
MinCapacity:
  Description: 'The minimum number of simultaneous tasks, that you want to run on the cluster.'
  Type: Number
  Default: 2
  ConstraintDescription: 'Must be >= 1'
  MinValue: 1

Now you can describe the resources in the CloudFormation template.

infrastructure/ecs.ymlGitHub
TaskDefinition:
  Type: 'AWS::ECS::TaskDefinition'
  Properties: 
    Family: !Ref 'AWS::StackName'
    NetworkMode: bridge
    ContainerDefinitions:
    - Name: main
      Image: !Ref Image # This is where the Docker image is configured
      Memory: 128
      PortMappings:
      - ContainerPort: 3000 # The image exposes the app on port 3000
        Protocol: tcp
      Essential: true
      LogConfiguration:
        LogDriver: awslogs
        Options:
          'awslogs-region': !Ref 'AWS::Region'
          'awslogs-group':
            'Fn::ImportValue': !Sub '${ParentClusterStack}-LogGroup'
          'awslogs-stream-prefix': !Ref 'AWS::StackName'
# The role is using the managed policy AmazonEC2ContainerServiceRole
ServiceRole:
  Type: 'AWS::IAM::Role'
  Properties:
    ManagedPolicyArns:
    - 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole'
    AssumeRolePolicyDocument:
      Version: '2008-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service: 'ecs.amazonaws.com'
        Action: 'sts:AssumeRole'
Service:
  Type: 'AWS::ECS::Service'
  DependsOn: HttpListener
  Properties:
    Cluster:
      'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
    DeploymentConfiguration: # This is the configuration for the rolling update
      MaximumPercent: 200
      MinimumHealthyPercent: 50
    DesiredCount: !Ref DesiredCount
    LoadBalancers:
    - ContainerName: main
      ContainerPort: 3000 # The image exposes the app on port 3000
      TargetGroupArn: !Ref DefaultTargetGroup
    Role: !GetAtt 'ServiceRole.Arn'
    TaskDefinition: !Ref TaskDefinition

Let’s recap what you implemented: A Task definition to define the containers that are managed by the service, an IAM role that is accessible inside the containers, and the ECS service that used the task definition to launch tasks (a bunch of containers) in the cluster. Logs from the containers are already shipped to CLoudWatch Logs by the awslogs log driver and are visible in the Log Group that is part of the ECS cluster template.

Two things are missing:

  1. Scalability of containers
  2. Alerting if containers have issues

Let’s tackle those issues step by step.

Auto scaling of containers

If you auto scale the number of containers, your ECS cluster must be able to auto scale as well. if you use our free template on GitHub, the cluster will auto scale as well

Auto Scaling works similar compared to the EC2 based approach. To scale based on the load you need to add:

  • Scaling Policies to define what should happen if the system should scale up/down
  • CloudWatch Alarms to trigger a Scaling Policy based on a metric such as CPU utilization
  • Additionally, you need a so-called Scalable Target. The Scalable Target can be an ECS service, but it could also be a Spot Fleet or an EMR Instance Group.

Again, you have to add those resources to the Resources section of your template:

infrastructure/ecs.ymlGitHub
ScalableTargetRole: # based on http://docs.aws.amazon.com/AmazonECS/latest/developerguide/autoscale_IAM_role.html
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service: 'application-autoscaling.amazonaws.com'
        Action: 'sts:AssumeRole'
    Path: '/'
    Policies:
    - PolicyName: ecs
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action:
          - 'ecs:DescribeServices'
          - 'ecs:UpdateService'
          Resource: '*'
    - PolicyName: cloudwatch
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action:
          - 'cloudwatch:DescribeAlarms'
          Resource: '*'
ScalableTarget:
  Type: 'AWS::ApplicationAutoScaling::ScalableTarget'
  Properties:
    MaxCapacity: !Ref MaxCapacity
    MinCapacity: !Ref MinCapacity
    ResourceId: !Sub
    - 'service/${Cluster}/${Service}'
    - Cluster:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
      Service: !GetAtt 'Service.Name'
    RoleARN: !GetAtt 'ScalableTargetRole.Arn'
    ScalableDimension: 'ecs:service:DesiredCount'
    ServiceNamespace: ecs
ScaleUpPolicy:
  Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
  Properties:
    PolicyName: !Sub '${AWS::StackName}-scale-up'
    PolicyType: StepScaling
    ScalingTargetId: !Ref ScalableTarget
    StepScalingPolicyConfiguration: 
      AdjustmentType: PercentChangeInCapacity
      Cooldown: 300
      MinAdjustmentMagnitude: 1
      StepAdjustments:
      - MetricIntervalLowerBound: 0
        ScalingAdjustment: 25
ScaleDownPolicy:
  Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
  Properties:
    PolicyName: !Sub '${AWS::StackName}-scale-down'
    PolicyType: StepScaling
    ScalingTargetId: !Ref ScalableTarget
    StepScalingPolicyConfiguration: 
      AdjustmentType: PercentChangeInCapacity
      Cooldown: 300
      MinAdjustmentMagnitude: 1
      StepAdjustments:
      - MetricIntervalLowerBound: 0
        ScalingAdjustment: -25
CPUUtilizationHighAlarm:
  Type: 'AWS::CloudWatch::Alarm'
  Properties:
    AlarmDescription: 'Service is running out of CPU'
    Namespace: 'AWS/ECS'
    Dimensions:
    - Name: ClusterName
      Value:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
    - Name: ServiceName
      Value: !GetAtt 'Service.Name'
    MetricName: CPUUtilization
    ComparisonOperator: GreaterThanThreshold
    Statistic: Average
    Period: 60
    EvaluationPeriods: 1
    Threshold: 70
    AlarmActions:
    - !Ref ScaleUpPolicy
CPUUtilizationLowAlarm:
  Type: 'AWS::CloudWatch::Alarm'
  Properties:
    AlarmDescription: 'Service is wasting CPU'
    Namespace: 'AWS/ECS'
    Dimensions:
    - Name: ClusterName
      Value:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
    - Name: ServiceName
      Value: !GetAtt 'Service.Name'
    MetricName: CPUUtilization
    ComparisonOperator: LessThanThreshold
    Statistic: Average
    Period: 60
    EvaluationPeriods: 1
    Threshold: 30
    AlarmActions:
    - !Ref ScaleDownPolicy

The number of tasks is now increased if the CPU utilization of the service goes above 70%, while the number of tasks is decreased if the CPU utilization falls below 30%.

Monitoring

Last but not least, you have to add CloudWatch Alarms to get alerted if something is wrong with your service. In the Resources section:

infrastructure/ecs.ymlGitHub
# Sends an alert if the average CPU load of the past 5 minutes is higher than 85%
CPUTooHighAlarm:
  Type: 'AWS::CloudWatch::Alarm'
  Properties:
    AlarmDescription: 'Service is running out of CPU'
    Namespace: 'AWS/ECS'
    Dimensions:
    - Name: ClusterName
      Value:
        'Fn::ImportValue': !Sub '${ParentClusterStack}-Cluster'
    - Name: ServiceName
      Value: !GetAtt 'Service.Name'
    MetricName: CPUUtilization
    ComparisonOperator: GreaterThanThreshold
    Statistic: Average
    Period: 300
    EvaluationPeriods: 1
    Threshold: 85
    AlarmActions:
    - !Ref Alerts

The infrastructure is ready now. Read the next part of the series to learn how to setup the CI/CD pipeline to deploy the ECS based app.

Series

AWS Velocity Cover

  1. Set the assembly line up
  2. Local development environment
  3. CI/CD Pipeline as Code
  4. Running your application
  5. EC2 based app
    a. Infrastructure
    b. CI/CD Pipeline
  6. Containerized ECS based app
    a. Infrastructure (you are here)
    b. CI/CD Pipeline
  7. Serverless app
  8. Summary

You can find the source code on GitHub.

Michael Wittig

Michael Wittig

I’ve been building on AWS since 2012 together with my brother Andreas. We are sharing our insights into all things AWS on cloudonaut and have written the book AWS in Action. Besides that, we’re currently working on bucketAV, attachmentAV, HyperEnv, and marbot.

Here are the contact options for feedback and questions.

Further reading