cloudonaut
Home Blog Podcast YouTube Games Books Logout
Subscribe Logout

AWS Velocity Series: Containerized ECS based app CI/CD pipeline

Michael Wittig – Updated 07 Nov 2017

In the previous article, you learned how to use CloudFormation to describe a production-ready infrastructure for a containerized app running on ECS. In this article you will learn to:

  • Automate the creation of a Docker image that contains the app
  • Deploy a CloudFormation stack based infrastructure/ecs.yml with AWS CodePipeline
  • Run acceptance tests on AWS CodeBuild against the infrastructure created in the previous step
  • Deploy another CloudFormation stack for the production environment

You can follow step by step or get the full source code here: https://github.com/widdix/aws-velocity

The pipeline is based on the CI/CD Pipeline as Code part of this series.

AWS Velocity Cover

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Docker image

The Docker image contains the runtime and your application. You will use Amazon Linux, Node.js, and the factorial application, developed in a previous post to create your Docker image. The Docker image is later used by Docker containers running on the ECS cluster.

Docker Image flow

A Docker image is build using the docker build command. A Dockerfile defines the steps that are necessary to produce the image. To create the image that contains the factorial application, the following steps are needed:

  1. Update the operating system based on amazonlinux:2016.09 with yum update
  2. Install Node.js 6.x
  3. Create a directory for the app and copy the local app folder into the image
  4. Set the workdir and expose the application’s port 3000
  5. Set the command to start the app to node index.js

Let’s so how this looks in a Dockerfile.

infrastructure/DockerfileGitHub
FROM amazonlinux:2016.09

RUN yum -y update

RUN curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -
RUN yum -y install nodejs

RUN mkdir -p /usr/src/app
ADD app /usr/src/app

WORKDIR /usr/src/app

EXPOSE 3000

CMD [ "node", "index.js" ]

If you run docker build -f infrastructure/Dockerfile . inside the project directory, you will create a Docker image locally. But this article is about automation, so you need to embed this step into the pipeline. Let’s see how this works next.

Copy the deploy/pipeline.yml file to deploy/pipeline_ecs.yml to get the starting point right. If you don’t have the deploy/pipeline.yml file you can download it from https://github.com/widdix/aws-velocity.

To integrate the docker build step into the pipeline, add the following resources to the Resources section of deploy/pipeline_ecs.yml to create a CodeBuild project to run docker with the above configuration. Docker also needs a bunch of IAM permissions which are also added.

deploy/pipeline_ecs.ymlGitHub
# Private repository that stores Docker images
ArtifactsRepository:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  DeletionPolicy: Retain
  Type: 'AWS::ECR::Repository'
  Properties: {}
# Docker needs a set of access rights to push images to the repository
ImageCodeBuildRole:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'codebuild.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    Policies:
    - PolicyName: ServiceRole
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        #[...]
        - Sid: ECR
          Effect: 'Allow'
          Action: 'ecr:GetAuthorizationToken'
          Resource: '*'
        - Sid: ECRRepo
          Effect: 'Allow'
          Action:
          - 'ecr:BatchCheckLayerAvailability'
          - 'ecr:CompleteLayerUpload'
          - 'ecr:InitiateLayerUpload'
          - 'ecr:PutImage'
          - 'ecr:UploadLayerPart'
          Resource: !Sub 'arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ArtifactsRepository}'
# build the Docker image and publish the tag of the image as an artifact
ImageProject:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::CodeBuild::Project'
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: 'BUILD_GENERAL1_SMALL'
      Image: 'aws/codebuild/docker:1.12.1'
      Type: 'LINUX_CONTAINER'
    Name: !Sub '${AWS::StackName}-image'
    ServiceRole: !GetAtt 'ImageCodeBuildRole.Arn'
    Source:
      Type: CODEPIPELINE
      BuildSpec: !Sub |
        version: 0.1
        phases:
          build:
            commands:
            - 'docker build -f infrastructure/Dockerfile -t ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ArtifactsRepository}:${!CODEBUILD_BUILD_ID#*:} .'
            - 'eval $(aws ecr get-login --no-include-email) && docker push ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ArtifactsRepository}:${!CODEBUILD_BUILD_ID#*:}'
            - 'echo "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ArtifactsRepository}:${!CODEBUILD_BUILD_ID#*:}" > infrastructure/image.txt'
            - 'echo "{\"image\": \"${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ArtifactsRepository}:${!CODEBUILD_BUILD_ID#*:}\"}" > infrastructure/image.json'
        artifacts:
          files:
          - 'infrastructure/image.json'
          - 'infrastructure/image.txt'
    TimeoutInMinutes: 10

You also need to make a small modification to the exiting BuildSpec in the AppProject resource to include the image files into the App artifact. This is a hack because CodeBuild does not support multiple input artifacts at the moment. Change the artifacts section to:

deploy/pipeline_ecs.ymlGitHub
artifacts:
  files:
  - 'app/**/*'
  - 'infrastructure/image.json' # this is a hack because we can not pass multiple Artifacts as an input to CodeBuild at the moment
  - 'infrastructure/image.sh' # this is a hack because we can not pass multiple Artifacts as an input to CodeBuild at the moment

Now the CodeBuild project needs to be called in the pipeline, therefore change the Pipeline resource in the file deploy/pipeline_ecs.yml and add a new build action:

deploy/pipeline_ecs.ymlGitHub
Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: BuildAndTestAcceptance
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AcceptanceProject
        InputArtifacts:
        - Name: Source
        OutputArtifacts:
        - Name: Acceptance
        RunOrder: 1
      # NEW STUFF!
      - Name: BuildImage
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref ImageProject
        InputArtifacts:
        - Name: App
        OutputArtifacts:
        - Name: Image
        RunOrder: 2

Now, a new Docker image is automatically build and pushed to the ECR repository whenever the pipeline runs. The image will include the app and the latest security patches.

It’s time to deploy the app to the acceptance stage and too see if the app works.

Acceptance stage

The acceptance stage consists of a CloudFormation stack based on infrastructure/ecs.yml and the execution of the acceptance tests. To create the CloudFormation stack, you first have to provide a few parameters. Create a file infrastructure/ecs.json with the following content:

infrastructure/ecs.jsonGitHub
{
  "Parameters": {
    "ParentVPCStack": "vpc-2azs",
    "ParentClusterStack": "ecs-cluster",
    "Image": {"Fn::GetParam": ["Image", "infrastructure/image.json", "image"]},
    "DesiredCount": "2",
    "MaxCapacity": "4",
    "MinCapacity": "2",
    "AdminEmail": "your@email.com"
  }
}

Make sure to change the ParentVPCStack and the ParentClusterStack parameter in the infrastructure/ecs.yml accordingly to your stack names. Also change the value of the AdminEmail parameter to your email address. The other values can be stay as they are. Look at the Image parameter value. This is the way of getting a value out of a JSON artifact file in CodePipeline. The artifact is crated in the BuildImage action in the CodePipeline.

To run the acceptance tests, you also need another CodeBuild project, add the following resources to the Resources section of deploy/pipeline_ecs.yml:

deploy/pipeline_ecs.ymlGitHub
RunAcceptanceCodeBuildRole:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'codebuild.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    Policies:
    - PolicyName: ServiceRole
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudWatchLogsPolicy
          Effect: Allow
          Action: 
          - 'logs:CreateLogGroup'
          - 'logs:CreateLogStream'
          - 'logs:PutLogEvents'
          Resource: '*'
        - Sid: CodeCommitPolicy
          Effect: Allow
          Action: 'codecommit:GitPull'
          Resource: '*'
        - Sid: S3GetObjectPolicy
          Effect: Allow
          Action: 
          - 's3:GetObject'
          - 's3:GetObjectVersion'
          Resource: '*'
        - Sid: S3PutObjectPolicy
          Effect: 'Allow'
          Action: 's3:PutObject'
          Resource: '*'
    - PolicyName: CloudFormation
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudFormation
          Effect: Allow
          Action: 
          - 'cloudformation:DescribeStacks'
          Resource: !Sub 'arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${AWS::StackName}-acceptance/*'
RunAcceptanceProject:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::CodeBuild::Project'
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: 'BUILD_GENERAL1_SMALL'
      Image: 'aws/codebuild/nodejs:6.3.1'
      Type: 'LINUX_CONTAINER'
    Name: !Sub '${AWS::StackName}-run-acceptance'
    ServiceRole: !GetAtt 'RunAcceptanceCodeBuildRole.Arn'
    Source:
      Type: CODEPIPELINE
      BuildSpec: !Sub |
        version: 0.1
        phases:
          build: # execute acceptance tests against the acceptance stack
            commands:
            - 'cd acceptance/ && ENDPOINT=`aws cloudformation describe-stacks --stack-name ${AWS::StackName}-acceptance --query "Stacks[0].Outputs[?OutputKey==''URL''].OutputValue" --output text` ./node_modules/jasmine-node/bin/jasmine-node .'
    TimeoutInMinutes: 10

Now the CodeBuild project needs to be called in the pipeline, therefore change Pipeline resource in the file deploy/pipeline_ecs.yml to:

  1. deploy the CloudFormation stack suffixed with -acceptance
  2. run the acceptance tests
deploy/pipeline_ecs.ymlGitHub
Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: BuildImage
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref ImageProject
        InputArtifacts:
        - Name: App
        OutputArtifacts:
        - Name: Image
        RunOrder: 2
    # NEW STUFF!
    - Name: Acceptance
      Actions:
      - Name: Deploy
        ActionTypeId:
          Category: Deploy
          Owner: AWS
          Provider: CloudFormation
          Version: 1
        Configuration:
          ActionMode: CREATE_UPDATE
          Capabilities: CAPABILITY_IAM
          RoleArn: !GetAtt 'CloudFormationRole.Arn'
          StackName: !Sub '${AWS::StackName}-acceptance'
          TemplatePath: 'Source::infrastructure/ecs.yml'
          TemplateConfiguration: 'Source::infrastructure/ecs.json'
        InputArtifacts:
        - Name: Source
        - Name: Image
        RunOrder: 1
      - Name: Test
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref RunAcceptanceProject
        InputArtifacts:
        - Name: Acceptance
        RunOrder: 2

The acceptance stage is now ready.

Production stage

The production stage is pretty simple, just one CloudFormation stack. Change the Pipeline resource in the file deploy/pipeline_ecs.yml to add a new stage that looks familiar to the acceptance stage:

deploy/pipeline_ecs.ymlGitHub
Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    # [...]
      - Name: Test
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref RunAcceptanceProject
        InputArtifacts:
        - Name: Acceptance
        RunOrder: 2
    - Name: Production
      Actions:
      - Name: Deploy
        ActionTypeId:
          Category: Deploy
          Owner: AWS
          Provider: CloudFormation
          Version: 1
        Configuration:
          ActionMode: CREATE_UPDATE
          Capabilities: CAPABILITY_IAM
          RoleArn: !GetAtt 'CloudFormationRole.Arn'
          StackName: !Sub '${AWS::StackName}-production'
          TemplatePath: 'Source::infrastructure/ecs.yml'
          TemplateConfiguration: 'Source::infrastructure/ecs.json'
        InputArtifacts:
        - Name: Source
        - Name: Image
        RunOrder: 1

Now the Docker image containing the application is deployed to production with confidence and without disturbing the users. Try it and run the pipeline!

Summary

Let’s use my production-ready definition to summarize how each point is implemented:

  • Highly available: The load balancer (which is HA) sits in front of a fleet of Docker containers managed by the ECS service for maximum availability. In the case of an unhealthy container, the ECS service will replace that container.
  • Scalable: If the CPU utilization gets over 70%, a Cloud Watch Alarm triggers a Scaling Policy to add new containers automatically. If the ECS cluster runs out of capacity, it is scaled as well.
  • Frictionless deployment: To deploy a new version of the app, a new Docker image is created. This image is then rolled out to the acceptance environment by updating the CloudFormation stack with the new Image parameter. The ECS Service performs a rolling update to avoid the application being down during deployment.
  • Secure: During Docker image creation, the latest patches are applied. You must ensure that the pipeline runs often enough to keep up with new patches. Besides that, Security Groups control network traffic to the EC2 instances running the containers. When following the bastion host approach, you get maximum security. The EC2 instance is only allowed to send logs to CloudWatch Logs by following the least privileges approach. The container itself has no additional IAM permissions but has access to the instance permissions by default as well.
  • Operations: All logs are stored in CloudWatch Logs, important metrics are monitored, and alarms are defined.

If you now have the impression that deploying and running an app on ECS is a bit easier compared to the plain EC2 approach you are right. In the next article, you will learn about yet another option with even fewer responsibilities.

One more thing: You need to automate the patching of the EC2 instances as well. To do so, you could follow the approach that I demonstrated in the EC2 based app and use Packer to keep the AMI up-to-date in an automated way.

Series

AWS Velocity Cover

  1. Set the assembly line up
  2. Local development environment
  3. CI/CD Pipeline as Code
  4. Running your application
  5. EC2 based app
    a. Infrastructure
    b. CI/CD Pipeline
  6. Containerized ECS based app
    a. Infrastructure
    b. CI/CD Pipeline (you are here)
  7. Serverless app
  8. Summary

You can find the source code on GitHub.

Michael Wittig

Michael Wittig

I’ve been building on AWS since 2012 together with my brother Andreas. We are sharing our insights into all things AWS on cloudonaut and have written the book AWS in Action. Besides that, we’re currently working on bucketAV, HyperEnv for GitHub Actions, and marbot.

Here are the contact options for feedback and questions.

Further reading