AWS Velocity Series: Containerized ECS based app CI/CD pipeline

In the previous article, you learned how to use CloudFormation to describe a production-ready infrastructure for a containerized app running on ECS. In this article you will learn to:

  • Automate the creation of a Docker image that contains the app
  • Deploy a CloudFormation stack based infrastructure/ecs.yml with AWS CodePipeline
  • Run acceptance tests on AWS CodeBuild against the infrastructure created in the previous step
  • Deploy another CloudFormation stack for the production environment

You can follow step by step or get the full source code here: https://github.com/widdix/aws-velocity

The pipeline is based on the CI/CD Pipeline as Code part of this series.

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Docker image

The Docker image contains the runtime and your application. You will use Amazon Linux, Node.js, and the factorial application, developed in a previous post to create your Docker image. The Docker image is later used by Docker containers running on the ECS cluster.

Docker Image flow

A Docker image is build using the docker build command. A Dockerfile defines the steps that are necessary to produce the image. To create the image that contains the factorial application, the following steps are needed:

  1. Update the operating system based on amazonlinux:2016.09 with yum update
  2. Install Node.js 6.x
  3. Create a directory for the app and copy the local app folder into the image
  4. Set the workdir and expose the application’s port 3000
  5. Set the command to start the app to node index.js

Let’s so how this looks in a Dockerfile.

infrastructure/DockerfileGitHub
FROM amazonlinux:2016.09
RUN yum -y update
RUN curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -
RUN yum -y install nodejs
RUN mkdir -p /usr/src/app
ADD app /usr/src/app
WORKDIR /usr/src/app
EXPOSE 3000
CMD [ "node", "index.js" ]

If you run docker build -f infrastructure/Dockerfile . inside the project directory, you will create a Docker image locally. But this article is about automation, so you need to embed this step into the pipeline. Let’s see how this works next.

Copy the deploy/pipeline.yml file to deploy/pipeline_ecs.yml to get the starting point right. If you don’t have the deploy/pipeline.yml file you can download it from https://github.com/widdix/aws-velocity.

To integrate the docker build step into the pipeline, add the following resources to the Resources section of deploy/pipeline_ecs.yml to create a CodeBuild project to run docker with the above configuration. Docker also needs a bunch of IAM permissions which are also added.

deploy/pipeline_ecs.ymlGitHub
# Private repository that stores Docker images
ArtifactsRepository:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
DeletionPolicy: Retain
Type: 'AWS::ECR::Repository'
Properties: {}
# Docker needs a set of access rights to push images to the repository
ImageCodeBuildRole:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- 'codebuild.amazonaws.com'
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: ServiceRole
PolicyDocument:
Version: '2012-10-17'
Statement:
#[...]
- Sid: ECR
Effect: 'Allow'
Action: 'ecr:GetAuthorizationToken'
Resource: '*'
- Sid: ECRRepo
Effect: 'Allow'
Action:
- 'ecr:BatchCheckLayerAvailability'
- 'ecr:CompleteLayerUpload'
- 'ecr:InitiateLayerUpload'
- 'ecr:PutImage'
- 'ecr:UploadLayerPart'
Resource: !Sub 'arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ArtifactsRepository}'
# build the Docker image and publish the tag of the image as an artifact
ImageProject:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::CodeBuild::Project'
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: 'BUILD_GENERAL1_SMALL'
Image: 'aws/codebuild/docker:1.12.1'
Type: 'LINUX_CONTAINER'
Name: !Sub '${AWS::StackName}-image'
ServiceRole: !GetAtt 'ImageCodeBuildRole.Arn'
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
phases:
build:
commands:
- 'docker build -f infrastructure/Dockerfile -t ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ArtifactsRepository}:${!CODEBUILD_BUILD_ID#*:} .'
post_build:
commands:
- 'eval $(aws ecr get-login) && docker push ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ArtifactsRepository}:${!CODEBUILD_BUILD_ID#*:}'
- 'echo "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ArtifactsRepository}:${!CODEBUILD_BUILD_ID#*:}" > infrastructure/image.txt'
- 'echo "{\"image\": \"${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ArtifactsRepository}:${!CODEBUILD_BUILD_ID#*:}\"}" > infrastructure/image.json'
artifacts:
files:
- 'infrastructure/image.json'
- 'infrastructure/image.txt'
TimeoutInMinutes: 10

You also need to make a small modification to the exiting BuildSpec in the AppProject resource to include the image files into the App artifact. This is a hack because CodeBuild does not support multiple input artifacts at the moment. Change the artifacts section to:

deploy/pipeline_ecs.ymlGitHub
artifacts:
files:
- 'app/**/*'
- 'infrastructure/image.json' # this is a hack because we can not pass multiple Artifacts as an input to CodeBuild at the moment
- 'infrastructure/image.sh' # this is a hack because we can not pass multiple Artifacts as an input to CodeBuild at the moment

Now the CodeBuild project needs to be called in the pipeline, therefore change the Pipeline resource in the file deploy/pipeline_ecs.yml and add a new build action:

deploy/pipeline_ecs.ymlGitHub
Pipeline:
Type: 'AWS::CodePipeline::Pipeline'
Properties:
# [...]
- Name: BuildAndTestAcceptance
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref AcceptanceProject
InputArtifacts:
- Name: Source
OutputArtifacts:
- Name: Acceptance
RunOrder: 1
# NEW STUFF!
- Name: BuildImage
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref ImageProject
InputArtifacts:
- Name: App
OutputArtifacts:
- Name: Image
RunOrder: 2

Now, a new Docker image is automatically build and pushed to the ECR repository whenever the pipeline runs. The image will include the app and the latest security patches.

It’s time to deploy the app to the acceptance stage and too see if the app works.

Acceptance stage

The acceptance stage consists of a CloudFormation stack based on infrastructure/ecs.yml and the execution of the acceptance tests. To create the CloudFormation stack, you first have to provide a few parameters. Create a file infrastructure/ecs.json with the following content:

infrastructure/ecs.jsonGitHub
{
"Parameters": {
"ParentVPCStack": "vpc-2azs",
"ParentClusterStack": "ecs-cluster",
"Image": {"Fn::GetParam": ["Image", "infrastructure/image.json", "image"]},
"DesiredCount": "2",
"MaxCapacity": "4",
"MinCapacity": "2",
"AdminEmail": "your@email.com"
}
}

Make sure to change the ParentVPCStack and the ParentClusterStack parameter in the infrastructure/ecs.yml accordingly to your stack names. Also change the value of the AdminEmail parameter to your email address. The other values can be stay as they are. Look at the Image parameter value. This is the way of getting a value out of a JSON artifact file in CodePipeline. The artifact is crated in the BuildImage action in the CodePipeline.

To run the acceptance tests, you also need another CodeBuild project, add the following resources to the Resources section of deploy/pipeline_ecs.yml:

deploy/pipeline_ecs.ymlGitHub
RunAcceptanceCodeBuildRole:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- 'codebuild.amazonaws.com'
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: ServiceRole
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: CloudWatchLogsPolicy
Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
- Sid: CodeCommitPolicy
Effect: Allow
Action: 'codecommit:GitPull'
Resource: '*'
- Sid: S3GetObjectPolicy
Effect: Allow
Action:
- 's3:GetObject'
- 's3:GetObjectVersion'
Resource: '*'
- Sid: S3PutObjectPolicy
Effect: 'Allow'
Action: 's3:PutObject'
Resource: '*'
- PolicyName: CloudFormation
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: CloudFormation
Effect: Allow
Action:
- 'cloudformation:DescribeStacks'
Resource: !Sub 'arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${AWS::StackName}-acceptance/*'
RunAcceptanceProject:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::CodeBuild::Project'
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: 'BUILD_GENERAL1_SMALL'
Image: 'aws/codebuild/nodejs:6.3.1'
Type: 'LINUX_CONTAINER'
Name: !Sub '${AWS::StackName}-run-acceptance'
ServiceRole: !GetAtt 'RunAcceptanceCodeBuildRole.Arn'
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
phases:
build: # execute acceptance tests against the acceptance stack
commands:
- 'cd acceptance/ && ENDPOINT=`aws cloudformation describe-stacks --stack-name ${AWS::StackName}-acceptance --query "Stacks[0].Outputs[?OutputKey==''URL''].OutputValue" --output text` ./node_modules/jasmine-node/bin/jasmine-node .'
TimeoutInMinutes: 10

Now the CodeBuild project needs to be called in the pipeline, therefore change Pipeline resource in the file deploy/pipeline_ecs.yml to:

  1. deploy the CloudFormation stack suffixed with -acceptance
  2. run the acceptance tests
deploy/pipeline_ecs.ymlGitHub
Pipeline:
Type: 'AWS::CodePipeline::Pipeline'
Properties:
# [...]
- Name: BuildImage
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref ImageProject
InputArtifacts:
- Name: App
OutputArtifacts:
- Name: Image
RunOrder: 2
# NEW STUFF!
- Name: Acceptance
Actions:
- Name: Deploy
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt 'CloudFormationRole.Arn'
StackName: !Sub '${AWS::StackName}-acceptance'
TemplatePath: 'Source::infrastructure/ecs.yml'
TemplateConfiguration: 'Source::infrastructure/ecs.json'
InputArtifacts:
- Name: Source
- Name: Image
RunOrder: 1
- Name: Test
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref RunAcceptanceProject
InputArtifacts:
- Name: Acceptance
RunOrder: 2

The acceptance stage is now ready.

Production stage

The production stage is pretty simple, just one CloudFormation stack. Change the Pipeline resource in the file deploy/pipeline_ecs.yml to add a new stage that looks familiar to the acceptance stage:

deploy/pipeline_ecs.ymlGitHub
Pipeline:
Type: 'AWS::CodePipeline::Pipeline'
Properties:
# [...]
- Name: Test
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref RunAcceptanceProject
InputArtifacts:
- Name: Acceptance
RunOrder: 2
- Name: Production
Actions:
- Name: Deploy
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt 'CloudFormationRole.Arn'
StackName: !Sub '${AWS::StackName}-production'
TemplatePath: 'Source::infrastructure/ecs.yml'
TemplateConfiguration: 'Source::infrastructure/ecs.json'
InputArtifacts:
- Name: Source
- Name: Image
RunOrder: 1

Now the Docker image containing the application is deployed to production with confidence and without disturbing the users. Try it and run the pipeline!

Summary

Let’s use my production-ready definition to summarize how each point is implemented:

  • Highly available: The load balancer (which is HA) sits in front of a fleet of Docker containers managed by the ECS service for maximum availability. In the case of an unhealthy container, the ECS service will replace that container.
  • Scalable: If the CPU utilization gets over 70%, a Cloud Watch Alarm triggers a Scaling Policy to add new containers automatically. If the ECS cluster runs out of capacity, it is scaled as well.
  • Frictionless deployment: To deploy a new version of the app, a new Docker image is created. This image is then rolled out to the acceptance environment by updating the CloudFormation stack with the new Image parameter. The ECS Service performs a rolling update to avoid the application being down during deployment.
  • Secure: During Docker image creation, the latest patches are applied. You must ensure that the pipeline runs often enough to keep up with new patches. Besides that, Security Groups control network traffic to the EC2 instances running the containers. When following the bastion host approach, you get maximum security. The EC2 instance is only allowed to send logs to CloudWatch Logs by following the least privileges approach. The container itself has no additional IAM permissions but has access to the instance permissions by default as well.
  • Operations: All logs are stored in CloudWatch Logs, important metrics are monitored, and alarms are defined.

If you now have the impression that deploying and running an app on ECS is a bit easier compared to the plain EC2 approach you are right. In the next article, you will learn about yet another option with even fewer responsibilities.

One more thing: You need to automate the patching of the EC2 instances as well. To do so, you could follow the approach that I demonstrated in the EC2 based app and use Packer to keep the AMI up-to-date in an automated way.

Series

The AWS Velocity series is split into readable chunks. Subscribe to our RSS feed or newsletter to get notified when new articles become available.

AWS Velocity Cover

  1. Set the assembly line up
  2. Local development environment
  3. CI/CD Pipeline as Code
  4. Running your application
  5. EC2 based app
    a. Infrastructure
    b. CI/CD Pipeline
  6. Containerized ECS based app
    a. Infrastructure
    b. CI/CD Pipeline (you are here)
  7. Serverless app
  8. Summary

You can find the source code on GitHub.

Published on


Subscribe to free monthly updates.

One subscriber wins our book AWS in Action every month.

Michael Wittig

Michael Wittig

I’m the author of Amazon Web Services in Action. I work as a software engineer, and independent consultant focused on AWS and DevOps.

Is anything missing in my article? I'm looking forward to your feedback! @hellomichibye or michael@widdix.de.

marbot

Are you part of a highly motivated DevOps team? Use marbot, a friendly chatbot, to forward all kind of alerts from your AWS infrastructure to Slack. Alerts are escalated across your team automatically allowing you to focus on your daily work.

Amazon Web Services in Action

Amazon Web Services in Action introduces you to computing, storing, and networking in the AWS cloud.

Customer Reviews:

Subscribe to free monthly updates.

One subscriber wins our book AWS in Action every month.


Subscribe for free