AWS Velocity Series: EC2 based app CI/CD pipeline

In the previous article, you learned how to use CloudFormation to describe a production-ready infrastructure for an EC2 based app. In this article you will learn to:

  • Automate the creation of an AMI that contains the app with Packer
  • Deploy a CloudFormation stack based infrastructure/ec2.yml with AWS CodePipeline
  • Run the Acceptance tests on AWS CodeBuild against the infrastructure created in the previous step
  • Deploy another CloudFormation stack for the production environment

You can follow step by step or get the full source code here: https://github.com/widdix/aws-velocity

The pipeline is based on the CI/CD Pipeline as Code part of this series.

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Additionally, the EC2 based app pipeline contains:

  • add a BuildAMI action to the Build stage.
  • add an Acceptance stage
  • add a Production stage

Copy the deploy/pipeline.yml file to deploy/pipeline_ec2.yml to get the starting point right. If you don’t have the deploy/pipeline.yml file you can download it from https://github.com/widdix/aws-velocity.

BuildAMI Action

EC2 instances start from an image (AMI) that contains the operating system including all the files that are needed. You can create your own AMI as well. Usually, you take one of the available AMIs like the Amazon Linux, make your modifications, and then create a new image from that. This whole procedure can also be automated with a tool called Packer. You will now see how you can run Packer in CodeBuild to create a new AMI that contains the app.

Packer itself needs configuration files. Create a file infrastructure/packer.json with the following content that build a new AMI based on ami-c51e3eb6 (Amazon Linux) and a Bash script that you will create later:

{
"variables": {
"ami_name": "{{env `CODEBUILD_BUILD_ID`}}"
},
"builders": [{
"type": "amazon-ebs",
"region": "eu-west-1",
"source_ami": "ami-c51e3eb6",
"instance_type": "t2.micro",
"ssh_username": "ec2-user",
"ami_name": "{{user `ami_name` | clean_ami_name}}",
"ami_regions": ["REGION"]
}],
"provisioners": [{
"type": "file",
"source": "app",
"destination": "/tmp"
}, {
"type": "shell",
"script": "infrastructure/packer.sh"
}]
}

Packer is configured to run a Bash script to provision the AMI and to upload the app folder. Create a file infrastructure/packer.sh with the following content to:

  • install the latest patches
  • install Node.js 6.x
  • install the CloudWatch Logs agent
  • install forever, a tool to run Node.js script in the background
infrastructure/packer.shGitHub
#!/bin/bash -ex
# this script runs as ec2-user

sudo yum -y update
curl --silent --location https://rpm.nodesource.com/setup_6.x | sudo bash -
sudo yum -y install nodejs awslogs
sudo npm install -g forever@0.15.3
sudo mv /tmp/app /opt

The script also moves the application files to the right place.

To integrate Packer into the pipeline, add the following resources to the Resources section of deploy/pipeline_ec2.yml to create a CodeBuild project to run Packer with the above configuration. Packer also needs a bunch of IAM permissions which are also added.

deploy/pipeline_ec2.ymlGitHub
# Packer needs a set of access rights as defined in https://www.packer.io/docs/builders/amazon.html
AMICodeBuildRole:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- 'codebuild.amazonaws.com'
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: ServiceRole
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: CloudWatchLogsPolicy
Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
- Sid: CodeCommitPolicy
Effect: Allow
Action: 'codecommit:GitPull'
Resource: '*'
- Sid: S3GetObjectPolicy
Effect: Allow
Action:
- 's3:GetObject'
- 's3:GetObjectVersion'
Resource: '*'
- Sid: S3PutObjectPolicy
Effect: 'Allow'
Action: 's3:PutObject'
Resource: '*'
- PolicyName: EC2
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: CloudFormation
Effect: Allow
Action:
- 'ec2:AttachVolume'
- 'ec2:AuthorizeSecurityGroupIngress'
- 'ec2:CopyImage'
- 'ec2:CreateImage'
- 'ec2:CreateKeypair'
- 'ec2:CreateSecurityGroup'
- 'ec2:CreateSnapshot'
- 'ec2:CreateTags'
- 'ec2:CreateVolume'
- 'ec2:DeleteKeypair'
- 'ec2:DeleteSecurityGroup'
- 'ec2:DeleteSnapshot'
- 'ec2:DeleteVolume'
- 'ec2:DeregisterImage'
- 'ec2:DescribeImageAttribute'
- 'ec2:DescribeImages'
- 'ec2:DescribeInstances'
- 'ec2:DescribeRegions'
- 'ec2:DescribeSecurityGroups'
- 'ec2:DescribeSnapshots'
- 'ec2:DescribeSubnets'
- 'ec2:DescribeTags'
- 'ec2:DescribeVolumes'
- 'ec2:DetachVolume'
- 'ec2:GetPasswordData'
- 'ec2:ModifyImageAttribute'
- 'ec2:ModifyInstanceAttribute'
- 'ec2:ModifySnapshotAttribute'
- 'ec2:RegisterImage'
- 'ec2:RunInstances'
- 'ec2:StopInstances'
- 'ec2:TerminateInstances'
Resource: '*'
# This IAM User is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
AMICodeBuildUser:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::IAM::User'
Properties:
Policies:
- PolicyName: Packer
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: EC2
Effect: Allow
Action:
- 'ec2:AttachVolume'
- 'ec2:AuthorizeSecurityGroupIngress'
- 'ec2:CopyImage'
- 'ec2:CreateImage'
- 'ec2:CreateKeypair'
- 'ec2:CreateSecurityGroup'
- 'ec2:CreateSnapshot'
- 'ec2:CreateTags'
- 'ec2:CreateVolume'
- 'ec2:DeleteKeypair'
- 'ec2:DeleteSecurityGroup'
- 'ec2:DeleteSnapshot'
- 'ec2:DeleteVolume'
- 'ec2:DeregisterImage'
- 'ec2:DescribeImageAttribute'
- 'ec2:DescribeImages'
- 'ec2:DescribeInstances'
- 'ec2:DescribeRegions'
- 'ec2:DescribeSecurityGroups'
- 'ec2:DescribeSnapshots'
- 'ec2:DescribeSubnets'
- 'ec2:DescribeTags'
- 'ec2:DescribeVolumes'
- 'ec2:DetachVolume'
- 'ec2:GetPasswordData'
- 'ec2:ModifyImageAttribute'
- 'ec2:ModifyInstanceAttribute'
- 'ec2:ModifySnapshotAttribute'
- 'ec2:RegisterImage'
- 'ec2:RunInstances'
- 'ec2:StopInstances'
- 'ec2:TerminateInstances'
Resource: '*'
# This IAM Access Key is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
AMICodeBuildUserAccessKey:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::IAM::AccessKey'
Properties:
UserName: !Ref AMICodeBuildUser
AMIProject:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::CodeBuild::Project'
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: 'BUILD_GENERAL1_SMALL'
Image: 'amazonlinux:2016.09'
Type: 'LINUX_CONTAINER'
EnvironmentVariables: # pass in the AWS credentials as environment variables is only temporarily necessary until https://github.com/mitchellh/packer/pull/4613 is fixed!
- Name: 'AWS_ACCESS_KEY_ID'
Value: !Ref AMICodeBuildUserAccessKey
- Name: 'AWS_SECRET_ACCESS_KEY'
Value: !GetAtt 'AMICodeBuildUserAccessKey.SecretAccessKey'
Name: !Sub '${AWS::StackName}-ami'
ServiceRole: !GetAtt 'AMICodeBuildRole.Arn'
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
phases:
install: # install Packer
commands:
- 'yum -y install unzip'
- 'curl -s -m 60 -o /opt/packer.zip https://releases.hashicorp.com/packer/0.12.3/packer_0.12.3_linux_amd64.zip'
- 'unzip /opt/packer.zip -d /opt'
pre_build: # replace the REGION placeholder with the stack's region
commands:
- 'sed -i "s:REGION:${AWS::Region}:g" infrastructure/packer.json'
build: # run packer
commands:
- '/opt/packer -machine-readable build infrastructure/packer.json | tee infrastructure/packer.txt'
post_build: # extract the AMI id into a JSON file
commands:
- 'echo "{" > infrastructure/ami.json'
- 'cat infrastructure/packer.txt | grep '',amazon-ebs,artifact,0,id,'' | awk -F'','' ''{print $6}'' | sed ''s/%!(PACKER_COMMA)/\''$''\n/g'' | awk -F'':'' ''{print "\"image\": \""$2"\","}'' | sed ''$ s/.$//'' >> infrastructure/ami.json'
- 'echo "}" >> infrastructure/ami.json'
artifacts:
files:
- 'infrastructure/packer.txt'
- 'infrastructure/ami.json'
TimeoutInMinutes: 10

You also need to make a small modification to the exiting BuildSpec in the AppProject resource to include the packer files into the App artifact. This is a hack because CodeBuild does not support multiple input artifacts at the moment. Change the artifacts section to:

deploy/pipeline_ec2.ymlGitHub
artifacts:
files:
- 'app/**/*'
- 'infrastructure/packer.json' # this is a hack because we can not pass multiple Arifacts as an input to CodeBuild at the moment
- 'infrastructure/packer.sh' # this is a hack because we can not pass multiple Arifacts as an input to CodeBuild at the moment

Now the CodeBuild project needs to be called in the pipeline, therefore change the Pipeline resource in the file deploy/pipeline_ec2.yml and add a new build action:

deploy/pipeline_ec2.ymlGitHub
Pipeline:
Type: 'AWS::CodePipeline::Pipeline'
Properties:
# [...]
- Name: BuildAndTestAcceptance
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref AcceptanceProject
InputArtifacts:
- Name: Source
OutputArtifacts:
- Name: Acceptance
RunOrder: 1
# NEW STUFF!
- Name: BuildAMI
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref AMIProject
InputArtifacts:
- Name: App
OutputArtifacts:
- Name: AMI
RunOrder: 2

Now, a new AMI is automatically created whenever the pipeline runs. The AMI will include the app and the latest patches.

It’s time to deploy the app to the acceptance stage and too see if the app works.

Acceptance stage

The acceptance stage consists of a CloudFormation stack based on infrastructure/ec2.yml and the execution of the acceptance tests. To create the CloudFormation stack, you first have to provide a few parameters. Create a file infrastructure/ec2.json with the following content:

infrastructure/ec2.jsonGitHub
{
"Parameters": {
"ImageId": {"Fn::GetParam": ["AMI", "infrastructure/ami.json", "image"]},
"ParentVPCStack": "vpc-2azs",
"ParentSSHBastionStack": "",
"KeyName": "",
"AdminEmail": "your@email.com"
}
}

If you don’t have a VPC stack based of our Free Templates for AWS CloudFormation (https://github.com/widdix/aws-cf-templates/tree/master/vpc) create a VPC stack first. Make sure to change the ParentVPCStack parameter in the infrastructure/ec2.yml accordingly. Also change the value of the AdminEmail parameter. The other values can be stay as they are. Look at the ImageId parameter value. This is the way of getting a value out of a JSON artifact file in CoePipeline.

To run the acceptance tests, you also need another CodeBuild project, add the following resources to the Resources section of deploy/pipeline_ec2.yml:

deploy/pipeline_ec2.ymlGitHub
ExtendedCodeBuildRole:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- 'codebuild.amazonaws.com'
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: ServiceRole
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: CloudWatchLogsPolicy
Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
- Sid: CodeCommitPolicy
Effect: Allow
Action: 'codecommit:GitPull'
Resource: '*'
- Sid: S3GetObjectPolicy
Effect: Allow
Action:
- 's3:GetObject'
- 's3:GetObjectVersion'
Resource: '*'
- Sid: S3PutObjectPolicy
Effect: 'Allow'
Action: 's3:PutObject'
Resource: '*'
- PolicyName: CloudFormation
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: CloudFormation
Effect: Allow
Action:
- 'cloudformation:DescribeStacks'
Resource: !Sub 'arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/${AWS::StackName}-acceptance/*'
RunAcceptanceProject:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::CodeBuild::Project'
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: 'BUILD_GENERAL1_SMALL'
Image: 'aws/codebuild/nodejs:6.3.1'
Type: 'LINUX_CONTAINER'
Name: !Sub '${AWS::StackName}-run-acceptance'
ServiceRole: !GetAtt 'ExtendedCodeBuildRole.Arn'
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.1
phases:
build: # execute acceptance tests against the acceptance stack
commands:
- 'cd acceptance/ && ENDPOINT=`aws cloudformation describe-stacks --stack-name ${AWS::StackName}-acceptance --query "Stacks[0].Outputs[?OutputKey==''URL''].OutputValue" --output text` ./node_modules/jasmine-node/bin/jasmine-node .'
TimeoutInMinutes: 10

Now the CodeBuild project needs to be called in the pipeline, therefore change Pipeline resource in the file deploy/pipeline_ec2.yml to:

  1. deploy the CloudFormation stack suffixed with -acceptance
  2. run the acceptance tests
deploy/pipeline_ec2.ymlGitHub
Pipeline:
Type: 'AWS::CodePipeline::Pipeline'
Properties:
# [...]
- Name: BuildAMI
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref AMIProject
InputArtifacts:
- Name: App
OutputArtifacts:
- Name: AMI
RunOrder: 2
# NEW STUFF!
- Name: Acceptance
Actions:
- Name: Deploy
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt 'CloudFormationRole.Arn'
StackName: !Sub '${AWS::StackName}-acceptance'
TemplatePath: 'Source::infrastructure/ec2.yml'
TemplateConfiguration: 'Source::infrastructure/ec2.json'
InputArtifacts:
- Name: Source
- Name: AMI
RunOrder: 1
- Name: Test
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref RunAcceptanceProject
InputArtifacts:
- Name: Acceptance
RunOrder: 2

The acceptance stage is now ready.

Production stage

The production stage is pretty simple, just one CloudFormation stack. Change the Pipeline resource in the file deploy/pipeline_ec2.yml to add a new stage that looks familiar to the acceptance stage:

deploy/pipeline_ec2.ymlGitHub
Pipeline:
Type: 'AWS::CodePipeline::Pipeline'
Properties:
# [...]
- Name: Test
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref RunAcceptanceProject
InputArtifacts:
- Name: Acceptance
RunOrder: 2
# NEW STUFF!
- Name: Production
Actions:
- Name: Deploy
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt 'CloudFormationRole.Arn'
StackName: !Sub '${AWS::StackName}-production'
TemplatePath: 'Source::infrastructure/ec2.yml'
TemplateConfiguration: 'Source::infrastructure/ec2.json'
InputArtifacts:
- Name: Source
- Name: AMI
RunOrder: 1

Now the AMI containing the application is deployed to production with confidence and without disturbing the users. Try it and run the pipeline!

Summary

Let’s use my production-ready definition to summarize how each point is implemented:

  • Highly available: The load balancer (which is HA) sits in front of a fleet of EC2 instances managed by the Auto Scaling Group for maximum availability. In case of an unhealthy instance, the Auto Scaling Group will replace that instance.
  • Scalable: If the CPU utilization gets over 70%, a Cloud Watch Alarm triggers a Scaling Policy to add new instances automatically.
  • Frictionless deployment: To deploy a new version of the app, a new AMI is created. This AMI is then rolled out to the acceptance environment by updating the CloudFormation stack with the new ImageId parameter. CloudFormation and the Auto Scaling Group perform a rolling update to avoid the application being down during deployment. If the application can not be started the Rolling Update fails and CloudFormation rolls back.
  • Secure: During AMI creation, the latest patches are applied. You must ensure that the pipeline runs often enough to keep up with new patches. Besides that, Security Groups control network traffic to the EC2 instances. When following the bastion host approach, you get maximum security. The EC2 instance is only allowed to send logs to CloudWatch Logs by following the least privileges approach.
  • Operations: All logs are stored in CloudWatch Logs, important metrics are monitored and alarms are defined.

If you now have the impression that running an app on EC2 is a lot of work you are right. In the next two articles, you will learn about other options with fewer responsibilities.

Series

The AWS Velocity series is split into readable chunks. Subscribe to our RSS feed or newsletter to get notified when new articles become available.

AWS Velocity Cover

  1. Set the assembly line up
  2. Local development environment
  3. CI/CD Pipeline as Code
  4. Running your application
  5. EC2 based app
    a. Infrastructure
    b. CI/CD Pipeline (you are here)
  6. Containerized ECS based app
    a. Infrastructure
    b. CI/CD Pipeline
  7. Serverless app
  8. Summary

You can find the source code on GitHub.

Published on


Subscribe to free monthly updates.

One subscriber wins our book AWS in Action every month.

Michael Wittig

Michael Wittig

I’m the author of Amazon Web Services in Action. I work as a software engineer, and independent consultant focused on AWS and DevOps.

Is anything missing in my article? I'm looking forward to your feedback! @hellomichibye or michael@widdix.de.

marbot

Are you part of a highly motivated DevOps team? Use marbot, a friendly chatbot, to forward all kind of alerts from your AWS infrastructure to Slack. Alerts are escalated across your team automatically allowing you to focus on your daily work.

Amazon Web Services in Action

Amazon Web Services in Action introduces you to computing, storing, and networking in the AWS cloud.

Customer Reviews:

Subscribe to free monthly updates.

One subscriber wins our book AWS in Action every month.


Subscribe for free