AWS Velocity Series: CI/CD Pipeline as Code

The Continuous Integration / Continuous Deployment pipeline is a major section of your software assembly line. It starts with the code repository and ends with the deployment into your production environment. CI/CD includes many steps that all depend on each other. That’s why this is a valuable area for automation.

AWS Velocity: CI/CD in the software assembly line

The CI/CD pipeline does the following:

  1. runs on every commit into your repository
  2. updates itself if needed
  3. builds the source code
  4. executes unit tests
  5. packages the application as an artifact
  6. creates or updates the acceptance environment if needed
  7. deploys the artifact into the acceptance environment
  8. executes acceptance tests against the acceptance environment
  9. creates or updates the production environment if needed
  10. deploys the artifact into the production environment

That’s a lot of work. Luckily AWS can help us out.

AWS Velocity: CI/CD in the software assembly line

  • AWS CodeCommit is a managed source code repository service.
  • AWS CodeBuild is a fully managed build server that can run anything you need to build and deploy your application within a container.
  • AWS CloudFormation is the Infrastructure as Code service from AWS that can convert YAML or JSON templates into running infrastructure stacks.
  • AWS CodePipeline is a managed service that glues all the above services together in a pipeline.

So what is your job?

  1. track your source code in a repository
  2. describe your pipeline as code
  3. write a script to build, test, and package application
  4. write a script to build, test, and package acceptance test
  5. describe your infrastructure as code

In this article, you will learn how to perform steps 1 to 4. The rest of this series focuses on step 5. In the end, you will be able to push an application to production without any manual work. That’s enough motivation to continue?

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Let’s get started.

Setting up the git repository

To create an AWS CodeCommit git repository, make sure you have the AWS CLI installed. Execute the following command in your terminal:

export AWS_DEFAULT_REGION=eu-west-1
aws codecommit create-repository --repository-name aws-velocity
#{
# "repositoryMetadata": {
# "repositoryName": "aws-velocity",
# "cloneUrlSsh": "ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity",
# "lastModifiedDate": 1486450175.193,
# "repositoryId": "11c6b1ec-95bb-4925-84ac-da9695ac6031",
# "cloneUrlHttp": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity",
# "creationDate": 1486450175.193,
# "Arn": "arn:aws:codecommit:eu-west-1:163732473262:aws-velocity",
# "accountId": "163732473262"
# }
#}

If you use AWS CodeCommit the first time, you need to upload your public SSH key to your IAM user and make sure that the IAM user has access right to CodeCommit. You can grant access to CodeCommit using the managed policy AWSCodeCommitPowerUser.

  1. Open the IAM Dashboard
  2. Click on your user
  3. Select the Security Credentials tab
  4. Click the gray Upload SSH public key button
  5. Insert your public key (mine is located at ~/.ssh/id_rsa.pub)
  6. Click the blue Upload SSH public key button
  7. Copy the SSH key ID of your uploaded public key (e.g. ASFKAAPNGA66RIIWSYMQ).
  8. Select the Permissions tab
  9. Click the blue Add permissions button
  10. Select Attach existing policies directly
  11. Search for AWSCodeCommitPowerUser in the table
  12. Select AWSCodeCommitPowerUser
  13. Click the blue Next: preview button
  14. Confirm by clicking the blue Add permissions button

Now you initialize the git repository locally and push your changes to CodeCommit.

  1. Make sure that you are in the project folder aws-velocity that you created in the previous part of the series.
  2. Run git init
  3. Run echo "node_modules/" > .gitignore
  4. Replace $YourSshKeyId with your SSH key ID and run

    git remote add origin ssh://$YourSshKeyId@git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity
  5. Run git add -A

  6. Run git commit -m 'initial commit'
  7. Run git push -u origin master

The source code is now available in AWS CodeCommit.

Describing the pipeline as code

AWS CloudFormation is the infrastructure as code service on AWS. You can also use CloudFormation to describe a pipeline. CloudFormation is based on templates in YAML or JSON. You will use YAML in the following example. The template has ~200 lines. I will present the template in pieces that you need to copy together to get the running version. Or you can download the file on GitHub.

The first part of the template describes some parameters that make the template reusable. It also contains the S3 bucket that is used to store the compressed artifacts. In the deploy folder, create a file pipeline.yml with the following content:

deploy/pipeline.ymlGitHub
---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Pipeline'
Parameters:
RepositoryName:
Type: String
Default: 'aws-velocity'
BranchName:
Type: String
Default: 'master'
Resources:
ArtifactsBucket:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
DeletionPolicy: Retain
Type: 'AWS::S3::Bucket'

You also need some IAM roles to allow CodePipeline, CloudFormation, and CodeBuild to access your account. Add the following content to deploy/pipeline.yml:

deploy/pipeline.ymlGitHub
PipelineRole:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- 'codepipeline.amazonaws.com'
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
CloudFormationRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- 'cloudformation.amazonaws.com'
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
CodeBuildRole:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- 'codebuild.amazonaws.com'
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: ServiceRole
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: CloudWatchLogsPolicy
Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
- Sid: CodeCommitPolicy
Effect: Allow
Action: 'codecommit:GitPull'
Resource: '*'
- Sid: S3GetObjectPolicy
Effect: Allow
Action:
- 's3:GetObject'
- 's3:GetObjectVersion'
Resource: '*'
- Sid: S3PutObjectPolicy
Effect: 'Allow'
Action: 's3:PutObject'
Resource: '*'

You will use CodeBuild to build, test and package the app and the acceptance test. This part also includes the scripts that are run when building, testing, and packaging your app. Add the following content to deploy/pipeline.yml:

deploy/pipeline.ymlGitHub
AppProject:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::CodeBuild::Project'
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: 'BUILD_GENERAL1_SMALL'
Image: 'aws/codebuild/nodejs:6.3.1'
Type: 'LINUX_CONTAINER'
Name: !Sub '${AWS::StackName}-app'
ServiceRole: !GetAtt 'CodeBuildRole.Arn'
Source:
Type: CODEPIPELINE
BuildSpec: |
version: 0.1
phases:
build:
commands:
- 'cd app/ && npm install'
- 'cd app/ && npm test'
post_build:
commands:
- 'rm -rf app/node_modules/'
- 'rm -rf app/test/'
- 'cd app/ && npm install --production'
artifacts:
files:
- 'app/**/*'
TimeoutInMinutes: 10
AcceptanceProject:
DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
Type: 'AWS::CodeBuild::Project'
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: 'BUILD_GENERAL1_SMALL'
Image: 'aws/codebuild/nodejs:6.3.1'
Type: 'LINUX_CONTAINER'
Name: !Sub '${AWS::StackName}-acceptance'
ServiceRole: !GetAtt 'CodeBuildRole.Arn'
Source:
Type: CODEPIPELINE
BuildSpec: |
version: 0.1
phases:
build:
commands:
- 'cd acceptance/ && npm install'
- 'cd acceptance/ && npm test'
post_build:
commands:
- 'rm -rf acceptance/node_modules/'
- 'rm -rf acceptance/test/'
- 'cd acceptance/ && npm install --production'
artifacts:
files:
- 'acceptance/**/*'
TimeoutInMinutes: 10

And finally the CodePipeline pipeline is described. The pipeline connects the dots. From repository, to pipeline update, to triggering the CodeBuild projects. Add the following content to deploy/pipeline.yml:

deploy/pipeline.ymlGitHub
Pipeline:
Type: 'AWS::CodePipeline::Pipeline'
Properties:
ArtifactStore:
Type: S3
Location: !Ref ArtifactsBucket
Name: !Ref 'AWS::StackName'
RestartExecutionOnUpdate: true
RoleArn: !GetAtt 'PipelineRole.Arn'
Stages:
- Name: Source
Actions:
- Name: FetchSource
ActionTypeId:
Category: Source
Owner: AWS
Provider: CodeCommit
Version: 1
Configuration:
RepositoryName: !Ref RepositoryName
BranchName: !Ref BranchName
OutputArtifacts:
- Name: Source
RunOrder: 1
- Name: Pipeline
Actions:
- Name: DeployPipeline
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt 'CloudFormationRole.Arn'
StackName: !Ref 'AWS::StackName'
TemplatePath: 'Source::deploy/pipeline.yml'
ParameterOverrides: !Sub '{"RepositoryName": "${RepositoryName}", "BranchName": "${BranchName}"}'
InputArtifacts:
- Name: Source
RunOrder: 1
- Name: Build
Actions:
- Name: BuildAndTestApp
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref AppProject
InputArtifacts:
- Name: Source
OutputArtifacts:
- Name: App
RunOrder: 1
- Name: BuildAndTestAcceptance
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref AcceptanceProject
InputArtifacts:
- Name: Source
OutputArtifacts:
- Name: Acceptance
RunOrder: 1

The Continuous Integration part of the pipeline is now done.

Make sure to commit the changes:

git add -A
git commit -m 'added pipeline template'
git push

Now it’s time to create the pipeline. You only need to execute this command once; the pipeline will update itself in the future.

aws cloudformation create-stack --stack-name aws-velocity-pipeline --template-body file://deploy/pipeline.yml --capabilities CAPABILITY_IAM

Creating the CloudFormation stack that contains the pipeline will take some time. Use the following to command to wait until the stack was created::

aws cloudformation wait stack-create-complete --stack-name aws-velocity-pipeline

Open the AWS CodePipeline Dashboard, select the aws-velocity-pipeline pipeline and wait until all boxes are green.

Continuous Integration pipeline

Now, the Continuous Integration part of the pipeline is done. You can find a copy of the running app in the S3 bucket that starts with aws-velocity-pipeline-artifactsbucket-* under aws-velocity-pipelin/App. Download the compressed file, unzip it, and you will find a copy of your app.

Describing the infrastructure as code

In the rest of this series, you will learn how to describe the infrastructure as code. Depending on your target platform things vary. You can choose from:

  • EC2 based app that runs in an Auto Scaling Group
  • Containerized app that runs on ECS
  • Serverless app

All resources that you created are part of the non-expiring AWS Free Tier. If you have no other pipelines running and do not use more than 100 minutes of build time, you will not be charged by AWS.
You can cleanup all the resources at the very end of this series (coming soon)

Series

The AWS Velocity series is split into readable chunks. Subscribe to our RSS feed or newsletter to get notified when new articles become available.

AWS Velocity Cover

  1. Set the assembly line up
  2. Local development environment
  3. CI/CD Pipeline as Code (you are here)
  4. Running your application
  5. EC2 based app
    a. Infrastructure
    b. CI/CD Pipeline
  6. Containerized ECS based app
    a. Infrastructure
    b. CI/CD Pipeline
  7. Serverless app
  8. Summary

You can find the source code on GitHub.

Published on


Subscribe to free monthly updates.

One subscriber wins our book AWS in Action every month.

Michael Wittig

Michael Wittig

I’m the author of Amazon Web Services in Action. I work as a software engineer, and independent consultant focused on AWS and DevOps.

Is anything missing in my article? I'm looking forward to your feedback! @hellomichibye or michael@widdix.de.

Amazon Web Services in Action

Amazon Web Services in Action introduces you to computing, storing, and networking in the AWS cloud.

Customer Reviews:

Subscribe to free monthly updates.

One subscriber wins our book AWS in Action every month.


Subscribe for free