AWS Velocity Series: CI/CD Pipeline as Code

The Continuous Integration / Continuous Deployment pipeline is a major section of your software assembly line. It starts with the code repository and ends with the deployment into your production environment. CI/CD includes many steps that all depend on each other. That’s why this is a valuable area for automation.

The CI/CD pipeline does the following:

1. runs on every commit into your repository
2. updates itself if needed
3. builds the source code
4. executes unit tests
5. packages the application as an artifact
6. creates or updates the acceptance environment if needed
7. deploys the artifact into the acceptance environment
8. executes acceptance tests against the acceptance environment
9. creates or updates the production environment if needed
10. deploys the artifact into the production environment

That’s a lot of work. Luckily AWS can help us out.

• AWS CodeCommit is a managed source code repository service.
• AWS CodeBuild is a fully managed build server that can run anything you need to build and deploy your application within a container.
• AWS CloudFormation is the Infrastructure as Code service from AWS that can convert YAML or JSON templates into running infrastructure stacks.
• AWS CodePipeline is a managed service that glues all the above services together in a pipeline.

So what is your job?

1. track your source code in a repository
2. describe your pipeline as code
3. write a script to build, test, and package application
4. write a script to build, test, and package acceptance test
5. describe your infrastructure as code

In this article, you will learn how to perform steps 1 to 4. The rest of this series focuses on step 5. In the end, you will be able to push an application to production without any manual work. That’s enough motivation to continue?

AWS Velocity Series

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production. Discover all posts!

Let’s get started.

Setting up the git repository

To create an AWS CodeCommit git repository, make sure you have the AWS CLI installed. Execute the following command in your terminal:

export AWS_DEFAULT_REGION=eu-west-1
aws codecommit create-repository --repository-name aws-velocity
#{
#    "repositoryMetadata": {
#        "repositoryName": "aws-velocity", 
#        "cloneUrlSsh": "ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity", 
#        "lastModifiedDate": 1486450175.193, 
#        "repositoryId": "11c6b1ec-95bb-4925-84ac-da9695ac6031", 
#        "cloneUrlHttp": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity", 
#        "creationDate": 1486450175.193, 
#        "Arn": "arn:aws:codecommit:eu-west-1:163732473262:aws-velocity", 
#        "accountId": "163732473262"
#    }
#}

If you use AWS CodeCommit the first time, you need to upload your public SSH key to your IAM user and make sure that the IAM user has access right to CodeCommit. You can grant access to CodeCommit using the managed policy AWSCodeCommitPowerUser.

1. Open the IAM Dashboard
2. Click on your user
3. Select the Security Credentials tab
4. Click the gray Upload SSH public key button
5. Insert your public key (mine is located at ~/.ssh/id_rsa.pub)
6. Click the blue Upload SSH public key button
7. Copy the SSH key ID of your uploaded public key (e.g. ASFKAAPNGA66RIIWSYMQ).
8. Select the Permissions tab
9. Click the blue Add permissions button
10. Select Attach existing policies directly
11. Search for AWSCodeCommitPowerUser in the table
12. Select AWSCodeCommitPowerUser
13. Click the blue Next: preview button
14. Confirm by clicking the blue Add permissions button

Now you initialize the git repository locally and push your changes to CodeCommit.

1. Make sure that you are in the project folder aws-velocity that you created in the previous part of the series.

2. Run git init

3. Run echo "node_modules/" > .gitignore

4. Replace $YourSshKeyId with your SSH key ID and run

   git remote add origin ssh://$YourSshKeyId@git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity

5. Run git add -A

6. Run git commit -m 'initial commit'

7. Run git push -u origin master

The source code is now available in AWS CodeCommit.

Describing the pipeline as code

AWS CloudFormation is the infrastructure as code service on AWS. You can also use CloudFormation to describe a pipeline. CloudFormation is based on templates in YAML or JSON. You will use YAML in the following example. The template has ~200 lines. I will present the template in pieces that you need to copy together to get the running version. Or you can download the file on GitHub.

The first part of the template describes some parameters that make the template reusable. It also contains the S3 bucket that is used to store the compressed artifacts. In the deploy folder, create a file pipeline.yml with the following content:

---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Pipeline'
Parameters:
  RepositoryName:
    Type: String
    Default: 'aws-velocity'
  BranchName:
    Type: String
    Default: 'master'
Resources:
  ArtifactsBucket:
    DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
    DeletionPolicy: Retain
    Type: 'AWS::S3::Bucket'

You also need some IAM roles to allow CodePipeline, CloudFormation, and CodeBuild to access your account. Add the following content to deploy/pipeline.yml:

PipelineRole:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'codepipeline.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    ManagedPolicyArns:
    - 'arn:aws:iam::aws:policy/AdministratorAccess'
CloudFormationRole:
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'cloudformation.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    ManagedPolicyArns:
    - 'arn:aws:iam::aws:policy/AdministratorAccess'
CodeBuildRole:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - 'codebuild.amazonaws.com'
        Action:
        - 'sts:AssumeRole'
    Policies:
    - PolicyName: ServiceRole
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Sid: CloudWatchLogsPolicy
          Effect: Allow
          Action: 
          - 'logs:CreateLogGroup'
          - 'logs:CreateLogStream'
          - 'logs:PutLogEvents'
          Resource: '*'
        - Sid: CodeCommitPolicy
          Effect: Allow
          Action: 'codecommit:GitPull'
          Resource: '*'
        - Sid: S3GetObjectPolicy
          Effect: Allow
          Action: 
          - 's3:GetObject'
          - 's3:GetObjectVersion'
          Resource: '*'
        - Sid: S3PutObjectPolicy
          Effect: 'Allow'
          Action: 's3:PutObject'
          Resource: '*'

You will use CodeBuild to build, test and package the app and the acceptance test. This part also includes the scripts that are run when building, testing, and packaging your app. Add the following content to deploy/pipeline.yml:

AppProject:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::CodeBuild::Project'
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: 'BUILD_GENERAL1_SMALL'
      Image: 'aws/codebuild/nodejs:6.3.1'
      Type: 'LINUX_CONTAINER'
    Name: !Sub '${AWS::StackName}-app'
    ServiceRole: !GetAtt 'CodeBuildRole.Arn'
    Source:
      Type: CODEPIPELINE
      BuildSpec: |
        version: 0.1
        phases:
          build:
            commands:
            - 'cd app/ && npm install'
            - 'cd app/ && npm test'
            - 'rm -rf app/node_modules/'
            - 'rm -rf app/test/'
            - 'cd app/ && npm install --production'
        artifacts:
          files:
          - 'app/**/*'
    TimeoutInMinutes: 10
AcceptanceProject:
  DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
  Type: 'AWS::CodeBuild::Project'
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Environment:
      ComputeType: 'BUILD_GENERAL1_SMALL'
      Image: 'aws/codebuild/nodejs:6.3.1'
      Type: 'LINUX_CONTAINER'
    Name: !Sub '${AWS::StackName}-acceptance'
    ServiceRole: !GetAtt 'CodeBuildRole.Arn'
    Source:
      Type: CODEPIPELINE
      BuildSpec: |
        version: 0.1
        phases:
          build:
            commands:
            - 'cd acceptance/ && npm install'
            - 'cd acceptance/ && npm test'
            - 'rm -rf acceptance/node_modules/'
            - 'rm -rf acceptance/test/'
            - 'cd acceptance/ && npm install --production'
        artifacts:
          files:
          - 'acceptance/**/*'
    TimeoutInMinutes: 10

And finally the CodePipeline pipeline is described. The pipeline connects the dots. From repository, to pipeline update, to triggering the CodeBuild projects. Add the following content to deploy/pipeline.yml:

Pipeline:
  Type: 'AWS::CodePipeline::Pipeline'
  Properties:
    ArtifactStore:
      Type: S3
      Location: !Ref ArtifactsBucket
    Name: !Ref 'AWS::StackName'
    RestartExecutionOnUpdate: true
    RoleArn: !GetAtt 'PipelineRole.Arn'
    Stages:
    - Name: Source
      Actions:
      - Name: FetchSource
        ActionTypeId:
          Category: Source
          Owner: AWS
          Provider: CodeCommit
          Version: 1
        Configuration:
          RepositoryName: !Ref RepositoryName
          BranchName: !Ref BranchName
        OutputArtifacts:
        - Name: Source
        RunOrder: 1
    - Name: Pipeline
      Actions:
      - Name: DeployPipeline
        ActionTypeId:
          Category: Deploy
          Owner: AWS
          Provider: CloudFormation
          Version: 1
        Configuration:
          ActionMode: CREATE_UPDATE
          Capabilities: CAPABILITY_IAM
          RoleArn: !GetAtt 'CloudFormationRole.Arn'
          StackName: !Ref 'AWS::StackName'
          TemplatePath: 'Source::deploy/pipeline.yml'
          ParameterOverrides: !Sub '{"RepositoryName": "${RepositoryName}", "BranchName": "${BranchName}"}'
        InputArtifacts:
        - Name: Source
        RunOrder: 1
    - Name: Build
      Actions:
      - Name: BuildAndTestApp
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AppProject
        InputArtifacts:
        - Name: Source
        OutputArtifacts:
        - Name: App
        RunOrder: 1
      - Name: BuildAndTestAcceptance
        ActionTypeId:
          Category: Build
          Owner: AWS
          Provider: CodeBuild
          Version: 1
        Configuration:
          ProjectName: !Ref AcceptanceProject
        InputArtifacts:
        - Name: Source
        OutputArtifacts:
        - Name: Acceptance
        RunOrder: 1

The Continuous Integration part of the pipeline is now done.

Make sure to commit the changes:

git add -A
git commit -m 'added pipeline template'
git push

Now it’s time to create the pipeline. You only need to execute this command once; the pipeline will update itself in the future.

aws cloudformation create-stack --stack-name aws-velocity-pipeline --template-body file://deploy/pipeline.yml --capabilities CAPABILITY_IAM

Creating the CloudFormation stack that contains the pipeline will take some time. Use the following to command to wait until the stack was created::

aws cloudformation wait stack-create-complete --stack-name aws-velocity-pipeline

Open the AWS CodePipeline Dashboard, select the aws-velocity-pipeline pipeline and wait until all boxes are green.

Now, the Continuous Integration part of the pipeline is done. You can find a copy of the running app in the S3 bucket that starts with aws-velocity-pipeline-artifactsbucket-* under aws-velocity-pipelin/App. Download the compressed file, unzip it, and you will find a copy of your app.

Describing the infrastructure as code

In the rest of this series, you will learn how to describe the infrastructure as code. Depending on your target platform things vary. You can choose from:

• EC2 based app that runs in an Auto Scaling Group
• Containerized app that runs on ECS
• Serverless app

All resources that you created are part of the non-expiring AWS Free Tier. If you have no other pipelines running and do not use more than 100 minutes of build time, you will not be charged by AWS.
You can cleanup all the resources at the very end of this series (coming soon)

Series

1. Set the assembly line up
2. Local development environment
3. CI/CD Pipeline as Code (you are here)
4. Running your application
5. EC2 based app
   a. Infrastructure
   b. CI/CD Pipeline
6. Containerized ECS based app
   a. Infrastructure
   b. CI/CD Pipeline
7. Serverless app
8. Summary

You can find the source code on GitHub.