AWS Velocity Series: CI/CD Pipeline as Code

Most of our clients use AWS to reduce time-to-market following an agile approach. But AWS is only one part of the solution. In this article series, I show you how we help our clients to improve velocity: the time from idea to production.

CI/CD Pipeline as Code

The Continuous Integration / Continuous Deployment pipeline is a major section of your software assembly line. It starts with the code repository and ends with the deployment into your production environment. CI/CD includes many steps that all depend on each other. That’s why this is a valuable area for automation.

AWS Velocity: CI/CD in the software assembly line

The CI/CD pipeline does the following:

  1. runs on every commit into your repository
  2. updates itself if needed
  3. builds the source code
  4. executes unit tests
  5. packages the application as an artifact
  6. creates or updates the acceptance environment if needed
  7. deploys the artifact into the acceptance environment
  8. executes acceptance tests against the acceptance environment
  9. creates or updates the production environment if needed
  10. deploys the artifact into the production environment

That’s a lot of work. Luckily AWS can help us out.

AWS Velocity: CI/CD in the software assembly line

  • AWS CodeCommit is a managed source code repository service.
  • AWS CodeBuild is a fully managed build server that can run anything you need to build and deploy your application within a container.
  • AWS CloudFormation is the Infrastructure as Code service from AWS that can convert YAML or JSON templates into running infrastructure stacks.
  • AWS CodePipeline is a managed service that glues all the above services together in a pipeline.

So what is your job?

  1. track your source code in a repository
  2. describe your pipeline as code
  3. write a script to build, test, and package application
  4. write a script to build, test, and package acceptance test
  5. describe your infrastructure as code

In this article, you will learn how to perform steps 1 to 4. The rest of this series focuses on step 5. In the end, you will be able to push an application to production without any manual work. That’s enough motivation to continue? Let’s get started.

Setting up the git repository

To create an AWS CodeCommit git repository, make sure you have the AWS CLI installed. Execute the following command in your terminal:

export AWS_DEFAULT_REGION=eu-west-1
aws codecommit create-repository --repository-name aws-velocity
#{
#    "repositoryMetadata": {
#        "repositoryName": "aws-velocity", 
#        "cloneUrlSsh": "ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity", 
#        "lastModifiedDate": 1486450175.193, 
#        "repositoryId": "11c6b1ec-95bb-4925-84ac-da9695ac6031", 
#        "cloneUrlHttp": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity", 
#        "creationDate": 1486450175.193, 
#        "Arn": "arn:aws:codecommit:eu-west-1:163732473262:aws-velocity", 
#        "accountId": "163732473262"
#    }
#}

If you use AWS CodeCommit the first time, you need to upload your public SSH key to your IAM user and make sure that the IAM user has access right to CodeCommit. You can grant access to CodeCommit using the managed policy AWSCodeCommitPowerUser.

  1. Open the IAM Dashboard
  2. Click on your user
  3. Select the Security Credentials tab
  4. Click the gray Upload SSH public key button
  5. Insert your public key (mine is located at ~/.ssh/id_rsa.pub)
  6. Click the blue Upload SSH public key button
  7. Copy the SSH key ID of your uploaded public key (e.g. ASFKAAPNGA66RIIWSYMQ).
  8. Select the Permissions tab
  9. Click the blue Add permissions button
  10. Select Attach existing policies directly
  11. Search for AWSCodeCommitPowerUser in the table
  12. Select AWSCodeCommitPowerUser
  13. Click the blue Next: preview button
  14. Confirm by clicking the blue Add permissions button

Now you initialize the git repository locally and push your changes to CodeCommit.

  1. Make sure that you are in the project folder aws-velocity that you created in the previous part of the series.
  2. Run git init
  3. Run echo "node_modules/" > .gitignore
  4. Replace $YourSshKeyId with your SSH key ID and run
    git remote add origin ssh://$YourSshKeyId@git-codecommit.eu-west-1.amazonaws.com/v1/repos/aws-velocity
    
  5. Run git add -A
  6. Run git commit -m 'initial commit'
  7. Run git push -u origin master

The source code is now available in AWS CodeCommit.

Describing the pipeline as code

AWS CloudFormation is the infrastructure as code service on AWS. You can also use CloudFormation to describe a pipeline. CloudFormation is based on templates in YAML or JSON. You will use YAML in the following example. The template has ~200 lines. I will present the template in pieces that you need to copy together to get the running version. Or you can download the file on GitHub.

The first part of the template describes some parameters that make the template reusable. It also contains the S3 bucket that is used to store the compressed artifacts. In the deploy folder, create a file pipeline.yml with the following content:

---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Pipeline'
Parameters:
  RepositoryName:
    Type: String
    Default: 'aws-velocity'
  BranchName:
    Type: String
    Default: 'master'
Resources:
  ArtifactsBucket:
    DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
    DeletionPolicy: Retain
    Type: 'AWS::S3::Bucket'

You also need some IAM roles to allow CodePipeline, CloudFormation, and CodeBuild to access your account. Add the following content to deploy/pipeline.yml:

  PipelineRole:
    DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - 'codepipeline.amazonaws.com'
          Action:
          - 'sts:AssumeRole'
      ManagedPolicyArns:
      - 'arn:aws:iam::aws:policy/AdministratorAccess'
  CloudFormationRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - 'cloudformation.amazonaws.com'
          Action:
          - 'sts:AssumeRole'
      ManagedPolicyArns:
      - 'arn:aws:iam::aws:policy/AdministratorAccess'
  CodeBuildRole:
    DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - 'codebuild.amazonaws.com'
          Action:
          - 'sts:AssumeRole'
      Policies:
      - PolicyName: ServiceRole
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Sid: CloudWatchLogsPolicy
            Effect: Allow
            Action: 
            - 'logs:CreateLogGroup'
            - 'logs:CreateLogStream'
            - 'logs:PutLogEvents'
            Resource: '*'
          - Sid: CodeCommitPolicy
            Effect: Allow
            Action: 'codecommit:GitPull'
            Resource: '*'
          - Sid: S3GetObjectPolicy
            Effect: Allow
            Action: 
            - 's3:GetObject'
            - 's3:GetObjectVersion'
            Resource: '*'
          - Sid: S3PutObjectPolicy
            Effect: 'Allow'
            Action: 's3:PutObject'
            Resource: '*'

You will use CodeBuild to build, test and package the app and the acceptance test. This part also includes the scripts that are run when building, testing, and packaging your app. Add the following content to deploy/pipeline.yml:

  AppProject:
    DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
    Type: 'AWS::CodeBuild::Project'
    Properties:
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        ComputeType: 'BUILD_GENERAL1_SMALL'
        Image: 'amazonlinux:2016.09'
        Type: 'LINUX_CONTAINER'
      Name: !Sub '${AWS::StackName}-app'
      ServiceRole: !GetAtt 'CodeBuildRole.Arn'
      Source:
        Type: CODEPIPELINE
        BuildSpec: |
          version: 0.1
          phases:
            install:
              commands:
              - 'curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -'
              - 'yum -y install nodejs'
            build:
              commands:
              - 'cd app/ && npm install'
              - 'cd app/ && npm test'
            post_build:
              commands:
              - 'rm -rf app/node_modules/'
              - 'rm -rf app/test/'
              - 'cd app/ && npm install --production'
          artifacts:
            files:
            - 'app/**/*'
      TimeoutInMinutes: 10
  AcceptanceProject:
    DependsOn: CloudFormationRole # make sure that CloudFormationRole is deleted last
    Type: 'AWS::CodeBuild::Project'
    Properties:
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        ComputeType: 'BUILD_GENERAL1_SMALL'
        Image: 'amazonlinux:2016.09'
        Type: 'LINUX_CONTAINER'
      Name: !Sub '${AWS::StackName}-acceptance'
      ServiceRole: !GetAtt 'CodeBuildRole.Arn'
      Source:
        Type: CODEPIPELINE
        BuildSpec: |
          version: 0.1
          phases:
            install:
              commands:
              - 'curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -'
              - 'yum -y install nodejs'
            build:
              commands:
              - 'cd acceptance/ && npm install'
              - 'cd acceptance/ && npm test'
            post_build:
              commands:
              - 'rm -rf acceptance/node_modules/'
              - 'rm -rf acceptance/test/'
              - 'cd acceptance/ && npm install --production'
          artifacts:
            files:
            - 'acceptance/**/*'
      TimeoutInMinutes: 10

And finally the CodePipeline pipeline is described. The pipeline connects the dots. From repository, to pipeline update, to triggering the CodeBuild projects. Add the following content to deploy/pipeline.yml:

  Pipeline:
    Type: 'AWS::CodePipeline::Pipeline'
    Properties:
      ArtifactStore:
        Type: S3
        Location: !Ref ArtifactsBucket
      Name: !Ref 'AWS::StackName'
      RestartExecutionOnUpdate: true
      RoleArn: !GetAtt 'PipelineRole.Arn'
      Stages:
      - Name: Source
        Actions:
        - Name: FetchSource
          ActionTypeId:
            Category: Source
            Owner: AWS
            Provider: CodeCommit
            Version: 1
          Configuration:
            RepositoryName: !Ref RepositoryName
            BranchName: !Ref BranchName
          OutputArtifacts:
          - Name: Source
          RunOrder: 1
      - Name: Pipeline
        Actions:
        - Name: DeployPipeline
          ActionTypeId:
            Category: Deploy
            Owner: AWS
            Provider: CloudFormation
            Version: 1
          Configuration:
            ActionMode: CREATE_UPDATE
            Capabilities: CAPABILITY_IAM
            RoleArn: !GetAtt 'CloudFormationRole.Arn'
            StackName: !Ref 'AWS::StackName'
            TemplatePath: 'Source::deploy/pipeline.yml'
            ParameterOverrides: !Sub '{"RepositoryName": "${RepositoryName}", "BranchName": "${BranchName}"}'
          InputArtifacts:
          - Name: Source
          RunOrder: 1
      - Name: Build
        Actions:
        - Name: BuildAndTestApp
          ActionTypeId:
            Category: Build
            Owner: AWS
            Provider: CodeBuild
            Version: 1
          Configuration:
            ProjectName: !Ref AppProject
          InputArtifacts:
          - Name: Source
          OutputArtifacts:
          - Name: App
          RunOrder: 1
        - Name: BuildAndTestAcceptance
          ActionTypeId:
            Category: Build
            Owner: AWS
            Provider: CodeBuild
            Version: 1
          Configuration:
            ProjectName: !Ref AcceptanceProject
          InputArtifacts:
          - Name: Source
          OutputArtifacts:
          - Name: Acceptance
          RunOrder: 1

The Continuous Integration part of the pipeline is now done.

Make sure to commit the changes:

git add -A
git commit -m 'added pipeline template'
git push

Now it’s time to create the pipeline. You only need to execute this command once; the pipeline will update itself in the future.

aws cloudformation create-stack --stack-name aws-velocity-pipeline --template-body file://deploy/pipeline.yml --capabilities CAPABILITY_IAM

Creating the CloudFormation stack that contains the pipeline will take some time. Use the following to command to wait until the stack was created::

aws cloudformation wait stack-create-complete --stack-name aws-velocity-pipeline

Open the AWS CodePipeline Dashboard, select the aws-velocity-pipeline pipeline and wait until all boxes are green.

Continuous Integration pipeline

Now, the Continuous Integration part of the pipeline is done. You can find a copy of the running app in the S3 bucket that starts with aws-velocity-pipeline-artifactsbucket-* under aws-velocity-pipelin/App. Download the compressed file, unzip it, and you will find a copy of your app.

Describing the infrastructure as code

In the rest of this series, you will learn how to describe the infrastructure as code. Depending on your target platform things vary. You can choose from:

  • EC2 based app that runs in an Auto Scaling Group
  • Containerized app that runs on ECS
  • Serverless app

All resources that you created are part of the non-expiring AWS Free Tier. If you have no other pipelines running and do not use more than 100 minutes of build time, you will not be charged by AWS.
You can cleanup all the resources at the very end of this series (coming soon)

Series

This AWS Velocity article series is split into readable chunks. Subscribe to our RSS feed or newsletter to get notified when new articles become available. The chunks are:

  1. Set the assembly line up
  2. Local development environment
  3. CI/CD Pipeline as Code (you are here)
  4. Running your application
  5. EC2 based app (coming soon)
    a. Deploy EC2 based app without downtime (coming soon)
    b. Acceptance Test your EC2 based app (coming soon)
    c. Automate AMI creation and deployment (coming soon)
    d. Monitor your EC2 based app (coming soon)
    e. Cleanup & Summary (coming soon)
  6. Containerized app (coming soon)
    a. Deploy ECS cluster without downtime (coming soon)
    b. Deploy your Docker Container without downtime (coming soon)
    c. Acceptance Test your ECS based app (coming soon)
    d. Automate AMI creation and deployment (coming soon)
    e. Monitor your ECS based app (coming soon)
    e. Cleanup & Summary (coming soon)
  7. Serverless app (coming soon)
    a. Deploy Serverless app without downtime (coming soon)
    b. Acceptance Test your Serverless app (coming soon)
    c. Monitor your Serverless app (coming soon)
    e. Cleanup & Summary (coming soon)
  8. Cleanup & Summary (coming soon)

Read on


Share

           

RSS

  RSS

Newsletter


Michael Wittig

Michael Wittig

I’m the author of Amazon Web Services in Action. I work as a software engineer, and independent consultant focused on AWS and DevOps. Hire me!

Is anything missing in my article? I'm looking forward to your feedback! @hellomichibye or michael@widdix.de.


Published on