Migrating CodePipeline to GitHub Actions to improve performance

Michael Wittig – 28 Oct 2022

Recently, we have become increasingly dissatisfied with the time our AWS CodePipeline pipeline takes to deploy a change to production. The pipeline builds the code, runs unit tests, deploys to a test environment, runs acceptance tests in the test environment, and deploys to production. It takes 27 minutes for a full run of our pipeline—too long for impatient developers like me. We analyzed the performance in detail and decided to migrate to GitHub Actions.

Impatient developer

Read on to learn about the pitfalls and reasons for the migration.

Pipeline

Before we start, here is an overview of the pipeline (click on the image for a huge CodePipeline screenshot). The pipeline deploys our serverless application: ChatOps for AWS - marbot.

marbot's high-level pipeline

Why CodePipeline and CodeBuild are slow

The following table lists the runtime of each stage of our CodePipeline pipeline.

Stage Runtime (mm:ss)
Commit 07:01
Acceptance 09:35
Production 09:56
Total 26:32

We identified the following performance issues:

  1. CodePipeline adds a tiny but noticeable overhead with each action. E.g., deploying a CloudFormation stack is usually a second slower than calling the CloudFormation API directly. This adds up if you run actions in sequence because of dependencies (e.g., database deployment must happen before app deployment).
  2. CodePipeline only orchestrates the pipeline. To run a script, you need to integrate CodePipeline with CodeBuild. CodeBuild adds significant overhead. Queuing times of 60 seconds and provisioning times of 10 seconds are not unusual (us-east-1). We use seven CodeBuild actions and see more than 7 minutes of overhead because of that.
  3. We use aws cloudformation package and the CloudFormation transform AWS::Serverless-2016-10-31, aka SAM, to deploy our application. aws cloudformation package uses a simple implementation to detect code changes that trigger a deployment of Lambda functions. We use esbuild to build our source code. Even if the content of the output files stay the same, the changing file creation date triggers a deployment.

Issues 1 and 2 are easier to fix with migrating to a different technology. Issue 2 could be partly addressed by combining multiple CodeBuild projects into one, making it harder to identify what went wrong quickly (downloading dependencies failed, tests failed, package failed, …) without checking the logs. Issue 3 has nothing to do with CodePipeline. To address issues 1 and 2, we decided to migrate to GitHub Actions. I also found a workaround for issue 3.

After we decided to deploy our AWS infrastructure and serverless app with GitHub Actions instead of CodePipeline, we learned a few things we want to share with you.

Artifacts

In CodePipeline, each action takes input artifacts (exceptions are the source actions at the very beginning of the pipeline) and optionally produces output artifacts. You must ensure (via runOrder) that action output artifacts are created before they are used as input artifacts in other actions.

GitHub Actions works differently. First, there is a difference between a job and a step. A job typically consists of multiple steps. All steps share the same runner environment and can access files created by previous steps (like in a CodeBuild build). If you want to pass artifacts from one job to another, you must use the GitHub Action upload-artifact and download-artifact.

AWS Authentication

Our pipeline deploys both: our serverless application and the required AWS infrastructure. Therefore, the pipeline requires access to the AWS APIs to configure the API Gateway or Lambda. Consequently, we are using AWS IAM to configure access permissions via policies.

In CodePipeline, you define the IAM role (via roleArn) that CodePipeline assumes to run your pipeline. Keep in mind that each CodeBuild project requires its own IAM Role.

In GitHub Actions, you also define the IAM role, but you need a small piece of AWS infrastructure to make it work called an OpenID Connect (OIDC) identity provider, part of the IAM service.

  1. Check out our CloudFormation template to set everything up in a minute.

  2. Add the following permissions to your workflows at the top level:

    permissions:
    id-token: write
    contents: read
  3. Use the GitHub Action aws-actions/configure-aws-credentials to assume the role in your workflow like this:

    permissions: {} # shortened
    jobs:
    demo:
    runs-on: ubuntu-22.04
    steps:
    - uses: actions/checkout@v3
    - uses: aws-actions/configure-aws-credentials@v1-node16
    with:
    role-to-assume: arn:aws:iam::123456789012:role/github-openid-connect
    role-session-name: github-actions
    aws-region: us-east-1

Running a script

I already mentioned that CodePipeline only orchestrates the pipeline. To run a script, you need to integrate CodePipeline with CodeBuild. The following CloudFormation snippet shows the required AWS resources and the mind-blowing complexity:


Looking for a new challenge?

  • DEMICON

    Senior Lead Full Stack Developer

    DEMICON • AWS Advanced Consulting Partner • Remote (Europe)
    AWS JavaScript/TypeScript Angular React
  • tecRacer

    Cloud Consultant • AWS DevOps

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Portugal, and Switzerland
    Infrastructure as Code Container Continuous Deployment

Resources:
ArtifactsBucket:
Type: 'AWS::S3::Bucket'
Properties: {} # shortened
CodeBuildRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: 'codebuild.amazonaws.com'
Action: 'sts:AssumeRole'
Policies: [] # shortened
UnitTestProject:
Type: 'AWS::CodeBuild::Project'
Properties:
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: 'BUILD_GENERAL1_SMALL'
Image: 'aws/codebuild/standard:6.0'
Type: 'LINUX_CONTAINER'
Name: !Sub '${AWS::StackName}-unit-test'
ServiceRole: !GetAtt 'CodeBuildRole.Arn'
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: '0.2'
phases:
install:
runtime-versions:
nodejs: 16
build:
commands:
- 'npm ci'
- 'npm run test-with-results'
artifacts:
files:
- 'test-results.xml'
- 'coverage/**/*'
enable-symlinks: yes
TimeoutInMinutes: 5
PipelineRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: 'codepipeline.amazonaws.com'
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
Pipeline:
Type: 'AWS::CodePipeline::Pipeline'
Properties:
ArtifactStore:
Type: S3
Location: !Ref ArtifactsBucket
Name: !Ref 'AWS::StackName'
RestartExecutionOnUpdate: true
RoleArn: !GetAtt 'PipelineRole.Arn'
Stages:
- Name: Source
Actions:
- Name: FetchSource
# shortened
OutputArtifacts:
- Name: Source
RunOrder: 1
- Name: 'Commit-Test'
Actions:
- Name: Test
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: '1'
Configuration:
ProjectName: !Ref UnitTestProject
InputArtifacts:
- Name: Source
OutputArtifacts:
- Name: UnitTestReports
RunOrder: 1
PipelineTriggerRole:
Type: 'AWS::IAM::Role'
Properties: {} # shortened
PipelineTriggerRule:
Type: 'AWS::Events::Rule'
Properties:
EventPattern:
source:
- 'aws.codecommit'
'detail-type':
- 'CodeCommit Repository State Change'
resources:
- !Sub 'arn:aws:codecommit:${AWS::Region}:${AWS::AccountId}:marbot'
detail:
referenceType:
- branch
referenceName:
- master
State: ENABLED
Targets:
- Arn: !Sub 'arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}'
Id: pipeline
RoleArn: !GetAtt 'PipelineTriggerRole.Arn'

GitHub Actions has native support for running scripts. The following workflow installs Node.js dependencies via npm and runs unit tests on each push to master. The test reports are archived as an artifact and can be inspected manually:

name: unit-test
on:
push:
branches:
- master
defaults:
run:
shell: bash
jobs:
unit-test:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 16
cache: npm
- name: unit-test
run: |
npm ci
npm run test-with-results
- uses: actions/upload-artifact@v3
with:
name: unit-test-reports
path: |
test-results.xml
coverage/**/*

The default shell for CodeBuild and GitHub Actions

CodeBuild defaults to sh and stops when a “command” from the commands list fails.

GitHub Actions defaults to bash with the following settings:

  • -e: Exit immediately if a command exits with a non-zero status.

I usually set the default shell to bash in my workflows at the top level:

defaults:
run:
shell: bash

GitHub Actions now runs bash with the following settings:

  • --noprofile: Do not read either the system-wide startup file /etc/profileor any of the personal initialization files/.bash_profile, /.bash_login, or ~/.profile`.
  • --norc: Do not read and execute the personal initialization file ~/.bashrc if the shell is interactive.
  • -e: Exit immediately if a command exits with a non-zero status.
  • -o pipefail: The return value of a pipeline is the status of the last command to exit with a non-zero status, or zero if no command exited with a non-zero status

AWS Integrations

CodePiepline integrates with a bunch of AWS services natively:

  • Amazon ECR
  • Amazon ECS
  • Amazon S3
  • AWS CloudFormation
  • AWS CodeBuild
  • AWS CodeCommit
  • AWS CodeDeploy
  • AWS Device Farm
  • AWS Elastic Beanstalk
  • AWS Lambda
  • AWS OpsWorks Stacks
  • AWS Service

AWS offers the following actions for GitHub Actions (some of them are not maintained):

On top of that, you can find many 3rd party actions in the GitHub Marketplace.

For example, we forked the official AWS CloudFormation action and added parallel stack deployments.

Parallelization

We learned that executing steps in parallel is not so easy in GitHub Actions. CodePipline provides much better support for that! That’s why we added parallel stack deployments to our fork of the official AWS CloudFormation action.

Reusability

In CodePipeline, there is no way to reuse a stage. My Acceptance and Production stages are very similar. If I make a change to one of them, I have to remember to make the change to the other stage as well. There is no way to reuse a stage or a pipeline.

GitHub Actions provides a way to reuse workflows.

I have created a deploy workflow like this with an input to indicate if dev or prod should be deployed:

---
name: deploy
on:
workflow_call:
inputs:
stage:
required: true
type: string
jobs:
deploy:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
# shortened
- name: cloudformation-dashboard
uses: widdix/aws-cloudformation-github-deploy@v2
with:
name: marbot-${{ inputs.stage }}-dashboard
template: template-dashboard.yml
parameter-overrides: ParentApiStack=marbot-${{ inputs.stage }}-api,Stage=${{ inputs.stage }}
- name: smoke-test
if: inputs.stage == 'dev'
run: |
npm run newman

Manual approval

CodePipeline supports manual approvals.

Unfortunately, GitHub Actions supports manual approvals only for organizations subscribed to the Enterprise plan.

Our workaround for marbot is this:

  • We deploy to dev when the master branch changes.
  • We create a tag v1.y.z to deploy to prod.

The main limitation of this workaround is that we have to use two workflows. There is no easy way to share artifacts between workflows (workarounds exist). For now, we run npm ci && npm run build twice and hope that the outcome is the same.

Fixing aws cloudformation package

If you agree with me that only a change in the content of a file/files is relevant to decide if a new deployment of a Lambda function is needed, you can add the following line before your aws cloudformation package command (assuming your build output is stored in the build folder):

find build/ -exec touch -m --date="2020-01-01" {} \;

The above command will set the creation time of all files inside the build folder to 2020-01-01 (the date doesn’t matter as long as it stays constant). Therefore, aws cloudformation package will only trigger a deployment of the Lambda function in case you made changes to your code.

Outcome

marbot's GitHub Actions pipeline

Migrating from CodePipeline to GitHub Actions and optimizing the deployment process reduced the deployment time from 27 minutes to 9 minutes (down 66%). Optimizing the existing CodePipeline pipeline would have yielded 20-50% performance improvements. We are delighted with the outcome of the migration.

Become a cloudonaut supporter

Michael Wittig

Michael Wittig ( Email Twitter LinkedIn Mastodon )

We launched the cloudonaut blog in 2015. Since then, we have published 366 articles, 60 podcast episodes, and 58 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Christopher Hipwell, e9e4e5f0faef, Jason Yorty, Jeff Finley, jhoadley, Johannes Grumböck, Johannes Konings, John Culkin, Jonas Mellquist, Jonathan Deamer, Juraj Martinka, Ken Snyder, Markus Ellers, Oriol Rodriguez, Ross Mohan, Ross Mohan, sam onaga, Satyendra Sharma, Simon Devlin, Thorsten Hoeger, Todd Valentine, Victor Grenu, waldensystems, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.