GitHub process automation: A bot to build pull requests authorized by core maintainers

We are maintaining multiple Open Source projects where we focus on CloudFormation templates. To ensure that the templates are of high quality, we leverage automated testing and pull requests. We could use the CodeBuild GitHub integration to execute yamllint, cfn-lint, and also aws cloudformation validate-template to make sure that a pull request is not breaking the templates. That’s a great way to allow contributions while ensuring high quality. You can also use Travis CI and others to run the static tests on each commit / pull request.

However, the mentioned tools only inspect the templates. They don’t create an actual CloudFormation stack. To be 100% sure that a pull request is not breaking any template, we have to create some CloudFormation stacks with the changes from the pull request. Creating a CloudFormation stack causes AWS costs (the actual AWS resources cause the costs, not CloudFormation). Usually, we spent 500 - 1000 USD per month to run automated tests on our templates.

The problem: If we create stacks automatically for every pull request, someone could open hundreds of pull requests and waste our money.

The solution: Let only core committers trigger the tests by commenting on the pull request.

You might come up with other processes based on GitHub Pull Requests that can be automated. E.g., you might want to push each change to a replication of the GitHub repository to CodeCommit. Read on if you are interested in the implementation using serverless technologies. You can find the full source code on GitHub.

Serverless architecture

The diagram was created with Cloudcraft - Visualize your cloud architecture like a pro.

GitHub triggers an HTTPS request for every comment made in a repository to an endpoint that you can define (aka webhook). You can create an HTTPS endpoint with API Gateway and forward the HTTPS request to a Lambda function.

The Lambda function inspects the GitHub data and kicks off a CodeBuild build.

To provide feedback to GitHub users, we watch CodeBuild with a CloudWatch Event Rule. Whenever a build status changes we trigger another Lambda function and signal the information back to GitHub.

The high-level architecture is described. Now it’s time to turn the plan into reality.

Serverless implementation

We recently released a new open source project to speed up CloudFormation projects in a modular way. I will use cfn-modules in the following implementation. Check out the details if you are interested: cfn-modules: Rapid CloudFormation. I will use cfn-modules in the following implementation.

First, you set up the project.

I assume that you have Node.js and the AWS CLI installed.

Create a new directory:

mkdir github-bot
cd github-bot/

cfn-modules relies on npm to install and update modules. Initialize the project with the following command:

npm init -y

Now it’s time to start adding the first modules.

Alerting

You should always monitor your AWS resources. Let’s start with that.

Run the following command to install the first cfn-module:

npm i @cfn-modules/alerting -P

Create a template.yml file with the following content:

---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'GitHub bot'
Resources:
  Alerting:
    Type: 'AWS::CloudFormation::Stack'
    Properties:
      Parameters:
        mail: 'your@mail.com'
        #HttpsEndpoint: '' # check out https://marbot.io/ if you look for something better than email alerts!
      TemplateURL: './node_modules/@cfn-modules/alerting/module.yml'

The alerting module provides an SNS topic with a topic policy that allows most AWS resources to send messages for alerting purposes.

Next, you have to define the CodeBuild project that executes the actual tests.

CodeBuild

I use CodeBuild in an unusual way. Usually, CodeBuild downloads the source for you. In this case, I use a dummy CodeCommit repository to satisfy this requirement. But the actual code for testing is downloaded with git clone and the commit information are passed in via an environment variables (GITHUB_OWNER, GITHUB_REPO, GITHUB_SHA).

Resources:
  # [...]
  Repository: # we only need this because the CodeBuild project cannot be created without a source.
    Type: 'AWS::CodeCommit::Repository'
    Properties:
      RepositoryName: !Sub '${AWS::StackName}-dummy'
  Role:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: 'codebuild.amazonaws.com'
          Action: 'sts:AssumeRole'
      ManagedPolicyArns: ['arn:aws:iam::aws:policy/AdministratorAccess'] # depending on your use case you can narrow down this policy
  LogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: !Sub '/aws/codebuild/${AWS::StackName}'
      RetentionInDays: 14
  Project:
    DependsOn: LogGroup
    Type: 'AWS::CodeBuild::Project'
    Properties:
      Artifacts:
        Type: NO_ARTIFACTS
      Description: 'github-bot'
      Environment:
        ComputeType: 'BUILD_GENERAL1_SMALL'
        Image: 'aws/codebuild/nodejs:8.11.0'
        Type: 'LINUX_CONTAINER'
      Name: !Ref 'AWS::StackName'
      ServiceRole: !GetAtt 'Role.Arn'
      Source:
        BuildSpec: |
          ---
          version: 0.2
          phases:
            install:
              commands: []
            pre_build:
              commands:
              - 'git clone "https://github.com/${GITHUB_OWNER}/${GITHUB_REPO}.git"'
              - 'cd "${GITHUB_REPO}"'
              - 'git reset --hard "${GITHUB_SHA}"'
              - 'git submodule update --init --recursive'
              - 'cd test/ && npm install && cd ../'
            build:
              commands:
              - 'cd test/ && npm test && cd ../'
        Location: !GetAtt 'Repository.CloneUrlHttp'
        Type: CODECOMMIT
      TimeoutInMinutes: 60

Milestone completed: you can run tests for every commit on GitHub.

API Gateway

Luckily, cfn-modules provide an easy way to create an API Gateway backed by a Lambda function.

npm i @cfn-modules/lambda-event-source-webhook -P
npm i @cfn-modules/lambda-function -P

Append the following to the template.yml:

Resources:
  # [...]
  WebhookEventSource:
    Type: 'AWS::CloudFormation::Stack'
    Properties:
      Parameters:
        LambdaModule: !GetAtt 'WebhookFunction.Outputs.StackName'
        AlertingModule: !GetAtt 'Alerting.Outputs.StackName'
        HttpMethod: POST
      TemplateURL: './node_modules/@cfn-modules/lambda-event-source-webhook/module.yml'
  WebhookFunction:
    Type: 'AWS::CloudFormation::Stack'
    Properties:
      Parameters:
        AlertingModule: !GetAtt 'Alerting.Outputs.StackName'
        Handler: 'webhook.handler'
        Runtime: 'nodejs8.10'
        EnvironmentVariable1: !Ref Project
      TemplateURL: './node_modules/@cfn-modules/lambda-function/module.yml'
  WebhookPolicy:
    Type: 'AWS::IAM::Policy'
    Properties:
      Roles: [!GetAtt 'WebhookFunction.Outputs.RoleName']
      PolicyName: 'codebuild'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action: 'codebuild:StartBuild'
          Resource: !GetAtt 'Project.Arn'

Find the source code of the Lambda function to trigger a CodeBuild build on GitHub. Store the webhook.js inside a new folder called lambda-src or check out the complete project at once.

Milestone completed: you can now receive GitHub webhooks.

CloudWatch Event Rule

To provide feedback to the contributor we use GitHub status API to update the status of the pull request. For example, a failed build will look like this in the GitHub Pull Request UI:

To provide feedback, you can listen to CodeBuild state changes using a CloudWatch Event Rule. Append the following to the template.yml:

Resources:
  # [...]
    RuleFunction:
    Type: 'AWS::CloudFormation::Stack'
    Properties:
      Parameters:
        AlertingModule: !GetAtt 'Alerting.Outputs.StackName'
        Handler: 'rule.handler'
        Runtime: 'nodejs8.10'
        EnvironmentVariable1: !Ref Project
      TemplateURL: './node_modules/@cfn-modules/lambda-function/module.yml'
  RulePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !GetAtt 'RuleFunction.Outputs.Name'
      Principal: 'events.amazonaws.com'
      SourceArn: !GetAtt 'Rule.Arn'
  Rule:
    Type: 'AWS::Events::Rule'
    Properties:
      EventPattern:
        source:
        - 'aws.codebuild'
        'detail-type':
        - 'CodeBuild Build State Change'
        detail:
          'build-status':
          - IN_PROGRESS
          - SUCCEEDED
          - FAILED
          - STOPPED
      State: ENABLED
      Targets:
      - Arn: !GetAtt 'RuleFunction.Outputs.Arn'
        Id: rule

Find the source code of the Lambda function to update GitHub status on GitHub. Store the rule.js inside a new folder called lambda-src or clone the complete project at once.

Finally, append the following to the template.yml to easily get the API Gateway endpoint URL.

Outputs:
  Url:
    Value: !GetAtt 'WebhookEventSource.Outputs.Url'

Find the full source code on GitHub including deployment information.

Summary

You used an API Gateway to receive GitHub webhooks and start CodeBuild builds. Asynchronously, in the background, you listen for build changes with a CloudWatch Event Rule and signal the build changes back to GitHub using the status API. With this pattern, you can implement many custom processes on top of GitHub to improve reliability and security of your software development and delivery capabilities.