Workaround: CodePipeline for GitHub Enterprise

Andreas Wittig – 21 Aug 2020

There is no question that AWS has a strong focus on customer obsession. However, sometimes it takes forever until popular feature requests get implemented. A good example: CodePipeline - the continuous delivery service - does support all kinds of source code repositories: CodeCommit, GitHub, Bitbucket, S3, and ECR. Although a very popular option is missing: GitHub Enterprise.

Luckily, there is a decent workaround to connect GitHub Enterprise with CodePipeline.

A webhook from GitHub Enterprise triggers CodeBuild.
CodeBuild fetches the latest changes (Git over HTTPS or SSH), bundles them into a ZIP file, and uploads the archive to S3.
The S3 bucket with versioning enabled stores the latest version of the repository.
A CloudWatch event rule triggers the pipeline whenever the CodeBuild project succeeded.
The source action of CodePipeline downloads the ZIP file, unpacks the archive, and hands over the source code to the next stage.

The solution: GitHub Enterprise, CodeBuild, S3, and CodePipeline.

Code Example

In the following, I will use Terraform to set up all the needed resources. First of all, we need to get Terraform up and running.

Make sure to fill in the following placeholders:

<GITHUB_ACCESS_TOKEN> a personal access token to access the GitHub Enterprise API. Please note, scope admin:repo_hook is required.
<GITHUB_ORGANIZATON> the name of your GitHub Enterprise organization (e.g., myorg).
<GITHUB_REPOSITORY_NAME> the name of your GitHub Enterprise repository (e.g., myrepo).
<GITHUB_REPOSITORY_URL> the URL of your GitHub Enterprise repository (e.g., https://git.example.com/myorg/myrepo.git).
<GITHUB_API_URL> the URL of the GitHub Enterprise API (e.g., https://git.example.com/api/).

terraform {
  required_version = ">= 0.12"
}

provider "aws" {
  region  = "eu-central-1"
  version = "~> 3.0"
}

provider "github" {
  token        = "<GITHUB_ACCESS_TOKEN>"
  organization = "<GITHUB_ORGANIZATON>"
  base_url     = "<GITHUB_API_URL>"
}

data "aws_kms_alias" "s3" {
  name = "alias/aws/s3"
}

Next, we configure the CodeBuild project that will fetch the source code from GitHub Enterprise and upload it to S3.

resource "aws_codebuild_project" "github_enterprise" {
  name          = "github-enterprise"
  build_timeout = "5"
  service_role  = aws_iam_role.github_enterprise.arn

  artifacts {
    type = "S3"
    name = "source.zip"
    location = aws_s3_bucket.artifacts.id
    packaging  = "ZIP"
  }

  cache {
    type  = "LOCAL"
    modes = ["LOCAL_SOURCE_CACHE"]
  }

  environment {
    compute_type                = "BUILD_GENERAL1_LARGE"
    image                       = "aws/codebuild/standard:2.0"
    type                        = "LINUX_CONTAINER"
    privileged_mode   = false
  }

  source {
    type            = "GITHUB_ENTERPRISE"
    location        = "<GITHUB_REPOSITORY_URL>"
    buildspec       = "github-enterprise/buildspec.yml"
  }
}

resource "aws_codebuild_source_credential" "github_enterprise" {
  auth_type = "PERSONAL_ACCESS_TOKEN"
  server_type = "GITHUB_ENTERPRISE"
  token = "<GITHUB_ACCESS_TOKEN>"
}

Next, we configure a webhook between GitHub Enterprise and CodeBuild.

resource "aws_codebuild_webhook" "github_enterprise" {
  project_name = aws_codebuild_project.github_enterprise.name

  filter_group {
    filter {
      type = "EVENT"
      pattern = "PUSH"
    }

    filter {
      type = "HEAD_REF"
      pattern = "master"
    }
  }
}

resource "github_repository_webhook" "github_enterprise" {
  active     = true
  events     = ["push"]
  repository = "<GITHUB_REPOSITORY_NAME>"

  configuration {
    url          = aws_codebuild_webhook.github_enterprise.payload_url
    secret       = aws_codebuild_webhook.github_enterprise.secret
    content_type = "json"
    insecure_ssl = false
  }
}

In the following step, you will create an S3 bucket to store the source code artifact - a zip file named source.zip. I’m using KMS encryption with the default key here.

resource "aws_s3_bucket" "artifacts" {
  versioning {
    enabled = true
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "aws:kms"
      }
    }
  }
}

resource "aws_s3_bucket_policy" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "${aws_s3_bucket.artifacts.arn}/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "",
          "s3:x-amz-server-side-encryption-aws-kms-key-id": "${data.aws_kms_alias.s3.target_key_arn}"
        }
      }
    }
  ]
}
POLICY
}

Nothing works without IAM. Let’s create the IAM role for the CodeBuild project to grant access to S3 and CloudWatch Logs.

resource "aws_iam_role" "github_enterprise" {
  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codebuild.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy" "github_enterprise" {
  role = aws_iam_role.github_enterprise.name

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ]
    },
    {
      "Effect": "Allow",
      "Resource": "${aws_s3_bucket.artifacts.arn}/*",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ]
    }
  ]
}
POLICY
}

The event rule triggers the pipeline whenever the CodeBuild project finished fetching and uploading the latest source code.

resource "aws_cloudwatch_event_rule" "artifacts" {
  event_pattern = <<PATTERN
{
  "source": [ 
    "aws.codebuild"
  ], 
  "detail-type": [
    "CodeBuild Build State Change"
  ],
  "detail": {
    "build-status": [
      "SUCCEEDED"
    ],
    "project-name": [
      "${aws_codebuild_project.github_enterprise.name}"
    ]
  }  
}
PATTERN
}

resource "aws_cloudwatch_event_target" "artifacts" {
  rule      = aws_cloudwatch_event_rule.artifacts.name
  arn       = aws_codepipeline.pipeline.arn
  role_arn  = aws_iam_role.events.arn
}

One more IAM role which allows the CloudWatch event to trigger the pipeline is needed.

resource "aws_iam_role" "events" {
  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy" "events" {
  role = aws_iam_role.events.id

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect":"Allow",
      "Action": "codepipeline:StartPipelineExecution",
      "Resource": "${aws_codepipeline.pipeline.arn}"
    }
  ]
}
POLICY
}

And finally, the pipeline itself.

resource "aws_codepipeline" "pipeline" {
  name     = "mypipeline"
  role_arn = aws_iam_role.codepipeline.arn

  artifact_store {
    location = aws_s3_bucket.artifacts.bucket
    type     = "S3"

    encryption_key {
      id   = data.aws_kms_alias.s3.target_key_arn
      type = "KMS"
    }
  }

  stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "AWS"
      provider         = "S3"
      version          = "1"
      output_artifacts = ["source"]

      configuration = {
        S3Bucket  = aws_s3_bucket.artifacts.id
        PollForSourceChanges   = "false"
        S3ObjectKey = "source.zip"
      }
    }
  }

  stage {
    name = "Deploy"

    action {
    run_order        = "1"
    name             = "Approval"
    category         = "Approval"
      owner            = "AWS"
      provider         = "Manual"
      version          = "1"
    }
  }
}

Last but not least, an IAM role for CodePipeline granting access to the artifacts bucket as well.

resource "aws_iam_role" "codepipeline" {
  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codepipeline.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy" "codepipeline" {
  role = aws_iam_role.codepipeline.id

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect":"Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:PutObject"
      ],
      "Resource": [
        "${aws_s3_bucket.artifacts.arn}/*"
      ]
    },
    {
      "Effect":"Allow",
      "Action": [
        "s3:GetBucketVersioning"
      ],
      "Resource": [
        "${aws_s3_bucket.artifacts.arn}"
      ]
    }
  ]
}
POLICY
}

One more thing. You need to add the file github-enterprise/buildspec.yml to your repository. The file should contain the following configuration.

---
version: 0.2
phases:
  build:
    commands:
    - 'echo "${CODEBUILD_RESOLVED_SOURCE_VERSION}" > SOURCE_VERSION'
artifacts:
  files:
  - '**/*'
  name: source

That’s it. You are ready to run terraform apply to set up CodePipeline for GitHub Enterprise.

Limitations

The code example only works when GitHub Enterprise is available over the Internet. In theory, it is possible to access GitHub Enterprise over private networks only as well. Doing so requires to configure a network interface to establish access to a VPC for the CodeBuild project.
CodePipeline does not know about the commit hash. It shows the version of the source.zip S3 object instead. However, the example adds a file SOURCE_VERSION to the source.zip archive, which contains the original commit hash.
Copying the source code to S3 adds additional latency (about 1-3 minutes) to your deployment pipeline.

Summary

Unfortunately, CodePipeline does not support GitHub Enterprise yet. Using CodeBuild and S3 is a decent workaround to get CodePipeline running for your GitHub Enterprise repository.

Andreas Wittig

I’ve been building on AWS since 2012 together with my brother Michael. We are sharing our insights into all things AWS on cloudonaut and have written the book AWS in Action. Besides that, we’re currently working on bucketAV and marbot.

Here are the contact options for feedback and questions.

Workaround: CodePipeline for GitHub Enterprise

Code Example

Limitations

Summary

Andreas Wittig

Further reading