What’s the AWS European Sovereign Cloud (EUSC)?

The AWS European Sovereign Cloud (aws-eusc) is an independent partition similar to the AWS GovCloud (aws-us-gov) and AWS China (aws-cn). In contrast to the GovCloud, the EUSC is open for all customers. And unlike AWS China, EUSC is not operated by a third party but by AWS itself.

Today, the EUSC consists of a single region in Brandenburg called eusc-de-east-1. The EUSC is designed to be independent from the US from both a technical and organizational perspective.

From a technical point of view, AWS did a great job. For example, AWS built a separate identity and access management (IAM), domain name system (DNS), trust service provider, and much more. All services are available under separate domains like amazonaws.eu and amazonaws-eusc.eu. Even the documentation for the EUSC stands on its own.

Besides that, AWS operates the EUSC exclusively by EU residents: operations, technical support, and customer service.¹ Although AWS has also established companies under German law, these are subsidiaries of the US company. Critics therefore question its legal independence.

Why does widdix go to AWS European Sovereign Cloud?

We decided to bring our products to the EUSC for three reasons:

1. The AWS EUSC opens up a new market. We want to leverage our advantages as first movers. Many of our competitors are not yet represented in the EUSC.
2. We like to try new things. It is a challenge to deal with the shortcomings of new services. And you can write wonderful rants.
3. In politically unstable times, we support the goal of establishing a sovereign European cloud infrastructure.

What did we learn about the AWS European Sovereign Cloud?

We opened our first EUSC account in February. Here is what we learned from practice.

Missing services and features

Not all AWS services are yet available in the AWS European Sovereign Cloud (EUSC). For example, the following services are not yet available: CodeCommit, CodeBuild, CodePipeline, Elastic Beanstalk, IAM Identity Center, CloudFront, Inspector, MQ, Managed Grafana, Managed Prometheus, and Security Hub. Check out the AWS Capabilities by Region overview for details.

It is annoying but understandable that not all services are available at the launch of the eusc-de-east-1 region. What totally surprised us is that even for services that are actually available, not all features are provided.

• EC2 Instance Connect to temporarily grant SSH access is not available.
• IAM Identity Center is missing, which makes access management of multiple accounts much harder.
• AWS Marketplace does not support AMI+CloudFormation products nor free trials for AMI products. Also, metered billing is missing.

Two instead of three availability zones

It is worth noting that the eusc-de-east-1 region consists of only two availability zones. We actually thought that those days were behind us and that all regions would provide at least three AZs. But here, it seems that a mistake from the past is repeating itself. Anyway, it’s nothing that can’t be solved with another condition in Infrastructure as Code.

So-so tooling support

Working with the new AWS partition aws-eusc requires all tools to send requests to different API endpoints due to a separate domain name. Therefore, you need to update tools, like the AWS CLI or Terraform, for example, to the latest version. In case you are using AWS SDKs, you need to upgrade to the latest version as well. Unfortunately, not all tools have shipped a new version that supports the EUSC already. Sometimes, there is a possibility to override the API endpoint with a configuration option.

Increasing service quotas requires support tickets

To ensure the high quality of our solutions, we rely on detailed integration tests. A high degree of parallelization of the tests requires increasing AWS service quotas. We have learned that many quotas, which can normally be increased automatically at AWS, require a support ticket at EUSC. This has significantly increased the amount of work for us.

AWS Support

Speaking of support tickets, be prepared for the fact that the plans for AWS support in the EUSC are different.

To subscribe to the business support plan in the EUSC, you need to open a support ticket. The Enterprise plan is only available through your account manager.

From my experience I can tell that it takes the support team in EUSC a little longer to respond. The team appears to be quite small; so far I’ve talked to the same two people for all my support issues.

In summary, bringing our first product to the AWS European Sovereign Cloud was a journey with obstacles. I hope our experiences help you to be better prepared for the EUSC.

What’s ahead for the AWS European Sovereign Cloud?

It remains to be seen how customers will respond to the AWS European Sovereign Cloud. Given the current political situation, it is questionable whether the target group of government organizations and regulated industries will embrace the AWS Cloud. We are watching the market closely.

Another question is how AWS will deal with the organizational divide between AWS in the US and AWS in Europe. How will the teams work together? What happens when troubleshooting becomes complicated? And how will further development succeed?

What is your experience with the AWS European Cloud? Let me know!

What is Service Quotas Automatic Management?

AWS released Service Quotas Automatic Management on Oct 7, 2025 to provide a solution for monitoring and automatically adjusting service quotas.

Service Quotas Automatic Management comes with the following features.

• Send notifications about service quotas
  • 80% utilization
  • 95% utilization
• Auto-adjust service quota

How to enable Service Quotas Automatic Management?

Unfortunately, neither CloudFormation/CDK nor Terraform supports enabling automatic management for service quotas.

The following command shows how to enable notifications when reaching service quotas with the help of the AWS CLI. Alternatively, you could use the AWS Management Console as well as the API.

aws service-quotas start-auto-management  --opt-in-level ACCOUNT --opt-in-type NotifyOnly

In case you want to automatically increase service quotas, use the opt-in type NotifyAndAdjust.

aws service-quotas start-auto-management  --opt-in-level ACCOUNT --opt-in-type NotifyAndAdjust

How to receive notifications when service quotas are close?

Service Quotas Automatic Management supports the following services to deliver notifications.

• AWS Health (Default)
• AWS User Notifications (Default)
• Email
• AWS Console Mobile App
• Amazon Q Developer chat

The following screenshot shows a service quota notification delivered by the AWS User Notifications service.

Also, it is possible to subscribe to events from Service Quotas Automatic Management via AWS EventBridge. The source of the events is aws.health. Use the eventTypeCode for more detailed filtering.

{
  "source": [
    "aws.health"
  ],
  "detail-type": [
    "AWS Health Event"
  ],
  "detail": {
    "service": [
      "SERVICEQUOTAS"
    ],
    "eventTypeCode": [
      "AWS_SERVICEQUOTAS_THRESHOLD_BREACH",
      "AWS_SERVICEQUOTAS_INCREASE_REQUEST_FAILED",
      "AWS_SERVICEQUOTAS_APPROACHING_THRESHOLD"
    ],
    "eventTypeCategory": [
      "accountNotification"
    ]
  }
}

The following snippet shows an example of a Service Quotas Automatic Management event.

{
  "account": "1122334455566",
  "detail": {
    "actionability": "ACTION_REQUIRED",
    "affectedAccount": "1122334455566",
    "affectedEntities": [
      {
        "entityValue": "Service: dynamodb | Quota Name: Maximum number of tables | Utilization: 82.56%",
        "lastUpdatedTime": "Tue, 13 Jan 2026 10:10:34 GMT"
      }
    ],
    "backupEvent": "false",
    "communicationId": "c7f878c0ef502ffb41c58f8c0de6c733ddbd62fcf82d3891a296512bbccae440-1",
    "endTime": "Tue, 20 Jan 2026 10:10:33 GMT",
    "eventArn": "arn:aws:health:eu-west-1::event/SERVICEQUOTAS/AWS_SERVICEQUOTAS_APPROACHING_THRESHOLD/AWS_SERVICEQUOTAS_APPROACHING_THRESHOLD-DUB-1768299033803",
    "eventDescription": [
      {
        "language": "en_US",
        "latestDescription": "You are receiving this message because you opted to receive notifications from Service Quotas when utilization exceeds (80/95)% threshold.\\n\\nAt Tue, 13 Jan 2026 10:10:33 GMT, we detected that your AWS account is approaching the utilization threshold for one or more service quotas.\\n\\nA list of your affected service quota(s) can be found in the \\\"Affected resources\\\" tab of your AWS Health Dashboard under the format \\\"Service | Quota Name | Utilization Percentage\\\".\\n\\nTo proactively prevent any potential service disruption, we recommend:\\n1. Reviewing your current resource usage and growth patterns\\n2. Requesting a quota increase if needed via the Service Quotas console\\n\\nYou can manage your service quotas through the Service Quotas console (1) or APIs (2).\\n\\nIf you have any questions or concerns, please contact the AWS Support Team (3).\\n\\n(1) https://console.aws.amazon.com/servicequotas\\n(2) https://docs.aws.amazon.com/servicequotas/2019-06-24/apireference/API_Operations.html\\n(3) https://aws.amazon.com/support"
      }
    ],
    "eventRegion": "eu-west-1",
    "eventScopeCode": "ACCOUNT_SPECIFIC",
    "eventTypeCategory": "accountNotification",
    "eventTypeCode": "AWS_SERVICEQUOTAS_APPROACHING_THRESHOLD",
    "lastUpdatedTime": "Tue, 13 Jan 2026 10:10:34 GMT",
    "page": "1",
    "personas": [
      "OPERATIONS"
    ],
    "service": "SERVICEQUOTAS",
    "startTime": "Tue, 13 Jan 2026 10:10:33 GMT",
    "statusCode": "open",
    "totalPages": "1"
  },
  "detail-type": "AWS Health Event",
  "id": "929e5b05-48f3-8150-59ee-a98f2555dc90",
  "region": "eu-west-1",
  "resources": [
    "Service: dynamodb | Quota Name: Maximum number of tables | Utilization: 82.56%"
  ],
  "source": "aws.health",
  "time": "2026-01-13T10:11:16Z",
  "version": "0"
}

It’s a pity that the event is not very well designed. For example, it should include the quote code and service code, as this would allow you to call the RequestServiceQuotaIncrease API action automatically. Instead, all we get from the event details is the affected resources—for example Service: dynamodb | Quota Name: Maximum number of tables | Utilization: 82.56% - and an ID/link to the AWS Health notification.

Also, AWS not only sends a notification once for getting close to or breaching a service quota. Instead, AWS repeats sending a notification every 24 hours. So you need to implement deduplication to avoid notifications piling up.

How to increase AWS service quotas automatically?

Service Quotas Automatic Management comes with the ability to auto-adjust service quotas. Unlike manual quota increases, auto-adjust requests are handled through a specialized, automated workflow that does not require an AWS Support case.

In case a quota increase fails, you will receive a notification, but specific rejection details aren’t available.

Unfortunately, the feature is not working correctly. For testing, I created 50 AppSync GraphQL APIs to reach the service quota. However, the auto-adjustment did not work. I’m still waiting for a response from the AWS service team.

Limitations of Service Quotas Automatic Management

Even though you are using Service Quotas Automatic Management, you might run into issues caused by reaching service quotas.

First, Service Quotas Automatic Management does not support all quotas. Monitoring and notifications are only available for a subset of all service quotas. And only a subset of that supports auto-adjust.

To get a list of all supported quotas, go to the tab Automatic Management in the Service Quota management console and press the View supported quotas button.

For example, Automatic Management supports only three of the DynamoDB quotas. And auto-adjusting is not available at all for DynamoDB quotas.

Second, it takes between one and three hours until Automatic Management detects that a service quota is reached. The relatively long response time can lead to problems occurring during operation even before this point.

Summary

The monitoring of service quotes has become a lot easier with Service Quotas Automatic Management. Notifications allow you to increase service quotas before they affect workloads. The ability to auto-adjust quotas is even more convenient. Unfortunately, the usefulness of both the notifications and the auto-adjustment is limited, as not all quotas are supported. So you cannot rely on the solution.

By the way, marbot our solution for AWS monitoring, already supports service quota notifications, as shown in the following screenshot. Did you know that marbot event dedouplicate events? Quite helpful, as Service Quotas Automatic Management sends the same notifications multiple times.

Important to know, a CloudWatch dashboard supports the following internal data sources out of the box:

• CloudWatch metrics
• CloudWatch logs
• CloudWatch alarms

The possibility to add charts not based on metrics but on logs to a dashboard caught my attention.

First, I did the math to compare the costs between custom metrics and logs.

CloudWatch custom metrics pricing example

AWS charges $0.30 per custom metric per month.

Moreover, AWS charges $0.01 per 1,000 PutMetricData requests. Let’s assume we want to record a metric value every 30 seconds.

2 * 60 * 24 * 30 = 86,400 PutMetricData requests per month = $0.86 per month

That’s $0.86 for ingesting metric data per month.

When displaying custom metrics on a dashboard, AWS charges for the GetMetricData or GetMetricStatistics requests. We assume 100 requests per day.

100 * 30 = 3000 requests per month = $0.03 per month

So, we pay an additional $0.03 per month to display the metric per month.

In total, my estimation is a charge of $1.19 per custom metric per month.

Let’s compare the costs of custom metrics with logs next.

CloudWatch logs pricing example

Instead of using custom metrics, we send structured log messages to CloudWatch. For example, the following log message includes an event type and a value.

{event:'CUSTOM_METRIC_1',value:10}

For the following pricing example, let’s assume each metric data message consumes about 60 bytes of storage. AWS charges $0.50 per GB (us-east-1) for ingesting data.

2 per minute * 60 minutes * 24 hours * 30 days * 60 bytes = 0.005 GB per month = $0.0025 per month

In summary, data ingestion costs $0.0025 per metric and month.

Additionally, AWS charges $0.005 per GB of data scanned when querying the data. Let’s assume that the CloudWatch dashboard displaying information about the metric executes 100 CloudWatch Logs Insights queries per day and fetches data from the past 30 days.

30 days * 100 queries * 0.005 GB = 15 GB scanned data per month = $0.075 per month

Use field indexes to ensure each query is only scanning relevant data.

So the cost for querying the data is $0.075 per month.

In total, my estimation is a charge of $0.0775 per metric and month.

Note that using logs to display metrics on a dashboard is 93% cheaper than using custom metrics.

Pros and Cons

First, defining CloudWatch alarms to trigger automated actions like scaling out or in or to notify operators requires CloudWatch metrics. Using CloudWatch logs to collect and query metrics is a possible option when the data is needed to be analyzed on-demand, for example, to build a dashboard.

Second, dashboard widgets based on CloudWatch metrics load much faster than those relying on a CloudWatch logs query. It takes about 1 second to load a widget based on a custom metric and 5 to 10 seconds to load a widget based on a logs query.

Third, the available widget types for CloudWatch dashboards differ based on the data type.

The following screenshot shows the available widgets for CloudWatch metrics.

In contrast, the next screenshot shows the available widgets for CloudWatch logs.

Fourth, the query language to evaluate data differs between metrics and logs. A math expression allows you to combine multiple metrics, for example.

The following snippet shows a math expression to calculate the total network throughput of a NAT gateway.

(BytesInFromDestination+BytesInFromSource+BytesOutToDestination+BytesOutToSource)

In contrast, CloudWatch Logs Insights allows you to analyze metric data stored in log groups. Three different query languages are supported.

• Logs Insights QL
• PPL
• SQL

The following example shows a query, that fetches the median, p90, and p95 from log messages grouped by day.

SELECT window.start, approx_percentile(`message.diffStartInProgress`, 0.50) AS `Median`, approx_percentile(`message.diffStartInProgress`, 0.90) AS `p90`, approx_percentile(`message.diffStartInProgress`, 0.95) AS `p95` FROM `logGroups(logGroupIdentifier:['/aws/lambda/***'])` WHERE `message.eventId` = 'JOB_IN_PROGRESS' GROUP BY window(`@timestamp`, '24 hours')

Summary

Collecting and analyzing metric data with CloudWatch logs is a cost-effective alternative to using custom metrics. The approach comes with a few drawbacks. For example, it is not possible to automate actions or send notifications based on logs out-of-the-box. However, building a dashboard to provide insights into key metrics is definitely possible with data stored in log groups instead of custom metrics.

The challenge

GitHub webhooks enable applications to subscribe to events, like the workflow_job event informing about jobs getting created, starting to run, or completing. To ensure HyperEnv provides a self-hosted runner for every job, even in case of failures, HyperEnv needs to keep track of every GitHub job. Doing so allows HyperEnv to retry launching a self-hosted runner in case a job does not start running within 5 minutes, for example.

The solution

The following diagram illustrates the solution. Let me walk you through the architecture diagram from left to right.

First, an API Gateway receives the GitHub webhook events via HTTP and invokes the Lambda function Webhook.

Next, the Lambda function Webhook starts the execution of the state machine Job State Machine.

The following screenshot shows the state machine in more detail.

• Start is where the state machine starts.
• Queued is the initial state of a GitHub job.
• InProgress indicates that the GitHub job is running.
• Completed means that the GitHub job finished.
• End is the end of the state machine.

Then, the state machine Job State Machine invokes the Lambda function Job State by using the Wait for Callback integration (see Wait for a Callback with Task Token).

The Lambda function Job State creates an item in the DynamoDB table Job State and persists the task token needed to resolve the callback.

If next, GitHub sends a webhook event with a status update, the process continues.

The API Gateway receives the event. Then, the Lambda function Webhook queries the DynamoDB table Job State and fetches the task token. With the task token, the Lambda function Webhook sends a task success signal to the state machine Job State Machine. Which will resolve the wait for the Queued task and continue with the next one InProgress.

In the next section, I will share implementation details of the architecture.

Deep Dive: Step Functions and Lambda with Wait for Callback

The following listing shows an excerpt of the state machine’s definition.

The Lambda function is called with Wait for Callback (see arn:aws:states:::lambda:invoke.waitForTaskToken).

The task token is needed to send a success signal as soon as the state is done. Therefore, $states.context.Task.Token is added to the payload of the Lambda function invocation.

{
  "Comment": "Job State Machine",
  "QueryLanguage": "JSONata",
  "StartAt": "Queued",
  "States": {
    "Queued": {
      "Arguments": {
        "FunctionName": "arn:aws:lambda:eu-west-1:xxxxxxxxxxxx:function:JobStateFunctionFunction",
        "Payload": "{% $merge([$states.context.Execution.Input, {\"task\": \"Queued\", \"taskToken\": $states.context.Task.Token}]) %}" // <= Adds the task token to the payload when invoking the Lambda function
      },
      "Next": "InProgress",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", // <= Calls Lambda function in Wait for Callback mode
      "TimeoutSeconds": 300,
      "Type": "Task"
    },
    "InProgress": {
      "Arguments": {
        "FunctionName": "arn:aws:lambda:eu-west-1:xxxxxxxxxxxx:function:JobStateFunctionFunction",
        "Payload": "{% $merge([$states.context.Execution.Input, {\"task\": \"InProgress\", \"taskToken\": $states.context.Task.Token}]) %}" // <= Adds the task token to the payload when invoking the Lambda function
      },
      "Next": "Completed",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", // <= Calls Lambda function in Wait for Callback mode
      "TimeoutSeconds": 21600,
      "Type": "Task"
    },
    ...
  },
  "TimeoutSeconds": 3600
}

Next, let’s take a look into a possible implementation of the Lambda function Job State.

Depending on the event.task property, the function updates an item in the DynamoDB table.

The expire_at attribute is used to define a time-to-live for the DynamoDB item to ensure the stored state gets deleted from the table automatically.

The task token is stored in the item’s attribute task_token_queued or task_token_in_progress.

import { DynamoDBClient, UpdateItemCommand } from '@aws-sdk/client-dynamodb';
const ddbClient = new DynamoDBClient();

export async function handler(event) {
  switch(event.task) {
    case 'Queued': {
      await ddbClient.send(new UpdateItemCommand({
        ExpressionAttributeValues: {
          ':state': {
            S: 'queued'
          },
          ':task_token': {
            S: event.taskToken
          },
          ':expire_at': {
            N: Math.floor((new Date().getTime() + 1 * 24 * 60 * 60 * 1000) / 1000).toString(), // expire in 24 hours
          },
          ':created_at': {
            N: Math.floor(new Date().getTime() / 1000).toString()
          },
        },
        ExpressionAttributeNames: {
          '#s': 'state',
        },
        Key: {
          run_job_id: {
            S: `run-${event.runId}-job-${event.jobId}`
          }
        },
        ReturnValues: 'ALL_NEW',
        TableName: JOB_STATE_TABLE_NAME,
        UpdateExpression: 'SET #s = :state, task_token_queued = :task_token, expire_at = if_not_exists(expire_at, :expire_at),  created_at = if_not_exists(created_at, :created_at)'
      }));
      break;
    }
    case 'InProgress':
      await ddbClient.send(new UpdateItemCommand({
        ExpressionAttributeValues: {
          ':state': {
            S: 'in_progress'
          },
          ':task_token': {
            S: event.taskToken
          }
        },
        ExpressionAttributeNames: {
          '#s': 'state'
        },
        Key: {
          run_job_id: {
            S: `run-${event.runId}-job-${event.jobId}`
          }
        },
        ReturnValues: 'ALL_NEW',
        TableName: JOB_STATE_TABLE_NAME,
        UpdateExpression: 'SET #s = :state, task_token_in_progress = :task_token'
      }));
      break;
    case 'Completed':
      // ...
      break;
  }
}

Finally, let’s take a look into the implementation of the Lambda function Webhook.

The following snippet shows how the Lambda function Webhook starts the execution of the state machine Job State Machine for each new GitHub job.

if (body.action === 'queued') {
  await sfnClient.send(new StartExecutionCommand({
    stateMachineArn: JOB_STATE_MACHINE_ARN,
    input: JSON.stringify({
      organizationName: body.repository.owner.login,
      jobLabels: body.workflow_job.labels,
      runId: body.workflow_job.run_id,
      jobId: body.workflow_job.id,
      installationId: body.installation.id,
    }),
    name: `run-${body.workflow_job.run_id}-job-${body.workflow_job.id}`
  }));
  return {
    statusCode: 200,
    body: `created state machine execution run-${body.workflow_job.run_id}-job-${body.workflow_job.id}`
  };
}

Next, when GitHub sends a webhook indicating that the job changed its status from queued to in in_progress the Lambda function executes the following code.

1. Fetch the state of the GitHub job from the DynamoDB table Job State.
2. Use the task token task_token_queued to send a task success notification to the state machine.

if (body.action === 'in_progress') {
  const queryResult = await ddbClient.send(new QueryCommand({
    ExpressionAttributeValues: {
      ':run_job_id': {
        S: `run-${body.workflow_job.run_id}-job-${body.workflow_job.id}`
      }
    },
    KeyConditionExpression: 'run_job_id = :run_job_id',
    TableName: JOB_STATE_TABLE_NAME
  }));
  await sfnClient.send(new SendTaskSuccessCommand({
    taskToken: queryResult.Items[0].task_token_queued.S,
    output: JSON.stringify({})
  }));
  return {
    statusCode: 200,
    body: 'state updated'
  };
}

The process is similar for the transition from in_progress to completed.

Summary & Feedback

To track the progress of a workflow depending on 3rd parties, the following patterns are helpful when using AWS Step Functions.

• Use Wait for Callback mode to invoke Lambda functions. Store the task token in DynamoDB. Optionally call the 3rd party. Then wait for a response from the 3rd party.
• As AWS Step Functions does not provide a way to get information about the current task (aka. state), store information about the current state in DynamoDB.

I hope following along with my solution helps you to come up with a suitable approach for your use case. In case you found another approach, please share it with me. I’m happy to learn from you!

Running workloads on t3.nano, t3a.nano, or t4g.nano is a challenge. The 512 MB of RAM are mostly used entirely by the operating system Amazon Linux 2023. As soon as you (or your application) try to allocate memory, things get very slow. There are two tricks to address this problem by moving stuff from RAM to disk:

1. Move /tmp from RAM to disk.
2. Move SWAP from RAM to disk.

Move /tmp from RAM to disk

/tmp is typically where files that a program doesn’t need in the long-term go. In Amazon Linux 2023, /tmp is backed by tmpfs which stored the file in memory.

The problem: Memory is scarce on t3.nano.

To return to /tmp stored on disk, run:

sudo systemctl mask tmp.mount 
sudo systemctl stop tmp.mount

Move SWAP from RAM to disk

If the OS is running out of memory, it tries to move memory from RAM to another medium, aka swap. Often, swap is backed by disk storage, which is slow but works. In Amazon Linux 2023, swap is provided by zram on instances with less than 1 GB of memory. Zram creates a compressed block device in RAM for swap.

Two problems:

1. Compression/decompression is CPU-bound, and you don’t have a lot of CPU power on a nano instance.
2. RAM is already scarce. There simply is not a lot of RAM available.

To disable zram and return to swap backed by disk, run:

sudo dnf -y remove zram-generator-defaults zram-generator

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
cat <<"EOF" | sudo tee -a /etc/fstab > /dev/null
/swapfile none swap defaults 0 0
EOF

Summary

The most scarce resource on EC2 instance types like t3.nano, t3a.nano, or t4g.nano is memory. Unfortunately, Amazon Linux 2023 is much more memory-hungry than Amazon Linux 2. Therefore, you might need to tweak AL2023 to continue running your applications. The most important trick is to move swap away from zram to disk.

For more than a year, each month, I receive a budget alert for the AWS account that hosts our event-driven AWS Monitoring solution called marbot. The following figure shows the latest alert from July 2025.

For a long time, I accepted that I was spending more than what I defined in my budget years ago. This month, I investigated the bill increase and discovered something interesting.

Checking my AWS bill

I use AWS Cost Explorer to understand my AWS costs. The following figure shows the AWS costs for the last 12 months:

As I said, the AWS bill should somehow reflect the usage pattern of the workload. As an approximation, I used the number of alerts created by marbot to check the usage of our service against the AWS bill.

The number of alerts grew by 33%, from 120k to 160k, in the last 12 months. The AWS bill increased by 20% from $350 to $419. At the first glance, that looks great.

Investigating the Cost Spike

But by looking closer, one service has a significant increase in costs: AWS Config.

AWS Config costs increased by 1400% from $6 to $90.

Root Cause Analysis

Luckily, AWS Cost Explorer allows me to drill down into the AWS Config costs by grouping by Usage Type which quickly revealed that most of the costs are associated with usage type ConfigurationItemRecorded.

I had two questions:

1. Why are there so many more configuration changes recorded?
2. How can I reduce the recorded configuration changes?

We ship AWS config data to a centralized S3 bucket so I executed a bunch of Athena queries shared by AWS to learn that most of the changes are caused by AWS Fargate. When you launch a task, an ENI is created, which causes 4 changes to be recorded (ENI, VPC, subnet, security group). 4 AWS Config changes cost $0.012. Running a Fargate task for 1 minute costs $0.0008 (1 vCPU with 2 GB memory). In other words, I pay more for AWS Config than for Fargate which is crazy :)

You might ask, why use AWS Config at all? The answer is simple: Our ISO 27001 audit depends on AWS Security Hub, which depends on AWS Config.

AWS Config cost optimization

So the final question remains: How to reduce the changes recorded by AWS Config while still keeping Security Hub happy? The answer is simple: Switch from recording frequency continuous to daily.

The following CloudFormation snippet changes the frequency:

ConfigurationRecorder:
   Type: 'AWS::Config::ConfigurationRecorder'
   Properties:
     [...]
     RecordingMode:
       RecordingFrequency: DAILY

As you can see, my AWS Config costs are close to zero after applying the change on July 23rd. This will save me around $90 per month or $1080 per year. A fun and worthwhile time investment.

Summary

AWS Config is adding more and more resource types that it records. Therefore, your AWS Config bill is likely increasing over time as well. Besides that, volatile workloads create and delete resources that are recorded by AWS Config. For example, an Auto Scaling Group to manage a fleet of Spot EC2 instances or the Fargate tasks example from above.

The daily recording frequency approach I shared in this blog post will help you to reduce your AWS Config bill in volatile AWS environments. The downsides:

• Not all changes will be captured by AWS Config. Lucky me, I also have AWS CloudTrail enabled, which captures all API activity.
• Changes recorded in daily mode are 300% more expensive, $0.012 instead of $0.003. If your workloads are not volatile you might spend more!

Learn from my journey of finding the best way to generate SDKs for JavaScript, Java, Python, and TypeScript for a REST API.

Built-in function: Generate SDKs for REST APIs

First, I tried the built-in functionality Generate SDKs for REST APIs in API Gateway.

Generating the SDK was simple and fast. However, the result was mediocre. From a developer point of view, using the SDK did not provide significant convience compared to sending plain HTTP request. I was missing an layer of abstraction to simplifiy API usage.

Smithy: Interface Definition Language (IDL)

Second, I looked into Smithy, an Interface Defintion Language developed by AWS to generate their SDKs. Who, if not AWS, must know how to generate SDKs for APIs.

The concept of an extensible, typesafe, protocol agnostic IDL in combination with tools to build client SDKs for various programming languages resonated with me. I found the layer of abstraction, that I was missing while generating SDKs with Amazon API Gateway.

The following snippet shows the model, that I used to build clients for the attachmentAV API.

$version: "2.0"

namespace com.attachmentav.developer

use aws.protocols#restJson1
use aws.api#service

@title("attachmentAV - Virus and Malware Scan API (SaaS)")
@restJson1
@httpApiKeyAuth(name: "x-api-key", in: "header")
@service(sdkId: "VirusMalwareScanSaaS")
service VirusMalwareScanSaaS {
  version: "2006-03-01"
  operations: [CreateSyncDownloadScan, CreateSyncBinaryScan]
}

structure ScanResult {
    
  @required
  @notProperty
  status: String,

  @notProperty
  finding: String,

  @required
  @notProperty
  size: Long,

  @notProperty
  realfiletype: String

}

map DownloadHeaders {
  key: String,
  value: String
}

structure CreateSyncDownloadScanInput {
  @required
  download_url: String,
  download_headers: DownloadHeaders
}

@idempotent
@http(method: "POST", uri: "/v1/scan/sync/download")
operation CreateSyncDownloadScan {
  input: CreateSyncDownloadScanInput,
  output: ScanResult
}

structure CreateSyncBinaryScanInput {
  @httpPayload
  content: Blob
}

@idempotent
@http(method: "POST", uri: "/v1/scan/sync/binary")
operation CreateSyncBinaryScan {
  input: CreateSyncBinaryScanInput,
  output: ScanResult
}

However, I ran into two problems with Smithy. On the one hand, Smithy is clearly focused on generating SDKs for AWS services and therefore makes assumptions about endpoint URLs, authentication mechanisms, and more by default. Using Smithy outside these defaults was tricky for me. On the other hand, most client generators are not production-ready yet and are marked as developer preview. That led to generated code that was not compiling or did not work as intended.

OpenAPI Generator: Generate from OpenAPI documents

Third, after getting advice from Sebastian, Anton, Luciano on LinkedIn I looked into the popular OpenAPI Generator project.

As we use an OpenAPI document to deploy the API Gateway on AWS, giving OpenAPI Generator a try was pretty simple. All I needed to do was to hand over the OpenAPI document to the generator.

But, the result was not very promising. The usability of the generated SDK was pretty bad.

The following idea brought the turnaround: define a seperate OpenAPI document with the aim of generating an SDK that is easy to use. I removed some resources and methods, modified the resource definitions, and more.

The following snippet shows the OpenAPI document optimized to generate SDKs.

openapi: 3.0.0
info:
  description: 'Scan files for viruses, trojans, and other kinds of malware.'
  title: attachmentAV
  version: 1.0.0
servers:
- url: https://eu.developer.attachmentav.com/v1
security:
- apiKeyAuth: []
- bearerAuth: []
tags:
- name: attachmentAV
paths:
  /scan/sync/binary:
    post:
      description: 'Upload a file, scan the file, and return the scan result.'
      tags: [attachmentAV]
      requestBody:
        content:
          application/octet-stream:
            schema:
              format: binary
              type: string
        required: true
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ScanResult'
          description: Success
  /scan/sync/download:
    post:
      description: 'Download a file from a remote location (HTTP/HTTPS), scan the file, and return the scan result.'
      tags: [ attachmentAV ]
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/SyncDownloadScanRequest'
        required: true
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ScanResult'
          description: Success
  /scan/sync/s3:
    post:
      description: 'Download a file from S3, scan the file, and return the scan result. A bucket policy is required to grant attachmentAV access to the S3 objects.'
      tags: [attachmentAV]
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/SyncS3ScanRequest'
        required: true
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ScanResult'
          description: Success
components:
  schemas:
    ScanResult:
      example:
        size: size
        realfiletype: realfiletype
        finding: finding
        status: status
      properties:
        status:
          type: string
        finding:
          type: string
        size:
          type: string
        realfiletype:
          type: string
      type: object
    SyncDownloadScanRequest:
      properties:
        download_url:
          type: string
        download_headers:
          additionalProperties:
            type: string
          type: object
      required:
      - download_url
      type: object
    SyncS3ScanRequest:
      properties:
        bucket:
          type: string
        key:
          type: string
        version:
          type: string
      required:
      - bucket
      - key
      type: object
  securitySchemes:
    apiKeyAuth:
      in: header
      name: x-api-key
      type: apiKey
    bearerAuth:
      type: http
      scheme: bearer

I’m pleased with the results (see attachmentav-sdk-js, attachmentav-sdk-java, attachmentav-sdk-python, and attachmentav-sdk-ts).

The following snippet shows how to scan a file for viruses and malware.

import { AttachmentAVApi, Configuration } from '@attachmentav/virus-scan-sdk-ts';
import * as fs from 'fs';

const config = new Configuration({
  apiKey: '<API_KEY_PLACEHOLDER>'
});

const api = new AttachmentAVApi(config);

async function scanFile() {
  const fileBuffer = fs.readFileSync('./demo.txt');
  const blob = new Blob([fileBuffer]);
  const scanResult = await api.scanSyncBinaryPost({
    body: blob
  });
  console.log('Sync binary scan result:', scanResult);
}

scanFile();

Looking for a way to scan files for viruses, trojans and other kinds of malware. Try the Virus and Malware Scan API by attachmentAV!

Summary

Building SDKs for an API increases discoverability and simplifies integration. Implementing and maintaining SDKs by writing code manually is a huge effort. For us, using the OpenAPI Generator with an optimized OpenAPI document brought the best results.

A recent AWS feature release caught my attention: Amazon EC2 now enables you to delete underlying EBS snapshots when deregistering AMIs. As the author of aws-amicleaner, a tool designed to clean up AMIs based on configurable rules, I’m excited about the operational improvements this brings.

Streamlined implementation

With this new feature, you can now delete an AMI along with its associated snapshots in a single API call. Here’s a JavaScript implementation using the AWS SDK v3:

import { EC2Client, DeregisterImageCommand } from '@aws-sdk/client-ec2';

export async function deleteAMI(amiId) {
  const ec2 = new EC2Client({});
  return ec2.send(new DeregisterImageCommand({
    ImageId: amiId,
    DeleteAssociatedSnapshots: true
  }));
}

Benefits of the new approach

This enhancement eliminates the need for at least two additional API calls that were previously required to:

1. Retrieve the AMI’s block device mappings
2. Delete each associated snapshot individually

The result is a more efficient, reliable AMI cleanup process that reduces both API overhead and the risk of orphaned resources.

Conclusion

This AWS feature represents a significant improvement for infrastructure management. AMI deletion is now faster, more reliable, and less prone to human error—a welcome change for anyone managing EC2 images at scale.

We had doubts about getting ISO 27001 certified. First, we imagined that getting ISO 27001 certified is not for 2-person companies, as we are. Second, we expected the process to involve a lot of paperwork and bureaucracy without adding any real value. But things turned out differently.

What is ISO 27001?

ISO 27001 is an international standard for putting an information security management system (ISMS) in place to meet the following requirements:

• Identify and assess information security risks, including threats and weaknesses.
• Put in place effective security measures or other ways to handle unacceptable risks.
• Maintain a process to regularly review and update security controls as needed.

Our journey to ISO 27001

Our research showed that there are two main ways to prepare for an ISO 27001 audit:

• Work with a consultancy helping you to implement an ISMS and the necessary procedures.
• Buy a tool that guides you through the process in an automated way.

We decided to go with the second option. We looked at a few tools and chose Sprinto, which describes itself as a continuous security and compliance platform.

First, we set up policies and controls that define who is responsible for which part of information security in our company with the help of the Sprinto team. Second, we started implementing the controls. Luckily, Sprinto comes with integrations for most of the tools and platforms we use: Amazon Web Services, Google Workspaces, GitHub, and many more. So implementing controls means configuring the integrations and going through the checks in Sprinto. More on those checks in the following.

It took us eight weeks to implement the controls. Then, we handed over the collected evidence to an external auditor. And nine weeks after we started the project, we hold the ISO 27001 certificate in our hands.

Infrastructure

100% of our infrastructure is hosted on Amazon Web Services. Sprinto provides an integration that connects with the AWS API and creates a list of all assets. Those assets need to get classified as non-production or production. Then, Sprinto runs checks against the production assets to ensure, they comply with the specified controls.

It took us a while to roll out the Sprinto integration to all our AWS accounts because there was no way to automate the process. Luckily, Sprinto released a new feature allowing us to benefit from AWS Organizations and CloudFormation StackSets to roll out the integration to all AWS accounts.

Even though, our AWS infrastructure follows security best practices, Sprinto created a list of 100+ checks that needed to be addressed. At the beginning, we were strictly implementing any change raised by Sprinto. But we soon realized that some of the changes were not applicable to our infrastructure. And we started to document exceptions for those cases. For example, the checks raised concerns about missing backups for some DynamoDB tables. However, we are using DynamoDB tables to store sessions and cached data that is deleted after a short period of time (TTL) and therefore does not need to be backed up. Even worse, enabling backups for those tables where a lot of data gets created and deleted would have increased costs significantly.

Vulnerabilities

Another important part is vulnerability management. And that’s an area, where we did not have any established processes before. We enabled the vulnerability management in Sprinto and connected it with GitHub’s Dependabot and AWS Inspector. Now, when a vulnerability is detected, we are getting notified, and Sprinto tracks that we roll out an update to fix the vulnerability.

• Critical Vulnerabilities should be resolved in 3 days
• High Vulnerabilities should be resolved in 30 days
• Moderate Vulnerabilities should be resolved in 60 days
• Low Vulnerabilities should be resolved in 100 days

Meeting those deadlines is a challenge for a 2-person company, but we are confident that we can do it. And we are happy to have a process in place that ensures that we are doing our best to keep our customers’ data secure.

Staff & Access

While we are a 2-person company, we also hire freelancers to bring in expertise for specific tasks. Those freelancers need access to GitHub or some AWS resources. So far, we were going through the access management of our systems and removed freelancers who are currently not working for us. IS0 27001 requires a process to onboard and offboard employees and freelancers. And we are glad that we can automate those tasks with Sprinto. Now, we ensure that our freelancers have the same information security standards. For example, by ensuring encryption of data-at-rest is enabled on their devices. Also, when freelancers leave, the automated checks ensure that access to all systems is removed.

Summary

Thanks to a high level of automation, it took us only nine weeks to get ISO 27001 certified. We invested about 100 hours and had a four-digit budget. We learned a lot and improved vulnerability management as well as access management significantly. Furthermore, we hope the ISO 27001 certificate will help us to build trust with prospects and customers.

Besides sharing our learnings about all things AWS at cloudonaut, we’re building bucketAV, a solution to protect Amazon S3 and Cloudflare R2 from viruses and malware.

This is the first part of our series Made by cloudonaut where we present the solutions we are building: bucketAV, attachmentAV, HyperEnv, and marbot.

Video: S3 Virus Scan Step-by-Step Guide

Check out our video, where we walk you through the steps required to protect your S3 buckets from viruses and malware.

Why protect S3 with an antivirus solution?

We started bucketAV back in 2015 because one of our consulting clients had the need to scan files uploaded by users for viruses and malware, before publishing them on their website. Most traditional antivirus solutions scan file systems but don’t come with the ability to scan an object store, like bucketAV.

Besides that, here is a list with the main use cases where you should consider protecting your S3 buckets with an antivirus solution:

• User-generated content
• Comply with industry standards
• Incoming files from 3rd party data providers
• File storage of users and customers
• Content and software distribution

Next, let me explain the architecture of our antivirus solution.

Architecture: Fault-Tolerant, Cost-Effective, Scalable

When designing the architecture for bucketAV, we had the following goals in mind:

• Fault Tolerance
• Cost Effectiveness
• Scalability
• Security

The architecture consists of the following building blocks:

• S3 buckets stores files that need to be scanned.
• S3 Event Notifications, or EventBridge is used to trigger real-time scan jobs.
• SQS queue to receive and store scan jobs.
• Auto-scaling group to manage a fleet of EC2 instances.
• EC2 instance runs bucketAV and the antivirus engine (ClamAV/Sophos).
• SNS topic to notify add-ons and humans about the scan results.

You might wonder why we use EC2 instead of Lambda or Fargate. The short answer: EC2 is cost-efficient and provides the best performance. The long answer is: Why does bucketAV use EC2 to protect S3 from malware?.

Try bucketAV

Protect Amazon S3 from viruses, trojans, ransomware, and other kinds of malware. Get started with a free trial today!

EC2 Spot vs. On-Demand

There are three different pricing models for virtual machines on AWS.

• On-Demand is the default, depending on the instance type, you pay an hourly fee, which is usually charged by the second. For example, an m5.large instance (2 vCPUs and 8 GiB of memory) costs you $0.0960 per hour in region us-east-1.
• Spot grants discounts on unused capacity in AWS’s data centers. The price changes depending on the utilization of the data centers. While writing this, an m5.large instance is about $0.0348 in us-east-1 which is a 63.78% saving compared with on-demand. Here is the catch: AWS reserves the right to terminate a spot instance after 2 minutes notice, which AWS calls a spot interruption.
• Savings Plans are a simple deal: you commit to a specific use of EC2 instances and get a discount from AWS. The approach works best for static workloads that can be planned for 1 or even 3 years in advance.

So, running GitHub runners on spot instead of on-demand instances cuts down infrastructure costs by about 60%. However, there are a few caveats to consider.

Consideration: Ephemeral Runners

There are three approaches for hosting GitHub Runners on AWS:

• Long-running: Launch an EC2 instance, install and start GitHub runner. Keep the instance running 24/7.
• Auto-scaled: Launch and terminate EC2 instances depending on the number of waiting jobs.
• Ephemeral: Launch an EC2 instance for every job. Terminate the machine after completing the job.

A typical build job takes 5 to 15 minutes to complete. Therefore, ephemeral runners require a spot instance for a short time only, which reduces the risk of getting interrupted. That’s because AWS takes the runtime of a spot instance into consideration, when selecting a spot instance to interrupt to free up capacity.

So, ephemeral runners are a perfect fit for EC2 spot instances.

Consideration: Fallback to On-Demand

Depending on the utilization of the data center, the availability of spot instances might be limited. It is not unlikely, that AWS rejects a request to launch a spot instance due to no capacity.

When utilizing spot instances, it is essential to be aware that the availability of these instances can fluctuate based on demand. In some cases, AWS might not have enough capacity available, resulting in your request to launch a spot instance being rejected due to no available capacity.
To mitigate this risk, consider implementing a fallback strategy where you automatically switch to launching on-demand instances when spot instances are not available.

One way to achieve this is by using a combination of AWS Auto Scaling and CloudWatch metrics to monitor the availability of spot instances. Based on these metrics, your Auto Scaling group can scale up or down accordingly, ensuring that your GitHub runners have access to the necessary resources.
By implementing such a strategy, you can minimize the impact of spot instance unavailability and maintain a stable and reliable infrastructure for your GitHub runners.

This approach ensures high availability and allows you to take advantage of the cost savings offered by spot instances while minimizing the risk associated with their availability.

Here is what to do in such a situation:

1. Try to launch the spot instance in another availability zone, which means in another data center that might provide spot capacity.
2. Fallback to launching an on-demand instance.

Consideration: Does the GitHub workflow withstand an interruption?

While chances are low that an ephemeral runner that runs on an EC2 spot instance for a few minutes gets interrupted, it is not zero. For many GitHub workflows, it is not an issue when a job gets stuck and cancelled due to a spot interruption. A job like running unit tests, linting code, or building artifacts can typically be restarted without any side effects. However, there are jobs where an interruption might cause in corrupt state or unwanted side effects. For example, interrupting the run of terraform apply might cause the following job to fail because the Terraform state is still locked.

Therefore, being able to configure whether a spot instance should be used at the job level is necessary.

Architecture for GitHub Actions running on EC2 spot instances

So what would an AWS architecture for launching ephemeral GitHub runners on EC2 spot instances with a fallback to on-demand instances look like? The following diagram shows the solution that we ended up with for HyperEnv.

1. API Gateway receives an HTTP request from GitHub.
2. API Gateway invokes the Lambda function named webhook.
3. The Lambda function webhook verifies the incoming webhook event.
4. The Lambda function webhook starts an execution of the Step Function runner-orchestrator.
5. The Step Function invokes the Lambda function consumer which tries to launch a spot instance.
6. In case a spot instance is not available in the selected availability zone, the Step Function retries launching a spot instance in another availability zone by calling the Lambda function consumer a second time.
7. In case it is not possible to launch a spot instance, the Step Function continues and calls the Lambda function consumer again to launch an on-demand instance.

Give HyperEnv a try!

Do you prefer a production-ready and well-maintained solution instead of building this on your own? Check out our solution for self-hosted GitHub Actions runners for AWS: HyperEnv.

Prehistory

In 2012, Michael and I joined the same team at Tullius Walden Bank with the mission to build a trading platform for retail customers. We set ourselves the goal of shipping new features every week. Soon, we realized that we needed full control over the infrastructure and deployment process to release a new version weekly. Besides that, we had to fulfill the requirement of hosting the web application in two data centers to achieve high availability. Our research led us to Amazon Web Services. Ten years ago, it was almost inconceivable in Germany that a bank would move important business processes to the cloud. But we were excited about the possibilities offered by AWS and simply went for it. Step by step, we learned how to build highly available architectures on AWS and how to reduce the risk of shipping improvements and features weekly or even daily. Until 2014, we helped to migrate the whole IT infrastructure to the cloud, which ended with us emptying the racks in the colocating data center.

Unfortunately, the business model behind the trading platform did not work. And therefore, Tullius Walden Bank shut down their retail business, and we were given notice.

Founding a company

In 2015, We decided to found our own company, widdix. The name “widdix” is a variation of our surname and a domain name that we had already registered. At the beginning, we didn’t really know what to focus on. As we had built web and mobile applications and had some experience with AWS, we decided to start offering three services: web development, mobile app development, and cloud computing. After a few months, we noticed a high demand for experts in Amazon Web Services as organizations in Europe started to take a first look into cloud computing. Also, Manning asked us to write a book about Amazon Web Services, which we did to fill in the free slots between projects. Moreover, we started to work as freelancers for an early AWS consulting partner in Germany, tecRacer.

Freelancing and Consulting

The story of how we moved the first German bank to the cloud and our book Amazon Web Services in Action were a perfect door opener for winning consulting projects. Over the years, we consulted countless small and large organizations on their way to the cloud. And if you know us, then you know that we don’t just talk the talk, we also walk the walk. We helped small startups to automate their release process and large enterprises to lay the foundation for a change process. We got to know many people and companies during this exciting time. And we shared what we learned with the community via cloudonaut and open-source projects (e.g., widdix/aws-cf-templates or widdix/s3-getobject-accelerator). On top of that, we learned how to negotiate a reasonable daily rate and run a business.

A serious portrait from 2017.

Building products on the side

We had a lot of fun working as consultants, but we always had the dream of developing and selling our own software. There were two reasons for this. Firstly, we enjoy developing products based on customer feedback. Secondly, from a business perspective, selling software is much more exciting than selling your working time. That’s why we worked on products on the side. We aimed to spend at least 10% of our time on side projects.

We’re still doing consulting and in need of a serious portrait in 2019.

Step by step, we tried plenty of things and failed again and again. Here is an incomplete list of projects in which we invested a lot of time and energy but which never took off and were therefore discontinued.

• PageChimp a website builder for medical doctors
• Time Series Guru a time series database as a service
• Education apps for iPad

But luckily, some of our side projects were successful and grew over time.

• bucketAV protects Amazon S3 and Cloudflare R2 from viruses and malware
• marbot configures and receives alarms and notifications from AWS via Slack or Microsoft Teams

Besides building software products, we also produced and tried to sell content like self-published e-books and video courses. In 2020, we improved our video setup.

We are a software business

Since 2023, our dream has come true, we have handed over our consulting customers and are now a software company. This gives us the time and energy to further develop our existing products and, at the same time, build up new products. We have launched a whole range of new products.

You could say that our company is a lifestyle business. We have a lot of time for deep work and can organize our lives in a self-determined way. So far, we have decided against hiring employees. From time to time, we work with freelancers who are experts in their field. Over time, we have involuntarily learned a lot about marketing, sales, accounting and taxes.

With attachmentAV we bring virus and malware protection to more and more platforms.

• Jira
• Confluence
• Salesforce
• WordPress
• Virus and Malware Scan API (SaaS)
• Virus and Malware Scan API (Self-hosted on AWS)

Besides that, we are expanding into new territories as well. HyperEnv is our solution to deploy self-hosted GitHub Actions runners on AWS with ease.

Occasionally, we get sidetracked from building products. For example, by revising our book Amazon Web Services in Action for the third time.

Thank you!

We would like to thank all our business partners and supporters who have accompanied us over the years. We look forward to the next 10 years.

In the following blog post, I will discuss three different options to move DynamoDB tables by backing up and restoring data. All three options work in harmony with Infrastructure as Code tools such as Terraform and CloudFormation. Be aware that a downtime is necessary for both options to avoid data loss.

1. Restore table from backup
2. S3 Export and Import
3. Copy data with DynamoDB CLI tool dynein

Backup and Restore

The first approach is to back up and restore a DynamoDB table. AWS provides two ways to do so:

• DynamoDB’s built-in backup and restore functionality
• AWS Backup service

Use the AWS CLI to create an on-demand backup of the source table demo-source with DynamoDB’s built-in backup and restore functionality.

$ aws dynamodb create-backup --table-name demo-source --backup-name snapshot
{
    "BackupDetails": {
        "BackupArn": "arn:aws:dynamodb:eu-west-1:111111111111:table/demo-source/backup/01736522438603-834a89d1",
        "BackupName": "snapshot",
        "BackupSizeBytes": 0,
        "BackupStatus": "CREATING",
        "BackupType": "USER",
        "BackupCreationDateTime": "2025-01-10T16:20:38.603000+01:00"
    }
}

After the backup is complete, the following command creates a new table based on the backup.

aws dynamodb restore-table-from-backup --target-table-name demo-target --backup-arn arn:aws:dynamodb:eu-west-1:111111111111:table/demo-source/backup/01736522438603-834a89d1

Be aware, that DynamoDB’s built-in backup and restore functionality does not support restoring a table in another AWS account or region. To accomplish that, use the AWS Backup service instead.

Use the AWS CLI to create a backup with AWS Backup.

aws backup start-backup-job --backup-vault-name Default --resource-arn arn:aws:dynamodb:eu-west-1:111111111111:table/demo-source --iam-role-arn arn:aws:iam::111111111111:role/service-role/AWSBackupDefaultServiceRole
{
    "BackupJobId": "0a9c3b91-9cb0-48d7-be40-a379f66921cc",
    "RecoveryPointArn": "arn:aws:backup:eu-west-1:111111111111:recovery-point:39420416-139b-4c6a-9132-66869794cc82",
    "CreationDate": "2025-01-10T16:35:03.365000+01:00",
    "IsParent": false
}

Copy the backup to another region or account (see Backup and tag copy for details).

Then, restore into a new table named demo-target.

aws backup start-restore-job --recovery-point-arn arn:aws:backup:eu-west-1:111111111111:recovery-point:39420416-139b-4c6a-9132-66869794cc82 --metadata targetTableName=demo-target --iam-role-arn arn:aws:iam::111111111111:role/service-role/AWSBackupDefaultServiceRole

Next, import the resource into the Terraform state.

% terraform import aws_dynamodb_table.target demo-target

Importing a DynamoDB table is supported by CloudFormation as well (see Import AWS resources into a CloudFormation stack with a resource import) for details.

While this is a rock solid solution that should work with large data sets without any issues, it is a little clumsy to manually import the new table to the CloudFormation stack or Terraform state. Therefore, let’s take a look at the 2nd approach.

S3 Export and Import

The second approach for moving a DynamoDB table uses the export to S3 and import from S3 functionality of DynamoDB.

The following Terraform code snippet illustrates the procedure.

1. Prepare the scenario by creating a aws_dynamodb_table.source table and an S3 bucket aws_s3_bucket.export to store the export.
2. Use the aws_dynamodb_table_item resource to add some test data to the aws_dynamodb_table.source table.
3. Use the aws_dynamodb_table_export resource to export the source table to S3 once.
4. Use the import_table attribute when creating the new aws_dynamodb_table.target table.

# Configure Terraform
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Configure the AWS provider
provider "aws" {
  region = "eu-west-1"
}

# Create the source table
resource "aws_dynamodb_table" "source" {
  name           = "demo-source"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "id"
  point_in_time_recovery {
    enabled = true
  }
  
  attribute {
    name = "id"
    type = "S"
  }
}

# Add an item to the table
resource "aws_dynamodb_table_item" "source" {
  table_name = aws_dynamodb_table.source.name
  hash_key   = aws_dynamodb_table.source.hash_key

  item = <<ITEM
{
  "id": {"S": "1"},
  "data": {"S": "demo"}
}
ITEM
}

# Create an S3 bucket to store the DynamoDB table export
resource "aws_s3_bucket" "export" {
  bucket_prefix = "dynamodb-export-"
  force_destroy = true
}

# Create an export of the DynamoDB table
resource "aws_dynamodb_table_export" "source" {
  depends_on = [aws_dynamodb_table_item.source]
  table_arn = aws_dynamodb_table.source.arn
  s3_bucket = aws_s3_bucket.export.id
}

# Create a new table based on the data export stored on S3
resource "aws_dynamodb_table" "target" {
  name           = "demo-target"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "id"
  point_in_time_recovery {
    enabled = true
  }
  
  attribute {
    name = "id"
    type = "S"
  }

  import_table {
    input_format = aws_dynamodb_table_export.source.export_format
    input_compression_type = "GZIP"
    s3_bucket_source {
        bucket = aws_s3_bucket.export.id
        key_prefix = "${trimsuffix(aws_dynamodb_table_export.source.manifest_files_s3_key, "/manifest-summary.json")}/data/"
    }
  }
}

There is one thing that bothers me about this solution. The one-time process of data migration is defined in the Terraform code, even though the migration will no longer play a role in the future.

DynamoDB CLI: dynein

The third approach is my preferred approach, at least for small data sets: dynein is a command line interface for Amazon DynamoDB written in Rust.

Use your preferred Infrastructure as Code tool to create the target table. Then use dynein to export the data from the demo-source table.

$ dy export --table demo-source --region eu-west-1 --format jsonl --output-file export.jsonl
2 items processed (10948.89 items/sec)

The export file contains all items exported from the demo-source table.

$ cat export.jsonl
{"id":"1","data":"demo"}
{"id":"2","data":"test"}

Next, import the data to the demo-target table.

$ dy import --table demo-target --region eu-west-1 --format jsonl --input-file export.jsonl
2 items processed (50419.74 items/sec)

I like the approach because it does not interfere with the infrastructure code. There are two things to watch out for: AWS might stop maintaining the dynein project, and the CLI tool might not be a good fit for large data sets.

By the way, dynein is a powerful tool to interact with DynamoDB. Here is a list of all dy commands.

admin      Admin operations such as creating/updating table or GSIlist       List tables in the region. [API: ListTables]desc       Show detailed information of a table. [API: DescribeTable]scan       Retrieve items in a table without any condition. [API: Scan]get        Retrieve an item by specifying primary key(s). [API: GetItem]query      Retrieve items that match conditions. Partition key is required. [API: Query]put        Create a new item, or replace an existing item. [API: PutItem]del        Delete an existing item. [API: DeleteItem]upd        Update an existing item. [API: UpdateItem]bwrite     Put or Delete multiple items at one time, up to 25 requests. [API: BatchWriteItem]use        Switch target table context. After you use the command you don't need to specify table every time, but you may overwrite the target table with --table (-t) option.config     Manage configuration files (config.yml and cache.yml) from command linebootstrap  Create sample tables and load test data for bootstrappingexport     Export items from a DynamoDB table and save them as CSV/JSON file.import     Import items into a DynamoDB table from CSV/JSON file.backup     Take backup of a DynamoDB table using on-demand backuprestore    Restore a DynamoDB table from backup datahelp       Print this message or the help of the given subcommand(s)

Conclusion

So, you learned about three options to move a DynamoDB table to another region or account: Backup and Restore (with DynamoDB’s built in backup functionality or AWS Backup), S3 Export and Import, or the DynamoDB CLI dynein.

One more thin aspect, that you should consider when moving large amounts of data, that I haven’t covered in this blog post: cost. The three different approaches are charged differently. So do the math before moving large amounts of data.

Here’s a simplified version of the template. This setup involves an Auto Scaling Group (ASG) that depends on VPC resources such as network ACLs, route table associations, and a VPC gateway endpoint for DynamoDB:

Resources:
  NetworkAclEntryInPrivateAllowEphemeral: {}
  NetworkAclEntryOutPrivateAllowHTTPS: {}
  RouteTableAssociationPrivate: {}
  VPCEndpointForDynamoDB: {}
  AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    DependsOn:
    - NetworkAclEntryInPrivateAllowEphemeral
    - NetworkAclEntryOutPrivateAllowHTTPS
    - RouteTableAssociationPrivate
    - VPCEndpointForDynamoDB
    Properties: {}

This approach works well if all resources are mandatory. However, in my case, the VPCEndpointForDynamoDB is an optional resource that can be toggled via a parameter. Here’s how I attempted to handle this initially:

Parameters:
  DynamoDB:
    Type: String
    Default: 'true'
    AllowedValues: ['true', 'false']
Conditions:
  HasDynamoDB: !Equals [!Ref DynamoDB, 'true']
Resources:
  NetworkAclEntryInPrivateAllowEphemeral: {}
  NetworkAclEntryOutPrivateAllowHTTPS: {}
  RouteTableAssociationPrivate: {}
  VPCEndpointForDynamoDB:
    Type: AWS::EC2::VPCEndpoint
    Condition: HasDynamoDB
    Properties: {}
  AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    DependsOn:
    - NetworkAclEntryInPrivateAllowEphemeral
    - NetworkAclEntryOutPrivateAllowHTTPS
    - RouteTableAssociationPrivate
    - !If [HasDynamoDB, VPCEndpointForDynamoDB, !Ref AWS::NoValue] # Template format error: Every DependsOn value must be a string.
    Properties: {}

In theory, this approach should work by conditionally referencing VPCEndpointForDynamoDB using an !If statement. However, CloudFormation doesn’t support !If conditions within the DependsOn attribute and returns a template format error: Every DependsOn value must be a string. This limitation required a workaround to manage conditional dependencies effectively.

The Metadata Workaround

A reliable workaround is to use the Metadata attribute that CloudFormation supports for attaching arbitrary data to resources. By leveraging Metadata, you can create an implicit dependency without using DependsOn. Here’s how:

Parameters:
  DynamoDB:
    Type: String
    Default: 'true'
    AllowedValues: ['true', 'false']
Conditions:
  HasDynamoDB: !Equals [!Ref DynamoDB, 'true']
Resources:
  NetworkAclEntryInPrivateAllowEphemeral: {}
  NetworkAclEntryOutPrivateAllowHTTPS: {}
  RouteTableAssociationPrivate: {}
  VPCEndpointForDynamoDB:
    Type: AWS::EC2::VPCEndpoint
    Condition: HasDynamoDB
    Properties: {}
  AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Metadata:
      DynamoDB: !If [HasDynamoDB, !Ref VPCEndpointForDynamoDB, !Ref AWS::NoValue] # This is the workaround
    DependsOn:
    - NetworkAclEntryInPrivateAllowEphemeral
    - NetworkAclEntryOutPrivateAllowHTTPS
    - RouteTableAssociationPrivate
    Properties: {}

In this solution, the Metadata attribute of AutoScalingGroup includes a conditional reference to VPCEndpointForDynamoDB. When HasDynamoDB is true, the !Ref VPCEndpointForDynamoDB effectively creates an implicit dependency on the optional resource. This way, if VPCEndpointForDynamoDB is toggled on, CloudFormation ensures it’s created before AutoScalingGroup.

Summary

When managing optional resources in AWS CloudFormation, the DependsOn attribute doesn’t support conditional dependencies. To work around this limitation, use the Metadata attribute to create implicit dependencies based on conditions. This approach enables you to control optional resources’ deployment order without violating CloudFormation’s syntax rules.

What’s the Cloud Control API?

Each AWS services provides an API to manage its resources. Infrastructure as Code tools like Terraform utilize the APIs to manage resources. But as the teams at AWS operate independently, there has been no standard in how APIs look like, which comes with high maintenance costs. The Cloud Control API aims to simplify maintaining Infrastructure as Code tools by providing a consistent API to create, read, update, delete, and list (CRUDL) resources in a standardized way.

AWS announced the Cloud Control API in 2021¹. I remember being disappointed by the announcement because only 366 resources had been supported back then². But things improved within the past three years: the Clod Control API now supports more than 1000 resources. Even better, AWS is adding and extending resources constantly.³

A Terraform provider leveraging the Cloud Control API: awscc

The Terraform provider awscc utilizes the Cloud Control API. The provider is automatically generated based on the Cloud Control API specification, which ensures changes are becoming available in Terraform quickly.

Using the awscc is very similar to using the good old aws provider. The following code snippet illustrates how to create an S3 bucket with the awscc provider.

terraform {
  required_version = "~>1.0"
  required_providers {
    awscc = {
      source  = "hashicorp/awscc"
      version = "~>1.0"
    }
  }
}

provider "awscc" {
  region = "eu-west-1"
}

resource "awscc_s3_bucket" "demo" {
  bucket_name = "awscc-demo"
}

Comparing Terraform providers: aws and awscc

What are the differences between the aws and awscc Terraform provider?

As illustrated in the following table, the aws provider is still ahead when it comes to covering resources. But awscc is closing the gap step by step. However, keep in mind that the comparison is counting resources and data resources only. It does not take the depth of attributes of each resource into consideration.

The awscc provider comes with a data resource to list and get each resource. However, you need to provide the IDs of the resources. There is no way to filter by any other attributes, as illustrated in the following example.

For example, the aws provider allows us to fetch information about the default VPC.

data "aws_vpc" "demo" {
  default = true # Filter the default VPC
}

In contrast, the awscc provider requires the ID to fetch information about the VPC.

data "awscc_ec2_vpc" "demo" {
  id = "vpc-b1123fd5"
}

So, the awscc comes with more data resources, but the missing ability to fetch resources by attributes other than the ID limits their usefulness.

Is awscc ahead of aws when it comes to support new AWS features?

When working with Terraform, it is frustrating to run into missing resources, hindering to use new features. So, I tried to answer the question: Is the awscc provider ahead of the aws provider when it comes to supporting new features. I went through the GitHub issues of the aws provider, looking for missing features. In the following cases, the aws provider is missing new resources while the awscc provider already supports them.

• aws_bedrockagent_prompt
• iot_job_template
• sagemaker_mlflow_tracking_server

Mix and match

The good news, you don’t have to decide between using the aws and the awscc providers. Just use both, as illustrated in the following code example, it is simple to use both providers at the same time.

terraform {
  required_version = "~>1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~>5.0"
    }
    awscc = {
      source  = "hashicorp/awscc"
      version = "~>1.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

provider "awscc" {
  region = "eu-west-1"
}

data "aws_vpc" "demo" {
  default = true
}

data "aws_subnets" "demo" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.demo.id]
  }
}


resource "aws_security_group" "demo" {
  name        = "demo"
  description = "Allow HTTPS."
  vpc_id      = data.aws_vpc.demo.id
}

resource "aws_vpc_security_group_ingress_rule" "demo" {
  security_group_id = aws_security_group.demo.id
  ip_protocol       = "tcp"
  cidr_ipv4         = "0.0.0.0/0"
  from_port         = 443
  to_port           = 443
}

resource "awscc_ec2_instance" "demo" {
  instance_type      = "m5.large"
  security_group_ids = [aws_security_group.demo.id]
  subnet_id          = data.aws_subnets.demo.ids[0]
  image_id = "ami-0fed63ea358539e44"
}

Summary

Add the awscc provider to your Infrastructure as Code toolbox, as it helps to overcome issues with missing resources in the aws provider. Mixing the awscc and aws provider is not a big deal and worth a try.

In the following post, I will dive deep into Amazon GuardDuty Malware Protection for S3. I have a lot of experience in this field. In 2015, I released an open-source project to scan files uploaded to Amazon S3. In 2019, I co-founded bucketAV - Antivirus protection for Amazon S3. I might be biased but I have seen a lot of customer use cases, judge yourself.

Scan modes

Amazon GuardDuty Malware Protection for S3 can scan files in real-time, right after the file is uploaded. Unfortunately, that’s it. Each file is scanned only once. There is no way to trigger a scan programmatically. It is also not possible to scan files just before a download happens.
Imagine a file uploaded a year ago. In the meantime, a new security vulnerability is disclosed. Unfortunately, the bad guys knew about the vulnerability long before and actively used it to attack victims. Only after the good guys discover the vulnerability, the malware scanners can detect it. All files uploaded one year ago could be infected as well. We simply don’t know because back then, the malware engine had no idea about the threat. That’s why almost all malware scanners rescan all files from time to time or on access. GuardDuty does not.

• Real-time/on-upload file scan: ✅
• Scheduled bucket scan: ❌
• On-demand bucket scan: ❌
• On-demand file scan: ❌
• On-access file scan: ❌

Mitigation

Detecting a malicious file is important. Dealing with the malicious files is key. Amazon GuardDuty Malware Protection for S3 can tag S3 objects with the scan result. You can use this tag in S3 bucket policies or IAM policies to restrict access to clean files or block access to infected files. Unfortunately, that’s it. GuardDuty does not delete infected files or quarantine files (move them to a separate S3 bucket for further analysis).

• Tag: ✅
• Delete: ❌
• Quarantine/Move: ❌

Reporting

New security tools are always great. But someone must deal with all the findings, right? Even if the mitigation is automated (like deleting infected files), you still want to know what the tool is doing. Therefore, reporting is an important aspect. Amazon GuardDuty Malware Protection for S3 is working mostly in the dark. If you subscribe to GuardDuty, you will see findings created for malicious files. If you use Amazon GuardDuty Malware Protection for S3 in standalone mode, the scan results are not stored. You get some high-level CloudWatch metrics and that’s it. No dashboard, no notifications, no reports.

• Reports: ❌
• Notifications (email): ❌
• Notifications (Slack): ❌
• Notifications (Microsoft Teams): ❌
• Dashboard: ❌
• AWS Security Hub finding integration: ⚠️ (only if you subscribe to GuardDuty)
• AWS Systems Manager OpsCenter item integration: ❌
• Amazon GuardDuty finding integration: ⚠️ (only if you subscribe to GuardDuty)

Developer

AWS is like Lego bricks. You put many bricks together to build great things. Amazon GuardDuty Malware Protection for S3 publishes events like scan results to EventBridge. EventBridge rules can trigger other AWS services. For example, to implement your quarantine logic, you can trigger a Lambda function if a file is infected. Keep in mind that moving files in S3 is not easy. You first copy the file and then delete it. But you can not copy a file that is larger than 5 GB. You need to copy it in parts which can take a lot of time so you better use Step Functions to orchestrate it to avoid Lambda timeouts.

• Amazon EventBridge integration: ✅
• Amazon SNS integration: ❌
• Amazon CloudWatch metrics integration: ✅
• AWS API to scan files: ❌

Pricing model

I will use three example workloads to demonstrate the pricing model using us-east-1 prices.

• Tiny (90 GB/month): $57.68
• Small (3 TB / month): $1,991.07
• Larger (15 TB / month): $12,696.81

In the following, I present detailed cost estimations of all examples. I end with a detailed comparison of the pricing models.

Tiny workload

The customer scans 300 files per day with an average file size of 10 MB. This results in 9,000 files and 90 GB per month. Objects are tagged with scan results. AWS region is us-east-1.

Small workload

The customer scans 20,000 files per day with an average file size of 5 MB. This results in 600,000 files and 3,000 GB per month. Objects are tagged with scan results. AWS region is us-east-1.

| | Amazon GuardDuty Malware Protection for S3 || --- | ---: | ---: | ---: || Scanning | GB: $1,800.00
files: $129.00
$1,929.00 || Infrastructure | S3: $3.48
EventBridge: $0.60
GuardDuty: optional, AWS usage dependent
$4.08 || Support | At least $57.99 || Total | $1,991.07 |

Larger workload

The customer scans 500,000 files per day with an average file size of 1 MB. This results in 15,000,000 files and 15,000 GB per month. Objects are tagged with scan results. AWS region is us-east-1.

Detailed pricing model comparison

The following table shows the various aspects of the pricing models using us-east-1 prices.

Limitations

Last but not least, we dive into the technical limitations of Amazon GuardDuty Malware Protection for S3:

• Maximum S3 object size: 5 GB
• Maximum extracted archive size: 5 GB
• Maximum number of files in an archive: 1,000
• Maximum archive depth level: 5 (archive inside archive inside archive…)

Service Maturity Table

Each service review ends with the service maturity table.

Our maturity score for Amazon GuardDuty Malware Protection for S3 is 8.0 on a scale from 0 to 10. Amazon GuardDuty Malware Protection for S3 benefits from being part of the GuardDuty service which is very mature. When we look at Feature Completeness in isolation, the picture looks less rosy. If you are interested in how bucketAV compares with Amazon GuardDuty Malware Protection for S3 I have you covered.

However, it’s important to note that GitHub Enterprise Server does not come with built-in GitHub-hosted runners, so it is necessary to deploy self-hosted runners on your own infrastructure to run GitHub Actions workflows. Learn how to deploy self-hosted runners for GitHub Enterprise Server on AWS in the following.

GitHub Enterprise Server does not support GitHub-managed runners

The GitHub documentation states it clearly:

GitHub-hosted runners are not currently supported on GitHub Enterprise Server. You can see more information about planned future support on the GitHub public roadmap.

The linked issue on GitHub’s roadmap was created in July 2020. Up until now, there is no indicator that GitHub is planning to start working on the feature in the near future.

So we’re on our own. GitHub Enterprise Server supports self-hosted runners only.

Challenges of self-hosted GitHub runners

Deploying self-hosted GitHub runners for GitHub Enterprise Server presents a few key challenges:

• Security: Since self-hosted runners operate within the company’s infrastructure, extra care must be taken to secure them and ensure they cannot be misused as an entry point for malicious actors. Proper isolation, access controls, and monitoring are crucial.
• High availability: Depending on the workload, companies may need to deploy multiple self-hosted runners and implement strategies for high availability to ensure continuous service and efficient job execution.
• Scalability: As the number of concurrent jobs or workload increases, companies may need to implement auto-scaling mechanisms to dynamically provision and deprovision self-hosted runners to handle the demand efficiently.
• Cost efficiency: Provisioning self-hosted runners can lead to underutilized resources and higher costs if the workload is not consistent or predictable. Companies need to carefully plan and manage their runner infrastructure to optimize resource utilization and control costs.
• Maintenance: Self-hosted runners require regular updates and maintenance to keep them compatible with the latest GitHub Actions versions and to apply security patches or bug fixes.

Over the years, I’ve been implementing different approaches and improved the solution step by step. Here is the architecture that I currently think is the best way to deploy self-hosted runners.

• Configure GitHub webhooks to get notified when a GitHub job is waiting for a runner.
• Launch EC2 instance on-demand and register them as just-in-time runners.
• Terminate EC2 instance after the GitHub job finished.

This approach offloads the scalability challenge to AWS, as we just start EC2 instances when we need them. Also, each GitHub jobs runs on its own virtual machine, which provides a solid isolation boundary and therefore increases security.

How to deploy self-hosted GitHub runners on AWS?

Michael and I built a simple to use solution to deploy self-hosted GitHub runners on AWS: HyperEnv for GitHub Actions Runner. With it’s 2.0.0 release HyperEnv supports GitHub Free, Pro, Team, Enterprise Cloud and Enterprise Server. Here is how to deploy HyperEnv to your AWS account.

1. Go to the AWS Marketplace and subscribe to HyperEnv for GitHub Actions Runner.
2. Create a CloudFormation stack based on the provided template.
3. Install a private GitHub app to a GitHub organization.
4. Configure the GitHub workflows to run on the self-hosted runners.

For a more detailed explanation, please refer to the HyperEnv setup guide.

Summary

GitHub Actions allows you to automate workflows directly from GitHub repositories, but GitHub Enterprise Server requires self-hosted runners which present challenges around security, availability, scalability, cost, and maintenance. A solution like HyperEnv for GitHub Actions Runner can help deploy self-hosted runners on AWS by launching EC2 instances on-demand when jobs are triggered, providing isolation and auto-scaling capabilities.

In the following, I will share my learnings from writing unit tests by using aws-sdk-client-mock by Maciej Radzikowski.

aws-sdk-client-mock is simple to use!

Let’s start with a simple example.

The following code snippet shows the index.js file containing a handler() function, which could be deployed as a Lambda function. The handler() function lists all buckets belonging to an AWS account.

import { S3Client, ListBucketsCommand } from '@aws-sdk/client-s3';

export async function handler(event, context) {
  const s3Client = new S3Client();
  const response = await s3Client.send(new ListBucketsCommand({}));
  return response.Buckets;
}

So, how do I write a unit test? I prefer using the test framework mocha to write JavaScript tests. The following snippet shows the test/test.index.js file containing the skeleton to implement a test.

import { deepStrictEqual } from 'node:assert';

import { handler } from '../index.js';

describe('demo', () => {
  it('handler', async () => {
    const now = new Date().toISOString();
    const result = await handler({}, {}); // Call the handler() function
    deepStrictEqual(result, [{ // Verify the response
      Name: 'bucket-demo-1',
      CreationDate: now
    }])
  });
});

When executing the test, the handler() function will send a request to the S3 API. But doing so is not feasible for unit testing, as it is very challenging to ensure the response matches the assumptions in the unit test.

Instead of sending requests to the AWS APIs use a common testing technique called mocking. A mock simulates a dependency. So let’s mock the AWS Java Script SDK v3 by extending the test/test.index.js file.

import { S3Client, ListBucketsCommand } from '@aws-sdk/client-s3';
import { mockClient } from 'aws-sdk-client-mock';
import { deepStrictEqual } from 'node:assert';

import {handler} from '../index.js';

const s3Mock = mockClient(S3Client); // Creates a mock

beforeEach(() => {
  s3Mock.reset(); // Reset the mock before each test
  s3Mock.rejects(new Error('not mocked')); // by default, all commands are mocked, we change this here!
});

describe('demo', () => {
  it('handler', async () => {
    const now = new Date().toISOString();
    s3Mock.on(ListBucketsCommand).resolvesOnce({ // Mock the ListBucketsCommand and return hard-coded result
      Buckets: [{
        Name: 'bucket-demo-1',
        CreationDate: now
      }]
    });
    const result = await handler({}, {});
    deepStrictEqual(result, [{
      Name: 'bucket-demo-1',
      CreationDate: now
    }]);
  });
});

Want to run the example yourself? Here is the package.json that you need to setup the example.

{
  "dependencies": {
    "@aws-sdk/client-s3": "^3.583.0",
    "aws-sdk-client-mock": "^4.0.0"
  },
  "name": "aws-mock-demo",
  "version": "1.0.0",
  "main": "index.js",
  "devDependencies": {
    "aws-sdk-client-mock": "^4.0.0",
    "mocha": "^10.2.0"
    
  },
  "scripts": {
    "test": "mocha"
  },
  "type": "module",
  "author": "",
  "license": "ISC",
  "description": ""
}

Finally, run the test.

npm i
npm test

The testing framework outputs the following results.

> aws-mock-demo@1.0.0 test
> mocha
  demo
    ✔ handler

  1 passing (5ms)

Next, let me share a some lessons learned.

Creating a mock with aws-sdk-client-mock

It took me a little bit to understand that there are two ways to create a mock.

The following code creates a mock for a given client instance.

const s3Client = new S3Client({});
const s3Mock = mockClient(s3Client);

However, in many scenarios, you don’t have access to the AWS SDK client instances. In those scenarios, here is how you globally mock a client.

const s3Mock = mockClient(S3Client);

Testing pagination

The AWS JavaScript SDK v3 comes with built-in paginators. The following snippet shows how to page through all items stored in a DynamoDB table.

import { DynamoDBClient, paginateScan } from '@aws-sdk/client-dynamodb';

const dynamodbClient = new DynamoDBClient();

export async function handler(event, context) {
  let result = '';
  const paginator = paginateScan({ // pageinateScan calls ScanCommand multiple times to iterate over all result pages
    client: dynamodbClient
  }, {
    TableName: 'demo'
  });
  for await (const page of paginator) {
    result = result + page.Items[0].Data.S;
  }
  return result;
}

To write a unit test override the underlying command, ScanCommand in this example.

import { DynamoDBClient, ScanCommand } from '@aws-sdk/client-dynamodb';
import { mockClient } from 'aws-sdk-client-mock';
import { deepStrictEqual } from 'node:assert';

const dynamodbMock = mockClient(DynamoDBClient);
import { handler } from '../index.js';

beforeEach(() => {
  dynamodbMock.reset();
});

describe('demo', () => {
  it('handler', async () => {
    dynamodbMock.on(ScanCommand, {ExclusiveStartKey: undefined}).resolvesOnce({
      Items: [{Key: {S: '1'}, Data: {S: 'Hello '}}],
      Count: 1,
      LastEvaluatedKey: {Key: {S: '1'}}
    });
    dynamodbMock.on(ScanCommand, {ExclusiveStartKey: {Key: {S: '1'}}}).resolvesOnce({
      Items: [{Key: {S: '2'}, Data: {S: 'World'}}],
      Count: 1,
      LastEvaluatedKey: {Key: {S: '2'}}
    });
    dynamodbMock.on(ScanCommand, {ExclusiveStartKey: {Key: {S: '2'}}}).resolvesOnce({
      Items: [{Key: {S: '3'}, Data: {S: '!'}}],
      Count: 1
    });
    const result = await handler({}, {});
    deepStrictEqual(result, 'Hello World!');
  });
});

Mocks vs. real-world

The tricky part when writing mocks for the AWS SDK is to ensure comatiblity with the real-world. That’s why I do not rely on unit testing. On top of that, integration testing against the AWS APIs is necessary.

Summary

aws-sdk-client-mock is a handy tool when it comes to writing unit tests for code that interacts with the AWS JavaScript SDK v3. It has never been easier to write unit tests!

The problem and solution discussed in the following also apply to OpenTofu.

Here is the typical message that terraform test will print out in case cleaning up all the resources fails.

Terraform left the following resources in state after executing
tests/default.tftest.hcl/execute, and they need to be cleaned up manually:
  - aws_subnet.private[0]
  - aws_subnet.private[1]
  - aws_subnet.public[0]
  - aws_subnet.public[1]
  - aws_vpc.this
  - ...

Leftover AWS resources are an issue, especially when running tests in an automated manner, causing unwanted costs. Therefore, I was looking for a solution to tidy up AWS resources regularly. The tool aws-nuke, by rebuy, deletes all resources belonging to an AWS account.

The following snippet shows the configuration file nuke-config.yml for aws-nuke. First, define which regions aws-nuke shall remove resources. global is needed to delete global resources like IAM roles and policies. For safety reasons, defining an account-blocklist with AWS account IDs that you never want to tidy up is necessary. Next, you define the accounts where you want to remove all resources. The filters are required to keep some essential resources, such as the IAM role and policy used by aws-nuke to access the AWS account.

regions:
- eu-west-1
- global
account-blocklist:
- '999999999999'
accounts:
  '111111111111':
    filters:
      IAMRole:
      - 'nuke'
      IAMRolePolicy:
      - type: glob
        value: "nuke -> *"

My recommendation is to run aws-nuke with the dry run option activated - which is the default - and check for resources you want to keep. Then, add a filter for those resources. Learn how to install aws-nuke.

aws-nuke -c nuke-config.yml

For example, aws-nuke deletes the VPC and subnets left over from the Terraform test.

> aws-nuke-example

eu-west-1 - EC2Subnet - 'subnet-154d844e' - would remove
eu-west-1 - EC2Subnet - 'subnet-af12a261' - would remove
eu-west-1 - EC2Subnet - 'subnet-cd2fa222' - would remove
eu-west-1 - EC2Subnet - 'subnet-51223aff' - would remove
eu-west-1 - EC2VPC - 'vpc-c6159fa1' - would remove
Scan complete: 13 total, 5 nukeable, 10 filtered.

While it’s possible to run aws-nuke from your machine to ensure leftover AWS resources are regularly cleaned up, a scheduled job is the way to go. As I’m running the command terraform test within a CI/CD pipeline on GitHub, I decided to use a scheduled GitHub workflow to run aws-nuke once a day. The following snippet illustrates how to define a GitHub workflow to regularly run aws-nuke to delete resources belonging to an AWS account.

name: 'nuke'
on:
  workflow_dispatch:
  schedule:
    - cron:  '0 0 * * *'
concurrency:
  group: 'nuke'
  cancel-in-progress: false
permissions:
  id-token: write
  contents: read
jobs:
  nuke:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: 'Assuming IAM role'
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::111111111111:role/nuke
        role-session-name: nuke
        aws-region: eu-west-1
    - name: 'Tidying up AWS resources'
      run: |
        docker run -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} --rm -v ./nuke-config.yml:/home/aws-nuke/config.yml quay.io/rebuy/aws-nuke:v2.25.0 --config /home/aws-nuke/config.yml --force --no-dry-run

By the way, have you heard about our solution HyperEnv for GitHub Actions Runner to spin up EC2 instances on-demand for executing GitHub workflow jobs?

Summary

Watch out for leftover AWS resources after executing Terraform tests. Periodically running aws-nuke ensures all AWS resources are deleted to avoid unwanted costs.

The problem: payments, subscription, and access control

I’m building a WordPress plugin to protect blogs from malware. Whenever an editor uploads a new attachment, the plugin sends the file to our API, which scans it for malware. The infrastructure consists of an Application Load Balancer (ALB) and EC2 instances running the malware engine. So, how do we charge customers for accessing the API?

Let’s break down the problem into requirements.

• Manage a subscription (create, pause, cancel, …)
• Handle payments (different payment methods, worldwide, …)
• Control access to API (API key, throttling, …)

The options: API marketplaces and payment and subscription platforms

My first idea was to use an API marketplace. The AWS Marketplace supports selling API Gateway APIs. We are already selling products through the AWS Marketplace and are pretty happy with the solution. However, the AWS Marketplace works best if potential customers are already AWS customers. As I’m aiming to sell API access to WordPress users, the hurdle of creating an AWS account seems too high.

What about more generic API marketplaces? There are a few providers out there. I had a deeper look into Rapid API. From a technical point of view, the solution looks solid. However, Rapid API targets developers who want to integrate an API into their application. I could not find a way to integrate Rapid API into the checkout process for the users of our WordPress plugin. Besides that, I concluded that Rapid API is in the early stages of collecting payments and deducting taxes worldwide.

To have complete control over the checkout process, I looked into generic payment and subscription platforms. So, I looked into Stripe and a few other solutions. My pain point with all those solutions is tax compliance. It’s quite tricky to comply with all the tax laws worldwide. Therefore, I ended up with a provider we have used for years: FastSpring. From a technical and look and feel perspective, FastSpring is getting a bit long in the tooth. But FastSpring acts as a reseller. Therefore, FastSpring is responsible for tax deductions with customers from all over the world.

I decided to use FastSpring to handle payments and subscriptions. Next, I looked for the simplest possible implementation on AWS.

The solution: API Gateway (REST APIs), usage plans, API keys, and FastSpring

After all, I came up with the following solution to monetize a REST API.

1. The customer goes to the storefront provided by FastSpring to create a subscription. FastSpring generates a license key.
2. FastSpring sends a webhook event to the API Gateway, including the subscription ID and license key.
3. The API Gateway invokes a Lambda function. The Lambda function creates an API key using the value of the license key and assigns the API key to a usage plan.
4. The customer sends a request to the API Gatway. The request includes the license key (= API key) in the header.
5. The API Gateway validates the API key and usage plan and then forwards the request to the ALB.

What I like most about the solution is its simplicity.

API Gateway REST APIs have two major limitations: the payload size is limited to 10 MB, and the request timeout is limited to 30 seconds.

Next, let’s dive into some implementation details.

The Amazon API Gateway REST APIs support usage plans and API keys. A usage plan allows you to define the target request rate per customer, which is crucial to protecting your infrastructure from accidental or malicious request flooding. Additionally, it is possible to define a quota for the maximum number of requests per day, week, or month. The following CloudFormation snippet shows how to create a usage plan limiting access to 1 request per second and 10,000 per day, for example.

It’s important to mention, that AWS does not guarantee to apply throttling and quotas 100% accuartely. Here is what the AWS documentation says: “Usage plan throttling and quotas are not hard limits, and are applied on a best-effort basis. In some cases, clients can exceed the quotas that you set. Don’t rely on usage plan quotas or throttling to control costs or block access to an API.” In our scenario, that’s a limitation we can live with.

UsagePlan:
  Type: 'AWS::ApiGateway::UsagePlan'
  Properties:
    UsagePlanName: 'demo'
    Description: '1 req/sec and 10,000 req/day'
    ApiStages:
    - ApiId: !Ref ApiGateway
      Stage: !Ref ApiStage
    Throttle:
      BurstLimit: 5
      RateLimit: 1
    Quota:
      Limit: 10000
      Period: DAY

The “new” Amazon API Gateway HTTP APIs still do not support usage plans. I’m using the “legacy” option REST APIs here.

As described above, FastSpring sends webhook events whenever customers create or cancel a subscription. The following JavaScript snippet shows how a Lambda function parses the webhook event, creates an API key, and attaches the API key to the usage plan.

import { APIGatewayClient, CreateApiKeyCommand, GetApiKeysCommand, UpdateApiKeyCommand, CreateUsagePlanKeyCommand } from '@aws-sdk/client-api-gateway';
import { createHmac } from 'node:crypto';

const apigw = new APIGatewayClient();

const WEBHOOK_SECRET = '...';
const USAGE_PLAN_ID = '...';

function isValidSignature (event) {
  const fsSignature = event.headers['X-FS-Signature'];
  const computedSignature = createHmac('sha256', WEBHOOK_SECRET).update(event.body).digest().toString('base64');
  return fsSignature === computedSignature;
}

export const handler = async (event) => {
  if (event.path === '/v1/fastspring/webhook' && event.httpMethod === 'POST') {
    if (isValidSignature(event)) {
      const body = JSON.parse(event.body);
      for (const e of body.events) {
        if (e.type === 'subscription.activated') { // Customer subscribed via FastSpring
          const apiKey = await apigw.send(new CreateApiKeyCommand({
            name: `subscription-${e.data.subscription}`,
            description: `The license and API key for FastSpring subscription ${e.data.subscription}.`,
            enabled: true,
            value: e.data.fulfillments['license_0'][0].license
          }));
          await apigw.send(new CreateUsagePlanKeyCommand({
            usagePlanId: USAGE_PLAN_ID,
            keyId: apiKey.id,
            keyType: 'API_KEY'
          }));
        }
      }
      return {
        statusCode: 200,
        headers: {
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({})
      };
    } else {
      return {
        statusCode: 403,
        headers: {
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({error: 'Invalid signature.'})
      };  
    } 
  } else {
    return {
      statusCode: 404,
      headers: {
        'Content-Type': 'application/json'
      },
      body:  JSON.stringify({error: 'Not found.'})
    };    
  }
};

Last but not least, the API Gateway must be configured to validate the API key and usage plan. The following CloudFormation snippet shows how to configure the API Gateway.

ApiGateway:
  Type: 'AWS::ApiGateway::RestApi'
  Properties:
    ApiKeySourceType: HEADER
    Body:
      'Fn::Transform':
        Name: 'AWS::Include'
        Parameters:
          Location: './api-schema.yml'
    Description: 'A cloudonaut.io example.'
    Name: 'demo'
    EndpointConfiguration:
      Types: [ 'REGIONAL']

Details are defined in the Swagger configuration file api-schema.yml references from the previous CloudFormation snippet. Note that the path /v1/demo requires an api_key to grant access. The API Gateway forwards POST requests to /v1/demo to the backend system https://example.com/api/v1/demo.

---
swagger: '2.0'
basePath: '/'
schemes:
- https
info:
  title: 'demo-api'
  version: '1.0.0'
x-amazon-apigateway-request-validators:
  basic:
    validateRequestBody: false
    validateRequestParameters: true
x-amazon-apigateway-request-validator: basic
securityDefinitions:
  api_key:
    type: "apiKey"
    name: "x-api-key"
    in: "header"
x-amazon-apigateway-gateway-responses:
  INVALID_API_KEY:
    statusCode: 401
    responseTemplates:
      'application/json': '{"error": "Invalid API key."}'
  THROTTLED:
    statusCode: 429
    responseTemplates:
      'application/json': '{"error": "Rate limit exceeded."}'
  QUOTA_EXCEEDED:
    statusCode: 429
    responseTemplates:
      'application/json': '{"error": "Quota exceeded."}'
paths:
  '/v1/demo':
    post:
      security:
      - api_key: []
      responses:
        "200":
          description: OK
      x-amazon-apigateway-integration:
        type: 'http'
        httpMethod: 'POST'
        uri: 'https://example.com/api/v1/demo'
        responses:
          default:
            statusCode: '200'
        passthroughBehavior: 'when_no_match'
        contentHandling: 'CONVERT_TO_BINARY'
definitions:
  Error:
    properties:
      error:
        type: string
    required:
    - error

Need help with implementing a similar solution? Let me know!

Summary

When selling APIs to potential customers who are most likely already AWS customers, AWS Marketplace is a great choice. However, when selling to potential customers without an AWS account, a solution consisting of API Gateway, usage plans, API keys, Lambda, and FastSpring is a simple but powerful alternative.

Do you build AMIs automatically, for example, with Packer? Learn how to automatically clean up unused AMIs to reduce EBS storage consumption and costs.

About a year ago, the tool we used to remove unused AMIs stopped working. As we could not find a working alternative, we started the project aws-amicleaner.

How does the aws-amicleaner command line tool work?

1. Include AMIs by name or tag.
2. Exclude AMIs in use, younger than N days, or the newest N images.
3. Manually confirm the list of AMIs for deletion.
4. Delete AMIs and linked EBS snapshots.

Examples

Requires Node.js >= 18.

The following example shows how to delete all AMIs in eu-west-1 matching the following conditions:

• Name starts with amiprefix-
• Older than five days
• Not in use by EC2 instances, ASGs, Launch Configurations, and Launch Templates.

Also, keep the newest three images that match the conditions.

npx aws-amicleaner --region eu-west-1 --include-name 'amiprefix-*' --exclude-newest 3 --exclude-days 5 --exclude-in-use --verbose

The following listing shows a typical confirmation screen.

+-----------+-------------+----------------------+--------------------------+---------+-----------------+-------------------------+
| Region    | ID          | Name                 | Creation Date            | Delete? | Include reasons | Exclude reasons         | 
+-----------+-------------+----------------------+--------------------------+---------+-----------------+-------------------------+
| eu-west-1 | ami-0a...72 | amiprefix-1685107232 | 2023-05-27T13:32:21.000Z | no      | name match      | days not passed, newest | 
| eu-west-1 | ami-02...3f | amiprefix-1685103569 | 2023-05-27T12:30:50.000Z | no      | name match      | days not passed, newest | 
| eu-west-1 | ami-09...d5 | amiprefix-1685095689 | 2023-05-27T10:19:59.000Z | no      | name match      | days not passed, newest | 
| eu-west-1 | ami-0f...c7 | amiprefix-1685039741 | 2023-05-26T18:47:37.000Z | no      | name match      | days not passed         | 
| eu-west-1 | ami-0f...f0 | amiprefix-1685018189 | 2023-05-26T12:49:02.000Z | no      | name match      | days not passed         | 
| eu-west-1 | ami-06...a8 | amiprefix-1685015512 | 2023-05-26T12:04:39.000Z | no      | name match      | days not passed         | 
| eu-west-1 | ami-04...44 | amiprefix-1684998014 | 2023-05-25T07:14:42.000Z | yes     | name match      |                         |
| eu-west-1 | ami-02...6e | amiprefix-1684954911 | 2023-05-24T19:15:37.000Z | yes     | name match      |                         |
| eu-west-1 | ami-0e...da | amiprefix-1684952424 | 2023-05-24T18:32:06.000Z | yes     | name match      |                         |
| eu-west-1 | ami-0b...54 | amiprefix-1684949922 | 2023-05-24T17:50:26.000Z | yes     | name match      |                         |
| eu-west-1 | ami-0b...2c | amiprefix-1684937102 | 2023-05-24T14:17:53.000Z | yes     | name match      |                         |
| eu-west-1 | ami-0e...aa | amiprefix-1684915092 | 2023-05-24T08:09:42.000Z | yes     | name match      |                         |
+-----------+-------------+----------------------+--------------------------+---------+-----------------+-------------------------+

Do you want to continue and remove 6 AMIs [y/N] ? : 

The following example shows how to delete AMIs from all regions that start with eu-west-*, which resolves to eu-west-1, eu-west-2, and eu-west-3 those days. Also, the example filters AMIs by tag key CostCenter and tag value X342-*1111 and the age of 7 days. Besides that the aws-amicleaner keeps the newest five images matching those conditions.

npx aws-amicleaner --region 'eu-west-*' --include-tag-key CostCenter --include-tag-value 'X342-*-1111'

Looking for a way to run the aws-amicleaner in an automated way. Use the --force-delete option to skip the manual approval. We use GitHub Actions to remove unused AMIs once per month.

npx aws-amicleaner --region 'eu-west-*' --include-tag-key CostCenter --include-tag-value 'X342-*-1111' --force-delete

Arguments

Want more? Here is a list of all available arguments supported by aws-amicleaner.

-h, --help            show this help message and exit
--region REGION       The AWS region, e.g. us-east-1, arg can be used more than once, wildcard * supported
--include-name INCLUDENAME
                      The name that must be present, wildcard * supported
--include-tag-key INCLUDETAGKEY
                      The tag key that must be present
--include-tag-value INCLUDETAGVALUE
                      The tag value (for the tag key) that must be present, wildcard * supported
--exclude-newest EXCLUDENEWEST
                      Exclude the newest N AMIs
--exclude-days EXCLUDEDAYS
                      Exclude AMIs from deletion that are younger than N days
--exclude-in-use, --no-exclude-in-use
                      Exclude AMIs from deletion that are in use by EC2 instances, ASGs, Launch Configurations, and Launch Templates (default: true)
-f, --force-delete, --no-force-delete
                      Skip confirmation before deletion (default: false)
--verbose, --no-verbose
                      Display additional information (default: true)

Summary

The aws-amicleaner is a small but helpfull tool to delete unused AMIs and the corresponding EBS snapshots to reduce costs.

It’s simple to get started (requires Node.js >= 18).

npx aws-amicleaner --help

That’s why I will share how to reduce costs for GitHub Actions with Octolense by Sandro Volpicella and HyperEnv for GitHub Actions Runner made by Michael and me in the following.

Do you prefer watching a video instead of reading? Here you go!

Get insights into usage with Octolense

Analyzing usage is vital when it comes to cutting costs. The pricing model of GitHub-hosted runners is simple. You pay for every build minute depending on the operating system and number of CPUs. Here is an excerpt of the GitHub Actions pricing page.

How do you find out which repository consumes the most build minutes?

Unfortunately, GitHub only provides insights into the total amount of consumed build minutes.

The ability to generate a CSV report only helps a little to analyze spending.

Date,Product,SKU,Quantity,Unit Type,Price Per Unit ($),Multiplier,Owner,Repository Slug,Username,Actions Workflow,Notes
2024-02-22,Actions,Compute - UBUNTU,30,minute,0.008,1.0,stratocumulus,android,andreaswittig,.github/workflows/build.yml,
2024-02-22,Actions,Compute - UBUNTU,90,minute,0.008,1.0,stratocumulus,ios,andreaswittig,.github/workflows/build.yml,
2024-02-22,Actions,Compute - UBUNTU,60,minute,0.008,1.0,stratocumulus,spa,andreaswittig,.github/workflows/build.yml,

Luckily, Sandro built Octolense, a simple-to-use and powerful tool to get insights into GitHub Actions usage. After connecting with GitHub, Octolense opens a dashboard showing the key metrics Billable Minutes, Paid Minutes, and Workflow Runs. Besides that, the dashboard lists the Top Cost Intensive Repositories.

And there is more. Opening the detail view of a repository unveils insights into successful and failed workflow runs.

Octolense provides valuable insights into the use of GitHub actions and allows you to identify the repositories where optimizing your CI/CD workflows will have the most significant impact.

Get started with Octolense to analyze GitHub Actions usage!

Reduce costs with self-hosted runners by HyperEnv

By default, a workflow job runs on a GitHub-hosted runner. Alternatively, you can also use self-hosted runners to execute workflow jobs. Michael and I built HyperEnv for GitHub Actions Runner to make deploying self-hosted runners on AWS as easy as possible.

How does HyperEnv work? The following diagram illustrates the components automatically deployed to your AWS account within 10 minutes.

1. GitHub sends a webhook event whenever a new job gets created.
2. HyperEnv launches an EC2 instance with GitHub runner and build environments pre-installed.
3. The GitHub runner executes the job.
4. The EC2 instance terminates.

Switching from GitHub-hosted runners to HyperEnv reduces costs by at least 30%.

The following table shows a cost estimation for 20,000 build minutes running on t3.large instances (2 vCPU, 8 GB memory, 25 GB volume) in us-east-1.

That’s cost savings of 31% compared to GitHub-hosted runners.

Besides, many workflows do not require a machine with 2 vCPUs and 8 GB memory. By choosing a smaller instance type, you can reduce the costs for GitHub Actions executed on self-hosted runners even more.

Deploy HyperEnv for GitHub Actions Runner and reduce costs by at least 30%!

Summary

Gain insights into your GitHub actions usage with Octolense and reduce costs for GitHub Actions by switching from GitHub-hosted Runners to HyperEnv.

Do you have any other tips on how to reduce costs for GitHub Actions? Let me know!

Are you defining key policies to strictly restrict access to customer-managed keys? Then, the following will blow your mind. Under some circumstances, an IAM identity (user or role) can grant itself administrator access to any customer-managed key.

This is not an unknown security vulnerability. The procedured explained here is discussed at re:Post, for example.

The story

Bob, the owner of an AWS account, is granted AdministratorAccess. He creates a customer-managed key and assigns the following key policy. He wants to ensure that no one else can delete or modify the key, so he only grants his own user administrator access to the key.

{
  "Version": "2012-10-17",
  "Statement":[{
    "Sid": "Allow access for Key Administrators",
    "Effect": "Allow",
    "Principal": {
      "AWS": [
        "arn:aws:iam::091140455148:user/bob"
      ]
    },
    "Action": [
      "kms:Create*",
      "kms:Describe*",
      "kms:Enable*",
      "kms:List*",
      "kms:Put*",
      "kms:Update*",
      "kms:Revoke*",
      "kms:Disable*",
      "kms:Get*",
      "kms:Delete*",
      "kms:TagResource",
      "kms:UntagResource",
      "kms:ScheduleKeyDeletion",
      "kms:CancelKeyDeletion"
    ],
    "Resource": "*"
  }]
}

Alice, a colleague of Bob’s, has AdministratorAccess to the same AWS account. However, due to the key policy, she is not able to delete or modify the customer-managed key created by Bob. Nevertheless, Alice found a way to delete the key anyway.

First, Alice deletes the IAM user arn:aws:iam::091140455148:user/bob, which is not an issue as iam:DeleteUser is granted by the AdministratorAccess policy.

Next, Alice updates the phone number of the AWS account with her mobile number. The AdministratorAccess policy grants her access to the account:PutContactInformation action to do so.

Afterwards, Alice contacts AWS support and tells the story that by mistake the customer-managed key is no longer accessible to any key administrator. She asks to reset the key policy. The AdministratorAccess policy grants her access to the required support:CreateCase action.

AWS verifies the key policy. Indeed, the key is not accessible to any key administrator.

Next, AWS updates the support case and asks to create a new IAM user named recovery, that shall be used to reset the key. Also, AWS provides a one-time password that will be used to verify the policy reset later.

Alice creates a new IAM user as described by AWS and asks to proceed with the policy reset by updating the support case. All she needs is access to iam:CreateUser granted by the AdministratorAccess policy.

A few hours later, AWS tries to verify that the request to reset the key policy is valid. To do so, AWS calls the number specified in the contact details of the AWS account.

And guess what? Alice answers the call. She confirms the reset of the key policy.

So, the process continues. AWS resets the key policy and grants the IAM user recovery administrator access to the customer-managed key.

Alice creates access keys for the IAM user recovery, which requires access to the iam:CreateAccessKey action.

Finally, Alice uses these access keys to schedule the customer-managed key for deletion.

The preconditions

Access to the following IAM actions is needed.

• iam:DeleteUser to delete all IAM users listed as key administrators
• iam:DeleteRole to delete all IAM roles listed as key administrators
• account:PutContactInformation to update the contact information of the AWS account
• support:CreateCase to create a support case to ask for a key policy reset
• iam:CreateUser to create the IAM user needed for the key policy reset procedure
• iam:CreateAccessKey to create credentials for the IAM user used during the key policy reset procedure

In addition, the key policy must not grant administrator access to any other existing IAM identity. In particular, the key policy must not contain the following statement, which delegates authorization entirely to IAM and is used by AWS as the default.

{
  "Sid": "Enable IAM Permissions",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111122223333:root"
   },
  "Action": "kms:*",
  "Resource": "*"
}

Last but not least, an AWS support plan is needed.

The mitigations

As an AWS customer, you could restrict access to account:PutContactInformation using service control policies (SCPs). Alternatively, you could avoid relying on key policies and completely delegate authorization to IAM.

AWS needs to find another way to deal with key policies. A half-baked procedure executed by AWS support is not a solution and undermines my trust in KMS. In my opinion, it should not be possible to reset key policies at all.

• Execute database migrations for RDS (Relational Database Service).
• Run load or integration tests against internal ALBs or NLBs (Elastic Load Balancing).
• Seed ElasticSearch, OpenSearch, or ElastiCache with data.

In the following, I will demonstrate how to access a VPC (Virtual Private Cloud) from GitHub Actions with the help of HyperEnv for GitHub Actions Runner, a solution I built recently.

What are GitHub-hosted runners?

By default, GitHub Actions executes jobs on machines provided by GitHub, so-called GitHub-hosted runners. Each GitHub-hosted runner comes with the runner application and other preinstalled tools and is available with Ubuntu Linux, Windows, or macOS operating systems.

A GitHub-hosted runner is connected to the Internet but cannot access any private networks, like a VPC on AWS. Therefore, reaching resources like an RDS database, an internal ALB, an ElasticSearch/OpenSearch domain, or an ElastiCache instance from within a running job is impossible.

How to access private networks from GitHub Actions?

There are two options to access private networks from GitHub Actions.

First, establish a tunnel between the GitHub-hosted runner and the private network, for example, by using VPN or SSH.

Second, run GitHub Actions inside your VPC.

How is that possible? GitHub Actions not only provides GitHub-hosted runners but also supports self-hosted runners. As illustrated in the following figure, an EC2 instance acting as a self-hosted runner launched in a VPC allows all jobs running on the machine to connect to resources in private subnets such as RDS, ElasticSearch, OpenSearch, ElastiCache, and more.

How to deploy a self-hosted GitHub Actions runner on AWS?

But how do you deploy a self-hosted runner on AWS? I’ve previously written about self-hosted GitHub runners on AWS. The tricky part is to come up with a scalable and cost-efficient solution.

That’s why I built HyperEnv for GitHub Actions Runner. The following figure illustrates the solution.

1. GitHub sends a webhook event when starting a job.
2. The API Gateway receives the event.
3. A Lambda function validates the event and sends a message to SQS.
4. Another Lambda function reads the message from SQS and launches an EC2 instance in an VPC of your choice.
5. The EC2 instance starts and registers the GitHub runner.
6. The GitHub runner executes the job.
7. The EC2 instance terminates itself.

Deploy HyperEnv for GitHub Actions to your AWS account to enable GitHub Actions jobs to connect with RDS, ElasticSearch, OpenSearch, ElastiCache running in your VPC.

Here are some interesting facts.

5 Things you should know about the availability of EC2 instance types

• Only 9 instance families are available in all regions: c6g, i3en, i4i, m6g, m6gd, r6g, r7g, t3, t4g.
• New instance types are rolling out slowly. For example, i7i was released in April 2025 and is available in 9% of the regions only.
• Even older instance types like m7i, which was released in August 2023, are not availble in all regions yet.
• New regions support only a small subset of all available instance types. For example, ap-east-2 supports only 25 instance families.
• In some regions, not all instance types of an instance family are supported. For example, while the instance family i4i is supported in il-central-1, the instance types i4i.24xlarge and i4i.12xlarge are not.

Want to look into the details? The following tables are for you.

Region Coverage by Instance Family

The following high-score list shows which instance families are available in most regions.