(Erratum) VPC Endpoint increases DynamoDB latency by 30%

Andreas Wittig – 20 Aug 2020

Our reader Tom wrote in to tell me, that the latency for read requests to DynamoDB increased significantly after enabling a VPC endpoint a few weeks ago. Someone else reported a similar problem in the AWS discussion forums as well. Therefore, I started to investigate to write this article. After many hours of benchmarking, I came to the conclusion that using a VPC endpoint to connect to DynamoDB increase latency by 30% compared to connections through a internet gateway or NAT gateway. Shortly, after publishing the article, Petar send me a message via Twitter to tell me that something must be wrong with my benchmark. Unfortunatly, that was correct. I made a mistake when measuring the read latency for DynamoDB from EC2.

Therefore, I have depublished the original blog post. Instead you will find a post mortem analysis in the following.

VPC Endpoint increases DynamoDB latency by 30%

The problem starts with Node.js and the AWS SDK. I was using the following script to measure the read latency for DynamoDB.

const AWS = require('aws-sdk')
const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
AWS.config.maxRetries = 0;

async function benchmark() {
for (let index = 0; index < 10000; index++) {
let start = new Date().getTime();
let result = await dynamodb.getItem({
Key: {
"id": {
S: "1"
}
},
TableName: "benchmark"
}).promise();
let end = new Date().getTime();
console.log(`${index},${end-start}`);
await new Promise(r => setTimeout(r, 50));
}
}

benchmark();

With that code, requests through a VPC endpoint took about 10 ms - about 30% - longer than when connecting through an Internet gateway.

The problem with that Node.js code? Each DynamoDB API request create a new TCP connection. Doing so adds latency, and it seems like establishing a new TCP connection is taking about 30% longer when using a VPC endpoint.

Cover of Rapid Docker on AWS

Become a Docker on AWS professional!

Our book Rapid Docker on AWS is designed for DevOps engineers and web developers who want to run dockerized web applications on AWS. We lead you with many examples: From dockerizing your application to Continuous Deployment and Infrastructure as Code on AWS. No prior knowledge of Docker and AWS is required. Get the first chapter for free!

However, you should not use the Node.js default in production. Instead, you need to tell the AWS SDK to reuse exsisting connections. To do so, I modified my code as described in Reusing Connections with Keep-Alive in Node.js.

const AWS = require('aws-sdk')
const https = require('https');
const agent = new https.Agent({
keepAlive: true
});
const dynamodb = new AWS.DynamoDB({httpOptions: {agent}});
AWS.config.maxRetries = 0;

async function benchmark() {
for (let index = 0; index < 10000; index++) {
let start = process.hrtime.bigint();
let result = await dynamodb.getItem({
Key: {
"id": {
S: "1"
}
},
TableName: "benchmark"
}).promise();
let end = process.hrtime.bigint();
console.log(`${index},${end-start}`);
await new Promise(r => setTimeout(r, 50));
}
}

benchmark();

With that modification I could no longer measure any significant differences between connecting to DynamoDB through an Internet gateway, NAT gateway, or VPC endpoint.

I’m sorry about my mistake. Michael and I are working hard on publishing high-quality content.

Tags: aws vpc dynamodb
Andreas Wittig

Andreas Wittig

I’m an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I’m writing this blog and all other projects together with my brother Michael.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

You can contact me via Email, Twitter, and LinkedIn.

Briefcase icon
Hire me