(Erratum) VPC Endpoint increases DynamoDB latency by 30%

Andreas Wittig – 20 Aug 2020

Our reader Tom wrote in to tell me, that the latency for read requests to DynamoDB increased significantly after enabling a VPC endpoint a few weeks ago. Someone else reported a similar problem in the AWS discussion forums as well. Therefore, I started to investigate to write this article. After many hours of benchmarking, I came to the conclusion that using a VPC endpoint to connect to DynamoDB increase latency by 30% compared to connections through a internet gateway or NAT gateway. Shortly, after publishing the article, Petar send me a message via Twitter to tell me that something must be wrong with my benchmark. Unfortunatly, that was correct. I made a mistake when measuring the read latency for DynamoDB from EC2.

Therefore, I have depublished the original blog post. Instead you will find a post mortem analysis in the following.

VPC Endpoint increases DynamoDB latency by 30%

The problem starts with Node.js and the AWS SDK. I was using the following script to measure the read latency for DynamoDB.

const AWS = require('aws-sdk')
const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
AWS.config.maxRetries = 0;

async function benchmark() {
for (let index = 0; index < 10000; index++) {
let start = new Date().getTime();
let result = await dynamodb.getItem({
Key: {
"id": {
S: "1"
}
},
TableName: "benchmark"
}).promise();
let end = new Date().getTime();
console.log(`${index},${end-start}`);
await new Promise(r => setTimeout(r, 50));
}
}

benchmark();

With that code, requests through a VPC endpoint took about 10 ms - about 30% - longer than when connecting through an Internet gateway.

The problem with that Node.js code? Each DynamoDB API request create a new TCP connection. Doing so adds latency, and it seems like establishing a new TCP connection is taking about 30% longer when using a VPC endpoint.

Cover of Rapid Docker on AWS

Become a Docker on AWS professional!

Our book Rapid Docker on AWS is designed for DevOps engineers and web developers who want to run dockerized web applications on AWS. We lead you with many examples: From dockerizing your application to Continuous Deployment and Infrastructure as Code on AWS. No prior knowledge of Docker and AWS is required. Get the first chapter for free!

However, you should not use the Node.js default in production. Instead, you need to tell the AWS SDK to reuse exsisting connections. To do so, I modified my code as described in Reusing Connections with Keep-Alive in Node.js.

const AWS = require('aws-sdk')
const https = require('https');
const agent = new https.Agent({
keepAlive: true
});
const dynamodb = new AWS.DynamoDB({httpOptions: {agent}});
AWS.config.maxRetries = 0;

async function benchmark() {
for (let index = 0; index < 10000; index++) {
let start = process.hrtime.bigint();
let result = await dynamodb.getItem({
Key: {
"id": {
S: "1"
}
},
TableName: "benchmark"
}).promise();
let end = process.hrtime.bigint();
console.log(`${index},${end-start}`);
await new Promise(r => setTimeout(r, 50));
}
}

benchmark();

With that modification I could no longer measure any significant differences between connecting to DynamoDB through an Internet gateway, NAT gateway, or VPC endpoint.

I’m sorry about my mistake. Michael and I are working hard on publishing high-quality content.

Become a cloudonaut supporter

Andreas Wittig

Andreas Wittig ( Email, Twitter, or LinkedIn )

We launched the cloudonaut blog in 2015. Since then, we have published 360 articles, 50 podcast episodes, and 48 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Christopher Hipwell, Jaap-Jan Frans, Jason Yorty, Jeff Finley, Jens Gehring, jhoadley, Johannes Grumböck, Johannes Konings, John Culkin, Jonas Mellquist, Juraj Martinka, Kamil Oboril, Ken Snyder, Markus Ellers, Ross Mohan, Ross Mohan, sam onaga, Satyendra Sharma, Shawn Tolidano, Simon Devlin, Thorsten Hoeger, Todd Valentine, Victor Grenu, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.