šŸ‘‰ AWS Debug Games (Beta) - Prove your AWS expertise by solving tricky challenges.

šŸ‘‰ AWS Debug Games - Prove your AWS expertise.

(Erratum) VPC Endpoint increases DynamoDB latency by 30%

Andreas Wittig ā€“ 20 Aug 2020

Our reader Tom wrote in to tell me, that the latency for read requests to DynamoDB increased significantly after enabling a VPC endpoint a few weeks ago. Someone else reported a similar problem in the AWS discussion forums as well. Therefore, I started to investigate to write this article. After many hours of benchmarking, I came to the conclusion that using a VPC endpoint to connect to DynamoDB increase latency by 30% compared to connections through a internet gateway or NAT gateway. Shortly, after publishing the article, Petar send me a message via Twitter to tell me that something must be wrong with my benchmark. Unfortunatly, that was correct. I made a mistake when measuring the read latency for DynamoDB from EC2.

Therefore, I have depublished the original blog post. Instead you will find a post mortem analysis in the following.

VPC Endpoint increases DynamoDB latency by 30%

The problem starts with Node.js and the AWS SDK. I was using the following script to measure the read latency for DynamoDB.

const AWS = require('aws-sdk')
const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
AWS.config.maxRetries = 0;

async function benchmark() {
for (let index = 0; index < 10000; index++) {
let start = new Date().getTime();
let result = await dynamodb.getItem({
Key: {
"id": {
S: "1"
}
},
TableName: "benchmark"
}).promise();
let end = new Date().getTime();
console.log(`${index},${end-start}`);
await new Promise(r => setTimeout(r, 50));
}
}

benchmark();

With that code, requests through a VPC endpoint took about 10 ms - about 30% - longer than when connecting through an Internet gateway.

The problem with that Node.js code? Each DynamoDB API request create a new TCP connection. Doing so adds latency, and it seems like establishing a new TCP connection is taking about 30% longer when using a VPC endpoint.


Looking for a new challenge?

  • DEMICON

    Cloud Operations Lead

    DEMICON • AWS Advanced Consulting Partner • Remote (Europe)
    service-delivery-management hiring devops platform

However, you should not use the Node.js default in production. Instead, you need to tell the AWS SDK to reuse exsisting connections. To do so, I modified my code as described in Reusing Connections with Keep-Alive in Node.js.

const AWS = require('aws-sdk')
const https = require('https');
const agent = new https.Agent({
keepAlive: true
});
const dynamodb = new AWS.DynamoDB({httpOptions: {agent}});
AWS.config.maxRetries = 0;

async function benchmark() {
for (let index = 0; index < 10000; index++) {
let start = process.hrtime.bigint();
let result = await dynamodb.getItem({
Key: {
"id": {
S: "1"
}
},
TableName: "benchmark"
}).promise();
let end = process.hrtime.bigint();
console.log(`${index},${end-start}`);
await new Promise(r => setTimeout(r, 50));
}
}

benchmark();

With that modification I could no longer measure any significant differences between connecting to DynamoDB through an Internet gateway, NAT gateway, or VPC endpoint.

Iā€™m sorry about my mistake. Michael and I are working hard on publishing high-quality content.

Become a cloudonaut supporter

Andreas Wittig

Andreas Wittig ( Email Twitter LinkedIn Mastodon )

We launched the cloudonaut blog in 2015. Since then, we have published 365 articles, 67 podcast episodes, and 67 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, Christopher Hipwell, e9e4e5f0faef, Jason Yorty, Jeff Finley, jhoadley, Johannes Konings, John Culkin, Jonathan Deamer, Juraj Martinka, Ken Snyder, Markus Ellers, Oriol Rodriguez, Ross Mohan, sam onaga, Satyendra Sharma, Simon Devlin, Todd Valentine, Victor Grenu, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.