Serverless pattern: accessing public and private resources

Crossing the chasm between the old world - virtual machines isolated within a private network - and the new world - Serverless making use of publicly accessible APIs only - can be tricky. On the one hand, it is possible to configure VPC access for AWS Lambda. On the other hand, doing so comes with limitations taking away all the fun.

As soon as you are connecting your Lambda function with a VPC, the function is no longer able to access the Internet, even if you are choosing a public subnet with a route to the Internet gateway for your Lambda function (see Internet Access for Lambda Functions to learn more). So you can’t have both: access to resources within your VPC and through the Internet.

To be able to access your VPC as well as the Internet, you need to spin up a NAT Gateway. Or in some cases, you might get away with a VPC Endpoint.

The following example will illustrate the limitation and demonstrate a workaround.

Example: Query a database and publish result to CloudWatch

Let us assume your application stores the sessions of logged in users in a relational database, and you want to monitor the number of active sessions. So, you query your RDS database instance and publish the result to a custom CloudWatch metric periodically. All in all, a perfect use case for a Serverless application as you only need limited compute capacity for a few seconds every five minutes. The following figure shows the involved components:

• CloudWatch Scheduled Event is triggering the Lambda function every five minutes
• Lambda Function sends a query to the database and stores the result in the Custom CloudWatch Metric
• RDS Instance the database storing the active sessions
• VPC both the RDS instance as well as the Lambda function are placed into a private network
• NAT Gateway needed, so the Lambda function can access the CloudWatch API (public interface)

On component is taking away all the Serverless fun. Which one?

Problem: NAT Gateway

As shown in the following table the NAT Gateway is a significant cost driver.

To be fair, that is not a problem if the NAT Gateway is already in place and use by other parts of the infrastructure. Nevertheless, needing to add a NAT Gateway just to be able to store active sessions within a CloudWatch metric is not feasible.

Any idea how to work around the need of a NAT Gateway?

Workaround: Public and Private Lambda Function

Crossing the chasm requires extra creativity. The following figure shows the component allowing you to query data a database and publishing the result to CloudWatch.

1. The CloudWatch Scheduled Event is triggering the Public Lambda function.
2. The Public Lambda invokes the Private Lambda synchronously.
3. The Private Lambda queries the RDS Instance’s database.
4. The Public Lambda writes the result from the Private Lambda to the Custom CloudWatch Metric.

A 100% Serverless solution, no need for a NAT Gateway anymore.

So, what is the limitation of this workaround? The maximum amount of data passed from the Private Lambda to the Public Lambda is 6 MB.

Summary

Whenever a Lambda function needs to access resources inside and outside a VPC, you should think about splitting your tasks into a Lambda function running outside the VPC and a Lambda function running inside the VPC to avoid the need of a NAT Gateway or VPC Endpoint.