Serverless pattern: accessing public and private resources

Andreas Wittig – 23 Mar 2018

Crossing the chasm between the old world - virtual machines isolated within a private network - and the new world - Serverless making use of publicly accessible APIs only - can be tricky. On the one hand, it is possible to configure VPC access for AWS Lambda. On the other hand, doing so comes with limitations taking away all the fun.

As soon as you are connecting your Lambda function with a VPC, the function is no longer able to access the Internet, even if you are choosing a public subnet with a route to the Internet gateway for your Lambda function (see Internet Access for Lambda Functions to learn more). So you can’t have both: access to resources within your VPC and through the Internet.

To be able to access your VPC as well as the Internet, you need to spin up a NAT Gateway. Or in some cases, you might get away with a VPC Endpoint.

The following example will illustrate the limitation and demonstrate a workaround.

Example: Query a database and publish result to CloudWatch

Let us assume your application stores the sessions of logged in users in a relational database, and you want to monitor the number of active sessions. So, you query your RDS database instance and publish the result to a custom CloudWatch metric periodically. All in all, a perfect use case for a Serverless application as you only need limited compute capacity for a few seconds every five minutes. The following figure shows the involved components:

  • CloudWatch Scheduled Event is triggering the Lambda function every five minutes
  • Lambda Function sends a query to the database and stores the result in the Custom CloudWatch Metric
  • RDS Instance the database storing the active sessions
  • VPC both the RDS instance as well as the Lambda function are placed into a private network
  • NAT Gateway needed, so the Lambda function can access the CloudWatch API (public interface)

Serverless Anti Pattern for accessing public and private resources

On component is taking away all the Serverless fun. Which one?

Problem: NAT Gateway

As shown in the following table the NAT Gateway is a significant cost driver.


Looking for a new challenge?

  • tecRacer

    Cloud Consultant

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Spain, and Switzerland
    AWS only Infrastructure as Code EC2 Containers Serverless
  • tecRacer

    Cloud Migration Specialist

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Spain, and Switzerland
    Lift&Shift Transformation EC2 RDS VPC

Service Monthly Costs
Lambda (Free Tier) USD 0.00
CloudWatch (Custom Metric) USD 0.40
NAT Gateway USD 32.40

To be fair, that is not a problem if the NAT Gateway is already in place and use by other parts of the infrastructure. Nevertheless, needing to add a NAT Gateway just to be able to store active sessions within a CloudWatch metric is not feasible.

Any idea how to work around the need of a NAT Gateway?

Workaround: Public and Private Lambda Function

Crossing the chasm requires extra creativity. The following figure shows the component allowing you to query data a database and publishing the result to CloudWatch.

  1. The CloudWatch Scheduled Event is triggering the Public Lambda function.
  2. The Public Lambda invokes the Private Lambda synchronously.
  3. The Private Lambda queries the RDS Instance’s database.
  4. The Public Lambda writes the result from the Private Lambda to the Custom CloudWatch Metric.

Serverless Pattern for accessing public and private resources

A 100% Serverless solution, no need for a NAT Gateway anymore.

So, what is the limitation of this workaround? The maximum amount of data passed from the Private Lambda to the Public Lambda is 6 MB.

Summary

Whenever a Lambda function needs to access resources inside and outside a VPC, you should think about splitting your tasks into a Lambda function running outside the VPC and a Lambda function running inside the VPC to avoid the need of a NAT Gateway or VPC Endpoint.

Become a cloudonaut supporter

Andreas Wittig

Andreas Wittig ( Email, Twitter, or LinkedIn )

We launched the cloudonaut blog in 2015. Since then, we have published 345 articles, 45 podcast episodes, and 37 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Jaap-Jan Frans, Jason Yorty, Jeff Finley, Jens Gehring, jhoadley, Johannes Grumböck, John Culkin, Jonas Mellquist, Juraj Martinka, Kamil Oboril, Ken Snyder, Ross Mohan, Ross Mohan, sam onaga, Shawn Tolidano, Thorsten Hoeger, Todd Valentine, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.