🆕 Video Course Out Now: Rapid Docker on AWS

Private subnets are broken on AWS

Andreas Wittig – 01 Jul 2015 (updated 23 Aug 2016)

tl;dr
Think twice if you’re planning to use a VPC architecture containing a private subnet that includes EC2 instances that need to connect to the public Internet, especially if you have to do many requests or transfer a lot of data.

Being able to build a private network in the cloud with Virtual Private Cloud (VPC) is a key feature of AWS. But private subnets are broken on AWS.

A typical VPC setup consists of public and private subnets, as shown in the following figure. An EC2 instance in the private subnet running an application can’t connect to the Internet by default. But often there is a need to do that for the following reasons:

  • The EC2 instance needs to call the AWS API (DynamoDB, Route 53, and so on).
  • The EC2 instance needs to download packages from a public repository (yum, apt, and so on) to be provisioned or to install updates.

Traffic to 0.0.0.0/0 from the private subnet is routed to a single EC2 instance: the NAT instance. You can configure this with the help of the route table attached to the private subnet.

Cover of Amazon Web Services in Action

Level up, strengthen your AWS skills.

Our book Amazon Web Services in Action is a comprehensive introduction to computing, storing, and networking in the AWS cloud. You'll find clear, relevant coverage of all the essential AWS services, emphasizing best practices for security, high availability, and scalability. Get the first chapter for free!

Unfortunately, all traffic from the private subnet to the Internet has to pass a single EC2 instance. This is a problem because

  • The NAT instance becomes a single point of failure.
  • The NAT instance can only be scaled vertically.

There are best practices for building an HA setup for your NAT instance. For example, the article High Availability for Amazon VPC NAT Instances: An Example. The limit for scaling vertically is sufficient for most use cases. But is using a private subnet to prevent access from the Internet to your app server worth all the trouble?

Probably not. If you want an app server that can’t be reached from the Internet but that can connect to other services over the Internet, you can use the following setup (illustrated in the next figure):

  • App server running with a public IP address
  • App server running in a public subnet with a route to the Internet gateway
  • App server running in a security group that prohibits all incoming traffic
  • ACL attached to the subnet allowing only incoming high ports from the public Internet

If you need to, you can create an additional public subnet that allows connections from the public Internet (for example, to run an SSH bastion host).

Andreas Wittig

Andreas Wittig

I’m an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I’m writing this blog and all other projects together with my brother Michael.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

You can contact me via Email, Twitter, and LinkedIn.

Briefcase icon
Hire me