Private subnets are broken on AWS

Andreas WittigUpdated 23 Aug 2016

tl;dr
Think twice if you’re planning to use a VPC architecture containing a private subnet that includes EC2 instances that need to connect to the public Internet, especially if you have to do many requests or transfer a lot of data.

Being able to build a private network in the cloud with Virtual Private Cloud (VPC) is a key feature of AWS. But private subnets are broken on AWS.

A typical VPC setup consists of public and private subnets, as shown in the following figure. An EC2 instance in the private subnet running an application can’t connect to the Internet by default. But often there is a need to do that for the following reasons:

  • The EC2 instance needs to call the AWS API (DynamoDB, Route 53, and so on).
  • The EC2 instance needs to download packages from a public repository (yum, apt, and so on) to be provisioned or to install updates.

Traffic to 0.0.0.0/0 from the private subnet is routed to a single EC2 instance: the NAT instance. You can configure this with the help of the route table attached to the private subnet.

Andreas and Michael Wittig

Please support our work!

We have published 327 articles, 41 podcast episodes, and 15 videos. It's all free and means a lot of work in our spare time.

Thanks to Alex DeBrie, e9e4e5f0faef, Goran Opacic, jhoadley, Thorsten Hoeger, Todd Valentine, Vince Fulco, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt. It gives us great pleasure to send our t-shirts all over the world.

With your help, we can continue to produce independent & high-quality content focused on AWS. Please support us!

Support us

Unfortunately, all traffic from the private subnet to the Internet has to pass a single EC2 instance. This is a problem because

  • The NAT instance becomes a single point of failure.
  • The NAT instance can only be scaled vertically.

There are best practices for building an HA setup for your NAT instance. For example, the article High Availability for Amazon VPC NAT Instances: An Example. The limit for scaling vertically is sufficient for most use cases. But is using a private subnet to prevent access from the Internet to your app server worth all the trouble?

Probably not. If you want an app server that can’t be reached from the Internet but that can connect to other services over the Internet, you can use the following setup (illustrated in the next figure):

  • App server running with a public IP address
  • App server running in a public subnet with a route to the Internet gateway
  • App server running in a security group that prohibits all incoming traffic
  • ACL attached to the subnet allowing only incoming high ports from the public Internet

If you need to, you can create an additional public subnet that allows connections from the public Internet (for example, to run an SSH bastion host).

Andreas Wittig

Andreas Wittig

I launched cloudonaut.io in 2015 with my brother Michael. Since then, we have published hundreds of articles, podcast episodes, and videos. It’s all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.
Have you learned something new by reading, listening, or watching our content? If so, we kindly ask you to support us in producing high-quality & independent AWS content. We look forward to sharing our AWS knowledge with you.

Support us

Feedback? Questions? You can reach me via Email, Twitter, or LinkedIn.