Private subnets are broken on AWS

tl;dr
Think twice if you’re planning to use a VPC architecture containing a private subnet that includes EC2 instances that need to connect to the public Internet, especially if you have to do many requests or transfer a lot of data.

Being able to build a private network in the cloud with Virtual Private Cloud (VPC) is a key feature of AWS. But private subnets are broken on AWS.

A typical VPC setup consists of public and private subnets, as shown in the following figure. An EC2 instance in the private subnet running an application can’t connect to the Internet by default. But often there is a need to do that for the following reasons:

• The EC2 instance needs to call the AWS API (DynamoDB, Route 53, and so on).
• The EC2 instance needs to download packages from a public repository (yum, apt, and so on) to be provisioned or to install updates.

Traffic to 0.0.0.0/0 from the private subnet is routed to a single EC2 instance: the NAT instance. You can configure this with the help of the route table attached to the private subnet.

Unfortunately, all traffic from the private subnet to the Internet has to pass a single EC2 instance. This is a problem because

• The NAT instance becomes a single point of failure.
• The NAT instance can only be scaled vertically.

There are best practices for building an HA setup for your NAT instance. For example, the article High Availability for Amazon VPC NAT Instances: An Example. The limit for scaling vertically is sufficient for most use cases. But is using a private subnet to prevent access from the Internet to your app server worth all the trouble?

Probably not. If you want an app server that can’t be reached from the Internet but that can connect to other services over the Internet, you can use the following setup (illustrated in the next figure):

• App server running with a public IP address
• App server running in a public subnet with a route to the Internet gateway
• App server running in a security group that prohibits all incoming traffic
• ACL attached to the subnet allowing only incoming high ports from the public Internet

If you need to, you can create an additional public subnet that allows connections from the public Internet (for example, to run an SSH bastion host).