Advanced AWS Networking: Pitfalls That You Should Avoid

AWS offers shiny and powerful networking services. However, you should know about the pitfalls when designing advanced networking architectures for AWS. I will share some pitfalls that came to my attention when consulting clients to get the most out of AWS.

You will learn how to answer the following questions:

• VPC Peering or Transit Gateway?
• NAT Gateway or Public Subnet?
• VPC Endpoints or NAT Gateway?
• CloudFront or Akamai, Cloudflare, Fastly …?
• Route 53 Resolver or Public Hosted Zone?

Do you prefer listening to a podcast episode over reading a blog post? Here you go!

Let’s get started!

VPC Peering or Transit Gateway?

There are two different approaches to connect multiple VPCs. One option is to use VPC Peering. Creating a peering connection is simple: the owner of VPC A creates a peering request, and the owner of VPC B accepts the peering request. After the virtual peering is in place, all you need to do is to update the routing tables.

However, you need to set up a VPC Peering between every VPC. Therefore, the number of VPC Peering grows exponentially with the number of VPCs that you need to connect.

Since 2018, there is a second option to enable communication between networks: AWS Transit Gateway. First and foremost, Transit Gateway acts as a gateway connecting up to 5.000 networks. After attaching a VPC to a Transit Gateway, you need to update the routing tables of your subnets as well. Additionally, defining custom route tables to configure the routing within the Transit Gateway is possible as well.

At first, it looks like choosing Transit Gateway over VPC Peering is always a good idea. But that’s no longer true when having a look at the pricing.

A pricing example:

• Connect 4 VPCs with each other
• 2 VPCs connected with the on-premises network via VPN
• 1,000 GB traffic between VPCs
• 500 GB outgoing traffic via VPN

So using Transit Gateway doubles monthly costs for your networking infrastructure. Let’s have a look at the pricing details.

• Attaching a VPC to a Transit Gateway costs $36.00 per month.
• A VPN connection costs $36.00 per month.
• Traffic costs are the same for VPC Peering and Transit Gateway.

The baseline costs for a Site-to-Site VPN connect are $36.00 per month. The same is valid for attaching a VPC to a Transit Gateway. As long as you don’t need more than one VPN connection per VPC, you are better off with VPC Peering from a pricing perspective. Keep in mind that I haven’t taken the complexity for managing VPN connections and providing the on-premises hardware into account here.

NAT Gateway or Public Subnet?

AWS advocates to divide your VPC into public and private subnets. A public subnet comes with a route to the Internet Gateway. Therefore, a public subnet enables incoming and outgoing connections to the Internet. A private subnet is neither accessible from the Internet, nor is it possible to establish a connection to the Internet.

However, it is quite common that EC2 instances need to connect to a resource via the Internet. This is where the NAT Gateway comes in. The NAT Gateway enables outgoing Internet connectivity for a private subnet. It is important to note that you need to create a NAT Gateway for every Availability Zone that you have created private subnets to achieve high availability.

The described network architecture consisting of public subnets, private subnets, and NAT gateways works fine but comes with two downsides.

If keeping costs to a minimum is essential, the baseline costs of $32.00 per month per NAT Gateway could be a show stopper. When using three AZs, you will pay $96.00 per month for three NAT Gateways.

The NAT Gateway also increases costs for outbound traffic. You have to pay a premium of $0.045 per GB flowing from a private subnet to the Internet. That’s raising the costs for outgoing traffic by 50%.

Let’s have a look at a pricing example.

In summary, following the standard network architecture on AWS can become quite costly.

Keep in mind that the NAT Gateway bandwidth can scale up to 45 Gbps. Check out our CloudFormation templates to learn how to monitor this resource limit!

What’s the alternative? Place your workload - most likely, your EC2 instances - into the public subnets. By doing so, the NAT Gateways are no longer needed. Compared to the example with NAT Gateway from above, you will save $141.00 per month.

Keep in mind that your EC2 instances are now reachable from the Internet via their public IP addresses. Make sure you are blocking unwanted incoming traffic by making use of Security Groups and Network Access Control Lists.

When designing the network architecture, keep the costs for the NAT Gateways in mind. I have to mention at well, that deviation from the standard can cause troubles. For example, an external auditor is most likely not happy with placing an EC2 instance into a public subnet.

VPC Endpoints or NAT Gateway?

AWS services like EC2, RDS, and ElastiCache come with an Elastic Network Interface (ENI), which enables communication from within your VPCs. However, many AWS services provide a REST API, available via the Internet only. A few examples: S3, DynamoDB, CloudWatch, SQS, and Kinesis.

There are three options to make these services accessible from private subnets:

• A Gateway Endpoints is free of charge, but are only available for S3 and DynamoDB.
• An Interface Endpoint costs $7.20 per month and AZ plus $0.01 per GB and is available for most AWS services.
• A NAT Gateway can be used to access AWS services or any other services with a public API. Costs are $32.40 per month and AZ plus $0.045 per GB.

Keep the following rules of thumb in mind when designing your network architecture.

• Adding Gateway Endpoints for S3 and DynamoDB should the default.
• Do you need to access non-AWS resources via the Internet, add a NAT Gateway. Do the math if traffic to AWS services justifies additional Interface Endpoints.
• Are you only accessing AWS services from the private subnets? No more than four different services? Use Interface Endpoints. Otherwise, do the math to calculate costs for Interface Endpoints and NAT Gateway.

CloudFront or Akamai, Cloudflare, Fastly …?

Using a Content Delivery Network (CDN) to cache static content at edge locations close to your users improves the performance of your web application a lot. Read How we run our blog cloudonaut.io to learn how we are making use of CloudFront, Amazon’s Content Delivery Network.

Besides CloudFront, there are other popular CDN providers out there: Akamai, Cloudflare, and Fastly to name a few. When choosing your CDN provider, you should take into consideration, that outbound traffic from AWS to the Internet is priced at roundabout $0.09 per GB. There is one exception to that rule: when using CloudFront, you are not paying for outbound traffic from the AWS region where you operate your infrastructure. You are only paying for the outbound traffic from CloudFront to the Internet, which is valid for other CDN providers as well.

To put that in other words: when using another CDN provider than CloudFront, you are paying twice for outbound traffic. That’s not fair, but you have to deal with it.

Route 53 Resolver or Public Hosted Zone?

DNS is a crucial component of your network architecture.

To manage DNS names for your applications running on AWS with ease - for example, by using CloudFormation and Terraform - using a Private Hosted Zone is the right choice. But how do you enable machines from your Corporate Network to resolve records from the Private Hosted Zone? The simplest solution is to create a Route 53 Resolver Endpoint. The internal name server will forward the request for aws.myintra.net to the Inbound Endpoint. The Inbound Endpoint is capable of resolving names from the Private Hosted Zone.

Resolving names in the other direction is possible as well. To do so, you need to create Outbound Endpoints and define rules that forward queries to your internal name server.

In general, Route 53 Resolvers are a perfect fit for hybrid scenarios. Besides some minor technical changeless, that I will not discuss in this article.

However, Route 53 Resolvers are adding high costs to your networking infrastructure. You will pay $180 per month for an inbound endpoint deployed into two availability zones. The same is true for outbound endpoints. So the baseline costs for your name servers will be $360 per month.

On top of that, you will pay for recursive DNS queries.

• $0.40 per million queries (first 1 Billion queries per month)
• $0.20 per million queries (over 1 Billion queries per month)

From my point of view, there are two ways to minimize costs caused by Route 53 Resolvers.

1. Share outbound resolvers with all your VPCs in the same region, even across accounts. Read Simplify DNS management in a multi-account environment with Route 53 Resolver to learn more.
2. Use a Public Hosted Zone instead of a Private Hosted Zone, which makes it unnecessary to create an inbound resolver.

Instead of using a private name server to manage your records for aws.intra.net use a Public Hosted Zone, which allows you to resolve DNS requires over public name servers. Both from your VPC as well as from your corporate network.

Some might say, that publishing private IP addresses to a public zone is a no go. I understand the argument. But I’m not sure if the argument is still valid in 2020. For example, many AWS services publish private IP addresses to public zones by default (e.g., ALB, RDS, …) as well.

In any case, do the math when adding Route 53 Endpoints to your architecture and balance the pros and cons according to your scenario.

Summary

Remember the following pitfalls when designing your AWS network architecture.

1. A Transit Gateway simplifies peering VPCs. However, there is a baseline costs of $36.00 per month for each VPC attached to the Transit Gateway.
2. Adding NAT Gateways to your architecture costs $96.00 per month for 3 availability zones. Also, the costs for outbound network traffic will increase by 50%. From a economical point of view it makes sense to place workloads with high outgoing network throughput into public subnets.
3. Always add VPC Endpoints for S3 and DynamoDB. But do the math, when adding VPC Endpoints for other AWS services to your network. Using a NAT Gateway might be cheaper.
4. When choosing a CDN provider besides CloudFront, make sure you have taken into account the additional costs for outbound traffic.
5. Share Route 53 Outbound Resolvers between VPCs and AWS accounts to reduce costs. Think about using Public Hosted Zones instead of paying $180.00 per month for an Route 53 Inbound Resolver.