AWS Account Structure: Think twice before using AWS Organizations

What is an AWS account? I like to use the following two ways to describe the concept of an AWS account: a tenant in Amazon’s multi-tenant cloud or a virtual data center.

When running multiple workloads and environments using numerous AWS accounts is the best approach to draw the line between the following aspects:

• Billing and Cost Management
• Identity and Access Management
• Limit Management: Resources and API Requests

Being able to isolate workloads and environments from each other is one of the secrets of the success of cloud journeys. Unfortunately, AWS started to weaken that principle with AWS Organizations during the last few years.

Do you prefer listening to a podcast episode over reading a blog post? Here you go!

Evolution

A few years ago, I started exploring AWS by creating an AWS account with my private credit card. Step by step, I deployed more and more workloads and environments to my AWS account.

Later I learned, that AWS encourages us to create multiple accounts to isolate environments from each other. That blew my mind, as many web services even prohibit creating more than one account. So instead of running different workloads and environments within the same account, I created a second account to isolate the test environment from the production environment.

2010: Multiple AWS Accounts + Consolidated Billing

Hard to believe, but true: AWS introduced consolidated billing in 2010. That solved two problems:

1. A single monthly bill accumulates the spending among many AWS accounts.
2. Benefit from volume pricing across more than one AWS account.

That reduced the burden of managing more than one AWS account significantly.

2017: AWS Organizations 1.0: account management and billing

As more and more Enterprise customers chose to move their workloads to the cloud, AWS focused more on their needs for managing hundreds or even thousands of AWS accounts centrally. Therefore, AWS introduced AWS Organizations in 2017.

AWS Organizations is designed to govern your AWS accounts and started with two main features:

1. Ability to create and manage AWS accounts via the AWS Management Console and API.
2. Restricting the usage of AWS services per AWS account with a Service Control Policy (SCP).

2020: AWS Organizations 2.0: services are operating at an organization level

AWS added more and more features allowing us to govern cloud infrastructures across AWS accounts centrally. A few examples:

• When creating an account via AWS Organizations, an IAM role granting administrator access to the root account (also called master or payer account) is added to the new account by default. Therefore, an administrator for the root account of your organization gets administrator access to all AWS accounts belonging to your organization as well.
• AWS CloudFormation Stack Sets to deploy your Infrastructure as Code templates to multiple/all AWS accounts within your organization.
• The AWS Firewall Manager enables you to implement Web Application Firewall (WAF) rules for all Application Load Balancers and CloudFront distributions across multiple/all AWS accounts belonging to your organization.

The evolution from consolidated billing to AWS Organizations with services that are operating at an organization level sounds like a success story. But I have more and more reservations. Read on!

The truth about centralism

It doesn’t work. Centralism causes more problems than it solves. Let me share some observations with you.

First observation: an enterprise decided to use a Service Control Policy (SCP) to only grant access to AWS services and features that had been approved by a security and governance board. It sounds like a good idea at the beginning. But the SCP causes significant problems:

• The engineers deploying their workloads to AWS do not know about the SCP and waste a lot of time to find out why configuring certain services, or features result in Access Denied errors.
• The engineers need to ask for permission for every service or feature they want to test or use for production. That causes a dependency on the security and governance board that slows down the engineers.
• The organization administrators have a hard time making any changes to the SCP. Each change to the SCP could potentially interrupt production workloads in all AWS accounts of the organization.

In general, it is a great idea to make sure engineers think twice about the security, compliance, and cost implications of using new AWS services or features. However, solving that problem with centralism causes more problems than it solves.

Second observation: to increase the security of all the web applications within the company, a network security specialist has been instructed to roll out Web Application Firewall (WAF) rules for all public endpoints. The security expert was granted access to the root account of the AWS organization to do so. It sounds like a good idea at the beginning. But the centrally managed WAF rules cause major problems:

• Because the network security specialist is missing important information about the web applications, the newly deployed WAF rules cause several service interruptions due to mistakenly blocked requests.
• On the one hand, the developers feel less responsible for securing their applications as they start relying on the centrally managed WAF. On the other hand, the network security specialist does not have enough insight into the application to implement WAF rules with high quality. As a result, the overall security level declines.
• The developers have to ask for exceptions and changes to the WAF rules to make sure their applications are working correctly. Often, developers are cannot deploy changes to their applications as they have to wait for the network security specialist to implement changes to the WAF rules.

Third observation: an enterprise owns hundreds of AWS accounts and added those accounts to a single AWS organization. As a side effect, all employees that have been granted administrator access to the root account of the organization are also able to assume an IAM role that grants them administrator access to every AWS account belonging to the organization.

• In theory, a single person could delete all the data of your organization. Including the backups.
• In case an attacker takes over the computer of an organization administrator, the attacker has administrator access to all AWS accounts belonging to the organization. That would allow the attacker to steal sensitive data, intrusion into all systems, …

And that’s only three of many observations from my day-to-day consulting work with companies all over the world.

Decentralization for the win

Where do we go from here? First of all, use AWS accounts to isolate workloads and environments from each other. There is no other way to do so 100%.

But, resist the temptation to manage all your AWS accounts with a single AWS organization (aka. root account).

Instead, divide your AWS accounts into smaller units. I’m thinking about 10 to 50 AWS accounts per unit. Create an AWS organization (aka. root account) for each of those units. Also, use AWS Organizations but for consolidated billing and account creation only. Do not use Service Control Policies (SCP) or any of the deeper integrations (called Trusted Organization Access).

At first sight, you might argue that decentralization is expensive as you need to solve the same problems over and over again. I agree that decentralization adds extra work. However, I’m confident that it will pay off in the long run as you do avoid the obstacles described in my observations above.

Next, a few tips when going the decentralization path:

• Lead by example. Provide Infrastructure as Code templates that implement security best practices and are compliant with laws and regulations. Make sure you are collecting feedback and improve those templates continuously.
• Provide engineers with training and tools to detect security and reliability issues as well as to optimize cloud spending.
• Handing over control and responsibility for security to the smallest possible group - for example, a team of engineers. Make sure to support that team with everything they ask for to master this challenge.

Summary

Using multiple AWS accounts to isolate workloads has been a best practice, not only since AWS introduced consolidated billing in 2010. AWS made a huge step by introducing AWS Organizations in 2017 and has added more and more features on top of the formerly boundary of an AWS account. In my opinion, we have passed the sweet spot between centralism and isolated accounts. The possibilities powered by AWS Organizations ruin the concept of isolated accounts with limited blast radius.

I recommend, to manage no more than 50 AWS accounts per AWS organization. Use multiple AWS organizations instead. Also, think twice before using SCP or Trusted Organization Access, both features make centralism permanent. I haven’t seen a thriving, innovative, and centralized IT organization so far. Correct me if I’m wrong.