πŸ‘‰ AWS Debug Games (Beta) - Prove your AWS expertise by solving tricky challenges.

πŸ‘‰ AWS Debug Games - Prove your AWS expertise.

CloudFormation cfn-init pitfall: Auto scaling and throttling error rate exceeded

Michael Wittig – 11 Aug 2022

cfn-init is a little helper to install and configure EC2 instances managed with CloudFormation. Lately, I was running into issues when starting a more significant amount of EC2 (let’s say 50) during an auto scaling event. This blog post will teach you why the error happens and how to avoid it.

Throttling error rate exceeded

Introducing cfn-init

cfn-init configuration is added as metadata to a resource using the AWS::CloudFormation::Init key. The following example configures cfn-init to

  • create/update the file /etc/sample.conf.
  • enable & start the service sample (also restarts the service if /etc/sample.conf is changed).

cfn-init is usually executed in the user data script.

VirtualMachine:
Type: 'AWS::EC2::Instance'
Metadata:
'AWS::CloudFormation::Init':
config:
files:
'/etc/sample.conf':
content: !Sub |
[main]
region=${AWS::Region}
mode: '000400'
owner: root
group: root
services:
sysvinit:
sample:
enabled: true
ensureRunning: true
files:
- '/etc/sample.conf'
Properties:
# [...]
UserData:
'Fn::Base64': !Sub |
#!/bin/bash -ex
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource VirtualMachine --region ${AWS::Region}

The pitfall

The way cfn-init is implemented is this:


Looking for a new challenge?

  • DEMICON

    Cloud Operations Lead

    DEMICON • AWS Advanced Consulting Partner • Remote (Europe)
    service-delivery-management hiring devops platform

  1. Call the CloudFormation API DescribeStackResource to read the metadata.
  2. Validate and parse the metadata.
  3. Apply the configuration to the EC2 instance.

Unfortunately, the CloudFormation API has notorious low API rate limits, and cfn-init does not retry in the case of a rate exceeded error. Therefore, when many EC2 instances run cfn-init more or less at the same time, you will see the following error:

2022-06-02 07:13:14,838 [DEBUG] Response: 400 https://cloudformation.us-east-1.amazonaws.com/?Action=DescribeStackResource&LogicalResourceId=ScanLaunchTemplate&ContentType=JSON&StackName=***&Version=2010-05-15 [headers: {'x-amzn-RequestId': '***', 'Content-Type': 'application/json', 'Content-Length': '124', 'Date': 'Thu, 02 Jun 2022 07:13:14 GMT', 'Connection': 'close'}]
2022-06-02 07:13:14,838 [DEBUG] Response error: b'{""Error"":{""Code"":""Throttling"",""Message"":""Rate exceeded"",""Type"":""Sender""},""RequestId"":""***""}'

Solving the issue

How can we solve the issue?

  1. Do not use cfn-init at all.
  2. Load the metadata from a file.

To load the metadata from a file and not the CloudFormation API, create a file (e.g., metadata.json) like this:

{
"AWS::CloudFormation::Init": {
"config": {
"files": {
"/etc/sample.conf": {
"content": "[main]\nregion=eu-west-1\n",
"mode": "000400",
"owner": "root",
"group": "root"
}
},
"services": {
"sysvinit": {
"sample": {
"enabled": true,
"ensureRunning": true,
"files": [
"/etc/sample.conf"
]
}
}
}
}
}
}

And invoke cfn-init like this:

/opt/aws/bin/cfn-init -v metadata.json

I hope this article will help you avoid the pitfall.

Become a cloudonaut supporter

Michael Wittig

Michael Wittig ( Email Twitter LinkedIn Mastodon )

We launched the cloudonaut blog in 2015. Since then, we have published 365 articles, 68 podcast episodes, and 68 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, Christopher Hipwell, Jason Yorty, Jeff Finley, jhoadley, Johannes Konings, John Culkin, Jonathan Deamer, Juraj Martinka, Ken Snyder, Markus Ellers, Oriol Rodriguez, Ross Mohan, sam onaga, Satyendra Sharma, Simon Devlin, Todd Valentine, Victor Grenu, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.