🎉 We are launching a new weekly show: Hot off the Cloud

🎉 We are launching a new weekly show

CloudFormation cfn-init pitfall: Auto scaling and throttling error rate exceeded

Michael Wittig – 11 Aug 2022

cfn-init is a little helper to install and configure EC2 instances managed with CloudFormation. Lately, I was running into issues when starting a more significant amount of EC2 (let’s say 50) during an auto scaling event. This blog post will teach you why the error happens and how to avoid it.

Throttling error rate exceeded

Introducing cfn-init

cfn-init configuration is added as metadata to a resource using the AWS::CloudFormation::Init key. The following example configures cfn-init to

  • create/update the file /etc/sample.conf.
  • enable & start the service sample (also restarts the service if /etc/sample.conf is changed).

cfn-init is usually executed in the user data script.

VirtualMachine:
Type: 'AWS::EC2::Instance'
Metadata:
'AWS::CloudFormation::Init':
config:
files:
'/etc/sample.conf':
content: !Sub |
[main]
region=${AWS::Region}
mode: '000400'
owner: root
group: root
services:
sysvinit:
sample:
enabled: true
ensureRunning: true
files:
- '/etc/sample.conf'
Properties:
# [...]
UserData:
'Fn::Base64': !Sub |
#!/bin/bash -ex
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource VirtualMachine --region ${AWS::Region}

The pitfall

The way cfn-init is implemented is this:


Looking for a new challenge?

  • tecRacer

    Cloud Consultant • AWS Migrations

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Portugal, and Switzerland
    Assessment Transformation Change Management
  • DEMICON

    Senior Lead Full Stack Developer

    DEMICON • AWS Advanced Consulting Partner • Remote
    AWS JavaScript/TypeScript Angular React

  1. Call the CloudFormation API DescribeStackResource to read the metadata.
  2. Validate and parse the metadata.
  3. Apply the configuration to the EC2 instance.

Unfortunately, the CloudFormation API has notorious low API rate limits, and cfn-init does not retry in the case of a rate exceeded error. Therefore, when many EC2 instances run cfn-init more or less at the same time, you will see the following error:

2022-06-02 07:13:14,838 [DEBUG] Response: 400 https://cloudformation.us-east-1.amazonaws.com/?Action=DescribeStackResource&LogicalResourceId=ScanLaunchTemplate&ContentType=JSON&StackName=***&Version=2010-05-15 [headers: {'x-amzn-RequestId': '***', 'Content-Type': 'application/json', 'Content-Length': '124', 'Date': 'Thu, 02 Jun 2022 07:13:14 GMT', 'Connection': 'close'}]
2022-06-02 07:13:14,838 [DEBUG] Response error: b'{""Error"":{""Code"":""Throttling"",""Message"":""Rate exceeded"",""Type"":""Sender""},""RequestId"":""***""}'

Solving the issue

How can we solve the issue?

  1. Do not use cfn-init at all.
  2. Load the metadata from a file.

To load the metadata from a file and not the CloudFormation API, create a file (e.g., metadata.json) like this:

{
"AWS::CloudFormation::Init": {
"config": {
"files": {
"/etc/sample.conf": {
"content": "[main]\nregion=eu-west-1\n",
"mode": "000400",
"owner": "root",
"group": "root"
}
},
"services": {
"sysvinit": {
"sample": {
"enabled": true,
"ensureRunning": true,
"files": [
"/etc/sample.conf"
]
}
}
}
}
}
}

And invoke cfn-init like this:

/opt/aws/bin/cfn-init -v metadata.json

I hope this article will help you avoid the pitfall.

Become a cloudonaut supporter

Michael Wittig

Michael Wittig ( Email, Twitter, or LinkedIn )

We launched the cloudonaut blog in 2015. Since then, we have published 360 articles, 49 podcast episodes, and 48 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Christopher Hipwell, Jaap-Jan Frans, Jason Yorty, Jeff Finley, Jens Gehring, jhoadley, Johannes Grumböck, Johannes Konings, John Culkin, Jonas Mellquist, Juraj Martinka, Kamil Oboril, Ken Snyder, Markus Ellers, Ross Mohan, Ross Mohan, sam onaga, Satyendra Sharma, Shawn Tolidano, Simon Devlin, Thorsten Hoeger, Todd Valentine, Victor Grenu, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.