Prevent CloudFormation Change Sets from piling up

Andreas Wittig – 10 Oct 2019

Recently, I’ve stumbled upon a problem when using aws cloudformation deploy within deployment pipelines (Jenkins, GitLab CI, …) that I wanted to share with you.

Prevent CloudFormation Change Sets from piling up

Usually, I’m using the AWS CLI to deploy CloudFormation stacks.

aws cloudformation package --template-file example.yml --output-template-file output.yml --s3-bucket example
aws cloudformation deploy --template-file output.yml --stack-name example --no-fail-on-empty-changeset

The aws cloudformation package command packages the template example.yml and uploads dependencies like nested stack templates to S3.

The aws cloudformation deploy command creates or updates the stack example. The option --no-fail-on-empty-changeset makes sure the command does not throw an error in case the template has not changed, which is very likely when using a deployment pipeline.

It is essential to know that the pipeline creates a change set whenever executing aws cloudformation deploy. Deploying a stack without making any changes to the template or the parameters leads to a failed change set: The submitted information didn't contain changes. Submit different information to create a change set.

Looking for a new challenge?

  • tecRacer

    Cloud Consultant • AWS Migrations

    tecRacer • Premier AWS Consulting Partner • Germany, Austria, Portugal, and Switzerland
    Assessment Transformation Change Management

    Senior Lead Full Stack Developer

    DEMICON • AWS Advanced Consulting Partner • Remote
    AWS JavaScript/TypeScript Angular React

Unfortunately, the aws cloudformation deploy command does not clean up those failed change sets. This leads to change sets piling up. And one day or another your deployment pipeline will fail with the following error message:

An error occurred (LimitExceededException) when calling the CreateChangeSet operation: ChangeSet limit exceeded for stack ...

You have reached the maximum number of change sets for a stack.

How to fix that? It would be great if aws cloudformation deploy would clean up failed change sets automatically. Keep an eye on #4534 for progress on that.

For now, adding the following bash script to your deployment pipeline might help. The cleanup() function list all stacks starting with prefix, fetches the change sets for each stack, and deletes change sets in status FAILED.

# cleanup (region, prefix)
cleanup () {
stacks=$(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE --query "StackSummaries[?starts_with(StackName, \`$2\`) == \`true\`].StackName" --output text --region $1)
for stack in $stacks
echo "${stack}: cleaning up change sets"
changesets=$(aws cloudformation list-change-sets --stack-name $stack --query 'Summaries[?Status==`FAILED`].ChangeSetId' --output text --region $1)
for changeset in $changesets
echo "${stack}: deleting change set ${changeset}"
aws cloudformation delete-change-set --change-set-name ${changeset} --region $1

The following command executes the function cleanup() for region eu-west-1 and stack name prefix example-.

cleanup eu-west-1 example-

In summary, aws cloudformation deploy leaves behind waste that you need to deal with.

Become a cloudonaut supporter

Andreas Wittig

Andreas Wittig ( Email, Twitter, or LinkedIn )

We launched the cloudonaut blog in 2015. Since then, we have published 360 articles, 50 podcast episodes, and 48 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Christopher Hipwell, Jaap-Jan Frans, Jason Yorty, Jeff Finley, Jens Gehring, jhoadley, Johannes Grumböck, Johannes Konings, John Culkin, Jonas Mellquist, Juraj Martinka, Kamil Oboril, Ken Snyder, Markus Ellers, Ross Mohan, Ross Mohan, sam onaga, Satyendra Sharma, Shawn Tolidano, Simon Devlin, Thorsten Hoeger, Todd Valentine, Victor Grenu, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.