🆕 Video Course Out Now: Rapid Docker on AWS

6 unknown CloudFormation features you should know about

Michael Wittig – 15 Apr 2019 (updated 13 Aug 2020)

I was recently invited to a CloudFormation workshop with a group of early CloudFormation users. I soon realized that the group had a good understanding of the basics, so I started to introduce more advanced features.

6 CloudFormation features you should know about

Today, I would like to share with you six CloudFormation features that have inspired the workshop participants most.

Do you prefer listening to a podcast episode over reading a blog post? Here you go!

cfn-lint

Writing CloudFormation templates is error prone. We should use every tool available to make our lives easier. I use the following tools:

  1. yamllint Locally ensures that our yaml is correct
  2. cfn-lint Locally checks if the template makes sense
  3. aws cloudformation validate-template Sends the template to AWS for validation

yamllint and cfn-lint are integrated into my editor, and I execute aws cloudformation validate-template before I git commit.

Creation policy

A CloudFormation creation policy is helpful if you provision an EC2 instance, typically by using user data. By default, when CloudFormation creates and EC2 instance it will not wait for the operating system and application to be ready. With a creation policy, you can ask CloudFormation to wait for an external signal.

In the following snippet, you find a CreationPolicy that instructs CloudFormation to wait for a signal with a timeout of 10 minutes (PT10M). The /opt/aws/bin/cfn-signal command sends the signal to CloudFormation to indicate that the user data script finished successfully (--exit-code 0).

#[...]
Resources:
VirtualMachine:
Type: 'AWS::EC2::Instance'
CreationPolicy:
ResourceSignal:
Timeout: PT10M
Properties:
#[...]
UserData:
'Fn::Base64': !Sub |
#!/bin/bash -ex
# run some commands
/opt/aws/bin/cfn-signal --exit-code 0 --resource VirtualMachine --region ${AWS::Region} --stack ${AWS::StackName}

Keep in mind that the logical id VirtualMachine of the resource needs to be referenced in cfn-signal using --resource VirtualMachine.

You can even improve this snippet to let CloudFormation know if the user data script failed using #!/bin/bash -e, trap, and --exit-code 1:

#[...]
Resources:
VirtualMachine:
Type: 'AWS::EC2::Instance'
CreationPolicy:
ResourceSignal:
Timeout: PT10M
Properties:
UserData:
'Fn::Base64': !Sub |
#!/bin/bash -ex
trap '/opt/aws/bin/cfn-signal --exit-code 1 --resource VirtualMachine --region ${AWS::Region} --stack ${AWS::StackName}' ERR
# run some commands
/opt/aws/bin/cfn-signal --exit-code 0 --resource VirtualMachine --region ${AWS::Region} --stack ${AWS::StackName}

Check out the following links if you are interested in real-world examples:

  • EC2 instance: CreationPolicy and cfn-signal
  • Auto Scaling Group: CreationPolicy and cfn-signal (Keep in mind that the CreationPolicy is attached to the auto scaling group while the cfn-signal is added to the user data in the launch configuration)

Update policy

A CloudFormation update policy triggers a rolling update to make changes to an auto scaling group. A rolling update will execute your change in small batches (e.g., instance by instance).

The following CloudFormation snippet demonstrates a rolling update that waits for signals in the same way you learned about in the creation policy:

#[...]
Resources:
LaunchConfiguration:
Type: 'AWS::AutoScaling::LaunchConfiguration'
Properties:
# [...]
AutoScalingGroup:
Type: 'AWS::AutoScaling::AutoScalingGroup'
Properties:
# [...]
LaunchConfigurationName: !Ref LaunchConfiguration
UpdatePolicy:
AutoScalingRollingUpdate:
PauseTime: PT10M
WaitOnResourceSignals: true

Check out the following links if you are interested in real-world examples:

A update policy can do even more.

Deletion policy

A CloudFormation deletion policy can prevent serious data loss. If CloudFormation deletes a resource, you can instruct CloudFormation to perform a backup first. This works for the following resource types:

Cover of Amazon Web Services in Action

Level up, strengthen your AWS skills.

Our book Amazon Web Services in Action is a comprehensive introduction to computing, storing, and networking in the AWS cloud. You'll find clear, relevant coverage of all the essential AWS services, emphasizing best practices for security, high availability, and scalability. Get the first chapter for free!

  • AWS::RDS::DBInstance (enabled by default)
  • AWS::RDS::DBCluster (enabled by default)
  • AWS::EC2::Volume
  • AWS::ElastiCache::CacheCluster (only Redis)
  • AWS::ElastiCache::ReplicationGroup
  • AWS::Redshift::Cluster
  • AWS::Neptune::DBCluster

The following snippet shows a deletion policy:

#[...]
Resources:
Volume:
DeletionPolicy: Snapshot
Type: 'AWS::EC2::Volume'
Properties:
#[...]

Check out the following links if you are interested in real-world examples:

But what about other resource types? You can also instruct CloudFormation not to delete a resource at all. We use this to prevent deletions of KMS keys which can also cause data loss. Why is it dangerous to delete KMS keys? You can delete them even if they are in use. Sometimes it is now obvious that the key is still used in a snapshot. Once the key is deleted, the snapshot cannot be restored anymore.

#[...]
Resources:
Key:
DeletionPolicy: Retain
Type: 'AWS::KMS::Key'
Properties:
#[...]

Check out the following links if you are interested in real-world examples:

Update replace policy

The CloudFormation update replace policy is similar to the deletion policy. You can use it to prevent data loss. If you update an existing stack, CloudFormation figures out what resources need to be updated. For some attributes, an update requires replacement of the resource. Therefore, CloudFormation creates the new resource and when successful, deletes the old one. If this resource was your database, you are in troubles.

The following resource types can be backed up before they are replaced:

  • AWS::RDS::DBInstance
  • AWS::RDS::DBCluster
  • AWS::EC2::Volume
  • AWS::ElastiCache::CacheCluster (only Redis)
  • AWS::ElastiCache::ReplicationGroup
  • AWS::Redshift::Cluster
  • AWS::Neptune::DBCluster

The following snippet shows an update replace policy.

#[...]
Resources:
Volume:
UpdateReplacePolicy: Snapshot
Type: 'AWS::EC2::Volume'
Properties:
#[...]

Check out the following links if you are interested in real-life examples:

But what about other resource types? As with the deletion policy, you can also instruct CloudFormation to not delete a resource at all during a replacement.

#[...]
Resources:
Key:
UpdateReplacePolicy: Retain
Type: 'AWS::KMS::Key'
Properties:
#[...]

cfn-init

Writing shell scripts to configure and install en EC2 instance is error prone. Sometimes, you need a tool that is easier to use than a script but not as complex as Chef/Puppet/Ansible/…

cfn-init solves that problem. cfn-init is executed in the user data script and fetches configuration information from your CloudFormation stack. The configuration is attached as metadata to the resource and supports:

  • Execution of commands
  • Creation of files
  • Fetching of external sources (tar, tar+gzip, tar+bz2, and zip)
  • Creation of users and groups
  • Installation of packages (yum, rpm, gems)
  • Management of services

The following CloudFormation snippet shows how you can install the package awslogs.

#[...]
Resources:
VirtualMachine:
Type: 'AWS::EC2::Instance'
Metadata:
'AWS::CloudFormation::Init':
config:
packages:
yum:
awslogs: []
Properties:
#[...]
UserData:
'Fn::Base64': !Sub |
#!/bin/bash -ex
/opt/aws/bin/cfn-init -v --resource VirtualMachine --region ${AWS::Region} --stack ${AWS::StackName}

Check out the following links if you are interested in real-world examples:

The user data script only runs once. If you update the stack and change the metadata, nothing happens. cfn-hup is a way ro run cfn-init regularly to notice and apply updates to the metadata.

Summary

CloudFormation is a great service about which we have written more than 40 articles. There are many features to discover such as the numerous policies that you can attach to resources: CreationPolicy, DeletionPolicy, UpdatePolicy, and UpdateReplacePolicy. Besides that, the tooling around CloudFormation has also evolved in the past years. cfn-lint saves you time by spotting failures in your templates early.

The Reddit user coinclink mentioned another hidden feature that I use often: aws cloudformation package and aws cloudformation deploy.

Michael Wittig

Michael Wittig

I’m an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I’m writing this blog and all other projects together with my brother Andreas.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

You can contact me via Email, Twitter, and LinkedIn.

Briefcase icon
Hire me