6 unknown CloudFormation features you should know about

I was recently invited to a CloudFormation workshop with a group of early CloudFormation users. I soon realized that the group had a good understanding of the basics, so I started to introduce more advanced features.

Today, I would like to share with you six CloudFormation features that have inspired the workshop participants most.

Do you prefer listening to a podcast episode over reading a blog post? Here you go!

cfn-lint

Writing CloudFormation templates is error prone. We should use every tool available to make our lives easier. I use the following tools:

1. yamllint Locally ensures that our yaml is correct
2. cfn-lint Locally checks if the template makes sense
3. aws cloudformation validate-template Sends the template to AWS for validation

yamllint and cfn-lint are integrated into my editor, and I execute aws cloudformation validate-template before I git commit.

Creation policy

A CloudFormation creation policy is helpful if you provision an EC2 instance, typically by using user data. By default, when CloudFormation creates and EC2 instance it will not wait for the operating system and application to be ready. With a creation policy, you can ask CloudFormation to wait for an external signal.

In the following snippet, you find a CreationPolicy that instructs CloudFormation to wait for a signal with a timeout of 10 minutes (PT10M). The /opt/aws/bin/cfn-signal command sends the signal to CloudFormation to indicate that the user data script finished successfully (--exit-code 0).

#[...]
Resources:
  VirtualMachine:
    Type: 'AWS::EC2::Instance'
    CreationPolicy:
      ResourceSignal:
        Timeout: PT10M
    Properties:
      #[...] 
      UserData:
        'Fn::Base64': !Sub |
          #!/bin/bash -ex
          # run some commands
          /opt/aws/bin/cfn-signal --exit-code 0 --resource VirtualMachine --region ${AWS::Region} --stack ${AWS::StackName}

Keep in mind that the logical id VirtualMachine of the resource needs to be referenced in cfn-signal using --resource VirtualMachine.

You can even improve this snippet to let CloudFormation know if the user data script failed using #!/bin/bash -e, trap, and --exit-code 1:

#[...]
Resources:
  VirtualMachine:
    Type: 'AWS::EC2::Instance'
    CreationPolicy:
      ResourceSignal:
        Timeout: PT10M
    Properties:
      UserData:
        'Fn::Base64': !Sub |
          #!/bin/bash -ex
          trap '/opt/aws/bin/cfn-signal --exit-code 1 --resource VirtualMachine --region ${AWS::Region} --stack ${AWS::StackName}' ERR
          # run some commands
          /opt/aws/bin/cfn-signal --exit-code 0 --resource VirtualMachine --region ${AWS::Region} --stack ${AWS::StackName}

Check out the following links if you are interested in real-world examples:

• EC2 instance: CreationPolicy and cfn-signal
• Auto Scaling Group: CreationPolicy and cfn-signal (Keep in mind that the CreationPolicy is attached to the auto scaling group while the cfn-signal is added to the user data in the launch configuration)

Update policy

A CloudFormation update policy triggers a rolling update to make changes to an auto scaling group. A rolling update will execute your change in small batches (e.g., instance by instance).

The following CloudFormation snippet demonstrates a rolling update that waits for signals in the same way you learned about in the creation policy:

#[...]
Resources:
  LaunchConfiguration:
    Type: 'AWS::AutoScaling::LaunchConfiguration'
    Properties:
      # [...]
  AutoScalingGroup:
    Type: 'AWS::AutoScaling::AutoScalingGroup'
    Properties:
      # [...]
      LaunchConfigurationName: !Ref LaunchConfiguration
    UpdatePolicy:
      AutoScalingRollingUpdate:
        PauseTime: PT10M
        WaitOnResourceSignals: true

Check out the following links if you are interested in real-world examples:

• Auto Scaling Group

A update policy can do even more.

Deletion policy

A CloudFormation deletion policy can prevent serious data loss. If CloudFormation deletes a resource, you can instruct CloudFormation to perform a backup first. This works for the following resource types:

• AWS::RDS::DBInstance (enabled by default)
• AWS::RDS::DBCluster (enabled by default)
• AWS::EC2::Volume
• AWS::ElastiCache::CacheCluster (only Redis)
• AWS::ElastiCache::ReplicationGroup
• AWS::Redshift::Cluster
• AWS::Neptune::DBCluster

The following snippet shows a deletion policy:

#[...]
Resources:
  Volume:
    DeletionPolicy: Snapshot
    Type: 'AWS::EC2::Volume'
    Properties:
      #[...]

Check out the following links if you are interested in real-world examples:

• EBS volume
• ElastiCache for Redis

But what about other resource types? You can also instruct CloudFormation not to delete a resource at all. We use this to prevent deletions of KMS keys which can also cause data loss. Why is it dangerous to delete KMS keys? You can delete them even if they are in use. Sometimes it is now obvious that the key is still used in a snapshot. Once the key is deleted, the snapshot cannot be restored anymore.

#[...]
Resources:
  Key:
    DeletionPolicy: Retain
    Type: 'AWS::KMS::Key'
    Properties:
      #[...]

Check out the following links if you are interested in real-world examples:

• KMS key

Update replace policy

The CloudFormation update replace policy is similar to the deletion policy. You can use it to prevent data loss. If you update an existing stack, CloudFormation figures out what resources need to be updated. For some attributes, an update requires replacement of the resource. Therefore, CloudFormation creates the new resource and when successful, deletes the old one. If this resource was your database, you are in troubles.

The following resource types can be backed up before they are replaced:

• AWS::RDS::DBInstance
• AWS::RDS::DBCluster
• AWS::EC2::Volume
• AWS::ElastiCache::CacheCluster (only Redis)
• AWS::ElastiCache::ReplicationGroup
• AWS::Redshift::Cluster
• AWS::Neptune::DBCluster

The following snippet shows an update replace policy.

#[...]
Resources:
  Volume:
    UpdateReplacePolicy: Snapshot
    Type: 'AWS::EC2::Volume'
    Properties:
      #[...]

Check out the following links if you are interested in real-life examples:

• RDS instance
• Aurora cluster
• EBS volume
• ElastiCache for Redis

But what about other resource types? As with the deletion policy, you can also instruct CloudFormation to not delete a resource at all during a replacement.

#[...]
Resources:
  Key:
    UpdateReplacePolicy: Retain
    Type: 'AWS::KMS::Key'
    Properties:
      #[...]

cfn-init

Writing shell scripts to configure and install en EC2 instance is error prone. Sometimes, you need a tool that is easier to use than a script but not as complex as Chef/Puppet/Ansible/…

cfn-init solves that problem. cfn-init is executed in the user data script and fetches configuration information from your CloudFormation stack. The configuration is attached as metadata to the resource and supports:

• Execution of commands
• Creation of files
• Fetching of external sources (tar, tar+gzip, tar+bz2, and zip)
• Creation of users and groups
• Installation of packages (yum, rpm, gems)
• Management of services

The following CloudFormation snippet shows how you can install the package awslogs.

#[...]
Resources:
  VirtualMachine:
    Type: 'AWS::EC2::Instance'
    Metadata:
      'AWS::CloudFormation::Init':
        config:
          packages:
            yum:
              awslogs: []
    Properties:
      #[...] 
      UserData:
        'Fn::Base64': !Sub |
          #!/bin/bash -ex
          /opt/aws/bin/cfn-init -v --resource VirtualMachine --region ${AWS::Region} --stack ${AWS::StackName}

Check out the following links if you are interested in real-world examples:

• Metadata and cfn-init

The user data script only runs once. If you update the stack and change the metadata, nothing happens. cfn-hup is a way ro run cfn-init regularly to notice and apply updates to the metadata.

Summary

CloudFormation is a great service about which we have written more than 40 articles. There are many features to discover such as the numerous policies that you can attach to resources: CreationPolicy, DeletionPolicy, UpdatePolicy, and UpdateReplacePolicy. Besides that, the tooling around CloudFormation has also evolved in the past years. cfn-lint saves you time by spotting failures in your templates early.

The Reddit user coinclink mentioned another hidden feature that I use often: aws cloudformation package and aws cloudformation deploy.