ECS Deployment Options - From rolling updates to blue-green and canary

Philipp Garbe – 14 Jul 2020

“How often do you deploy to production?” - This is an important question as the best application is useless if you can’t deploy it. And being able to deploy regularly and automated is quite important.
The Elastic Container Service (ECS) is used by many AWS customers and offers different ways to deploy your containers. In this article, I’ll explain the different deployment options of ECS, show how they work, and when to use them.

ECS Deployment Options

Do you prefer listening to a podcast episode over reading a blog post? Here you go!

The different options can be set in the DeploymentController property of an ECS service. Let’s have a look:

ECS

The first option is ECS itself. The ECS service scheduler is responsible for performing a rolling update of newer versions. The current version of running tasks is replaced by newer versions. The optional DeploymentConfiguration parameter defines how many tasks should run during the deployment. MinimumHealthyPercent indicates the percentage of tasks that must remain in RUNNING status, and MaximumHealthyPercent parameter represents an upper limit on the number of your service’s tasks that are allowed in the RUNNING or PENDING state during a deployment. Both values define a percentage of the current desiredCount.

The configuration is easy and straight forward but also limits the possibilities. It’s not possible to cancel or rollback a deployment. And due to the limited configuration, it does not allow us to do a blue/green or canary deployment.

Another issue comes in combination with CloudFormation (see Deep dive on load balanced ECS Service deployments with CloudFormation for more details):

  • A CloudFormation stack creation will always complete successfully even with failing health checks and independent of any deploymentConfiguration
  • A CloudFormation stack update will fail only if minimumHealthyPercent is 100%, and the container health check is unhealthy (no other combination). And it takes CloudFormation 3 hours before it triggers the rollback.
  • CloudFormation rollback means that it triggers a new ECS deployment with the former taskdefinition (which could lead to some troubles as well)

Code Deploy

The next option is CODE_DEPLOY, where the deployment of an ECS service is orchestrated by CodeDeploy. During deployment, it creates tasks of the new version in parallel to existing tasks and then shifts the traffic over. The traffic can be shifted all-at-once, linear, or as canary (small percentage at the beginning and then the rest).

Hooks (basically lambda functions) that are executed before and after the traffic is shifted can ensure that the application works as expected. Failures in the hooks lead to automated rollbacks. In addition, CloudWatch alarms can be set that are watched during the deployment, and if they go on a rollback is triggered. The deployment is also “baked” for some time to allow a fast rollback in case something went wrong after the new version has been rolled out completely.

See also Clare Liguori‘s twitter thread, where she explains it in detail.

To set up ECS deployments with CodeDeploy a certain order must be followed:

  1. Create your infrastructure (like IAM roles, security groups, and load balancer)
  2. Set up an ECS Task Definition and Service (and optional a Cluster)
  3. Create a CodeDeploy Application and DeploymentGroup (referencing ECS service and cluster and two target groups)
  4. Set up a CodePipeline which builds the image, creates a taskdefinition.json and appsepc.yaml and calls CodeDeploy

Unfortunately, not everything can be configured in CloudFormation, as DeploymentGroup does not support ECS blue/green deployments.

Cover of Rapid Docker on AWS

Become a Docker on AWS professional!

Our book Rapid Docker on AWS is designed for DevOps engineers and web developers who want to run dockerized web applications on AWS. We lead you with many examples: From dockerizing your application to Continuous Deployment and Infrastructure as Code on AWS. No prior knowledge of Docker and AWS is required. Get the first chapter for free!

Another issue is that the CodeDeploy action needs not only an appspec.yaml but also a taskdefinition.json that contains our task definition (Although AWS states in their docs that TaskDefinition is optional for an ECS Service, I couldn’t deploy a service without a referenced task definition). With those dependencies, it’s not possible to create a pipeline first and deploy your application afterward. It must be the other way around. First, create your infrastructure and then your pipeline. Further updates to your task definition are deployed with CodeDeploy and not CloudFormation.

One advantage of CDK is that it can fill the gaps of CloudFormation and abstract complicated things away. Although blue/green or canary deployments for ECS in CodeDeploy are not part of the official CDK library, there is a community project (cloudcomponents/cdk-constructs) that does the heavy lifting for you.

External Deployments

The last option is EXTERNAL. External deployments allow anyone to perform deployments by using TaskSets. A TaskSet is a set of ECS tasks that are part of a deployment. They are also used internally by CodeDeploy.

There is an example for Jenkins that performs the following steps:

  1. Create a green task set with the new task definition
  2. Shift test traffic to the green task set
  3. Shift an increment of prod traffic to the green task set
  4. Shift all prod traffic to the green task set
  5. Remove blue task set with the old task definition

But also, CloudFormation supports blue/green deployments for Amazon ECS, which is good as you can define your whole infrastructure as code. But how is it different from CodeDeploy, and how does that work?

CloudFormation also uses the external deployment option. To perform the steps mentioned above, CloudFormation introduced a new transformation (“AWS::CodeDeployBlueGreen”) and a new “hooks” section (and an “AWS::CodeDeploy::BlueGreen” hook). Some resources need to be created for blue and green in CloudFormation (e.g., load balancer target group), and for some resources, only the “blue” part needs to be created (e.g., task definition or task set). During a stack creation or update, CloudFormation does multiple transformations of the template to perform the steps above.

I can’t link to any docs as they don’t exist yet. Just have a look at the example. And Clare Liguori‘s twitter thread explains it in more detail.

But the CloudFormation solution comes with a lot of limitations. Just a few:

  • SSM parameters can’t be resolved
  • Importing existing resources is not supported
  • Doesn’t work in combination with nested stacks
  • No visibility or management of in-progress deployments like in CodeDeploy.

Conclusion

The one who has the choice is in agony. Every option has it’s advantages but also drawbacks. Unfortunately, there’s no simple answer to what option you should use. I hope I could shed some light on it and help you with your decision.

Criteria ECS CodeDeploy External (CloudFormation)
Simple configuration ⚠️
CloudFormation support ⚠️
Blue/Green deployments
Canary deployments
Automatic rollback ⚠️
Transparency

Despite the variety, I don’t believe that this is what the customers want. Too many cooks spoil the broth: ECS, CodeDeploy, and CloudFormation offer you different ways to deploy an ECS service. But what I’d like to have is the simplicity of ECS rolling updates, the powerful configuration of CodeDeploy, and the infrastructure as code support of the CloudFormation solution.

PS: And a nice integration in CDK :)

Tags: aws ecs

Philipp Garbe

Living in Bavaria. Working in the Cloud. AWS Container Hero.

You can contact me via Email, Twitter, and LinkedIn.