🎉 We are launching a new weekly show: Hot off the Cloud

🎉 We are launching a new weekly show

EBS Snapshot Pitfalls: Does your backup withstand reality?

Michael Wittig – 11 Nov 2021

Does your disaster recovery plan deliver what it promises? Here are three reasons why your plan won’t stand up to reality. Learn about common pitfalls when backing up EC2 instances with the help of EBS snapshots.

EBS Snapshot Pitfalls

A crash-consistent snapshot leads to data corruption

AWS describes EBS snapshots as crash-consistent. What does that mean? Imagine that a machine suddenly breaks. In what condition is the data on the hard disk? We don’t know whether the application and operating system wrote a consistent state to disk before the interruption. In other words: a crash-consistent snapshot is worthless for disaster recovery. You might restore corrupt or inconsistent data.

EBS Snapshot Pitfalls #1

Why is that? All EBS cares about are data blocks; it reads and writes ones and zeros. The operating system running on EC2 is responsible for persisting data on behalf of the application.

Be aware that even official tools like AWS Backup do not offer a solution to this problem - not to mention the countless third-party providers.

Solution: Before creating a snapshot, tell the application and operating system to write a consistent state to disk. For example on Linux, use AWS Systems Manager Automation to halt the application and ask the operating system to flush caches before creating an EBS snapshot. Running on Windows? Check out Create a VSS application-consistent snapshot.

A restored EBS volume requires initialization

Restoring a volume based on an EBS snapshot typically takes a few seconds only. However, the data is not available from the beginning. Instead, EBS restores the data asynchronously. You will notice high latencies during that period.

EBS Snapshot Pitfalls #2

Free Monitoring Checklist + Mind Map

Find the blind spots in your AWS monitoring!

Setting up monitoring on AWS is hard. AWS provides countless features and sources of events. Overlooking the important settings is easy. Our prioritized checklist includes all parts of a basic monitoring setup for AWS. Additionally, use our mind map to map your monitoring goals to AWS services. Download Free Monitoring Checklist + Mind Map!

If latency matters to your system, you should initialize a restored volume before ramping up traffic. To do so, make sure to read all blocks from the volume once. The following command does the trick on Linux. See Initialize Amazon EBS volumes for more detailed information.

fio --filename=/dev/xvda --rw=read --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=ebs-restore

Depending on the volume size, the volume type, and the EC2 instance type, initializing the volume might take a while. For example, it will take about 23 minutes to initialize an EBS volume of type gp3 with 500 GB connected to an EC2 instance of type m5a.large. So make sure to add the initialization phase when planning the Recovery Time Objective (RTO). The bottleneck is the maximum throughput from EC2 instance to EBS volume of 360 MB/s. See Amazon EBS–optimized instances for more details.

It should also be mentioned that AWS offers a feature called Amazon EBS fast snapshot restore. By enabling fast restore, it is no longer necessary to initialize a restored volume as described above. The volume is ready for latency-critical workloads from the start. However, the pricing for fast snapshot restore make clear that this feature is not intended for this use case: $500 per snapshot and availability zone.

Solution: Consider the time it takes to initialize a recovered volume when planning for disaster recovery. While creating an EBS volume based on a snapshot usually takes a few seconds only, it is necessary to read all blocks of a restored volume to ensure low latency throughput.

A SLA is missing for restoring EBS volumes

When someone else is operating our infrastructure, we need to rely on SLAs to ensure the provider complies with our requirements. Therefore, most AWS services come with a well-defined SLA.

Unfortunately, the SLA for EC2 and EBS is very unspecific. AWS does not specify any objectives when it comes to restoring EBS snapshots. Therefore, it isn’t easy to evaluate to what extent the system can be relied upon. For example, what will happen during a significant outage when many customers decide to restore machines from snapshots to recover machines in another availability zone or region?

Solution: There is no technical solution to this problem. Talk to your AWS representative about this and ask them to be more specific about their SLA.

Summary

To ensure that you can recover all data in an emergency, there are a few stumbling blocks to avoid.

  1. By default creating an EBS snapshot results in a crash-consistent backup. Restoring the snapshot might lead to corrupt or inconsistent data. Make sure to halt the application and flush caches before creating a snapshot to avoid that.
  2. EBS restores data from snapshots asynchronously. Therefore, you should initialize the volume by reading all blocks before ramping up your workload.
  3. Unfortunately, AWS does not publish any information about the extent to which we can rely on the recovery process of EBS snapshots. Especially if you want to plan for significant outages on AWS, this is very unsatisfactory. AWS needs to improve here.

By the way, are you interested in an example on how to use AWS Systems Manager Automation to create application-consistent snapshots on Linux? Please let me know! I’m considering to write a blog post about that.

Become a cloudonaut supporter

Michael Wittig

Michael Wittig ( Email, Twitter, or LinkedIn )

We launched the cloudonaut blog in 2015. Since then, we have published 360 articles, 49 podcast episodes, and 48 videos. It's all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.

Please support us

Have you learned something new by reading, listening, or watching our content? With your help, we can spend enough time to keep publishing great content in the future. Learn more

$
Amount must be a multriply of 5. E.g, 5, 10, 15.

Thanks to Alan Leech, Alex DeBrie, ANTHONY RAITI, Christopher Hipwell, Jaap-Jan Frans, Jason Yorty, Jeff Finley, Jens Gehring, jhoadley, Johannes Grumböck, Johannes Konings, John Culkin, Jonas Mellquist, Juraj Martinka, Kamil Oboril, Ken Snyder, Markus Ellers, Ross Mohan, Ross Mohan, sam onaga, Satyendra Sharma, Shawn Tolidano, Simon Devlin, Thorsten Hoeger, Todd Valentine, Victor Grenu, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt.