Optimizing Amazon Linux 2023 for tiny EC2 instances: t3.nano, t3a.nano, or t4g.nano

I recently observed severe performance degradation when porting our S3 antivirus solution, bucketAV, from Amazon Linux 2 to Amazon Linux 2023 on tiny instance types such as t3.nano, t3a.nano, or t4g.nano. It took minutes to run commands that usually complete in a second. In this blog post, I share two tricks to get back to performance levels provided by Amazon Linux 2.

Running workloads on t3.nano, t3a.nano, or t4g.nano is a challenge. The 512 MB of RAM are mostly used entirely by the operating system Amazon Linux 2023. As soon as you (or your application) try to allocate memory, things get very slow. There are two tricks to address this problem by moving stuff from RAM to disk:

1. Move /tmp from RAM to disk.
2. Move SWAP from RAM to disk.

Move /tmp from RAM to disk

/tmp is typically where files that a program doesn’t need in the long-term go. In Amazon Linux 2023, /tmp is backed by tmpfs which stored the file in memory.

The problem: Memory is scarce on t3.nano.

To return to /tmp stored on disk, run:

sudo systemctl mask tmp.mount 
sudo systemctl stop tmp.mount

Move SWAP from RAM to disk

If the OS is running out of memory, it tries to move memory from RAM to another medium, aka swap. Often, swap is backed by disk storage, which is slow but works. In Amazon Linux 2023, swap is provided by zram on instances with less than 1 GB of memory. Zram creates a compressed block device in RAM for swap.

Two problems:

1. Compression/decompression is CPU-bound, and you don’t have a lot of CPU power on a nano instance.
2. RAM is already scarce. There simply is not a lot of RAM available.

To disable zram and return to swap backed by disk, run:

sudo dnf -y remove zram-generator-defaults zram-generator

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
cat <<"EOF" | sudo tee -a /etc/fstab > /dev/null
/swapfile none swap defaults 0 0
EOF

Summary

The most scarce resource on EC2 instance types like t3.nano, t3a.nano, or t4g.nano is memory. Unfortunately, Amazon Linux 2023 is much more memory-hungry than Amazon Linux 2. Therefore, you might need to tweak AL2023 to continue running your applications. The most important trick is to move swap away from zram to disk.