Migrating to Amazon Linux 2
I run all my EC2 workloads on Amazon Linux. It comes with a superb AWS integration, a secure default configuration, regular security updates, and I can open AWS Support tickets if I run into any problems.
In late December 2017, AWS announced the successor of Amazon Linux: Amazon Linux 2.
AWS also announced that Amazon Linux 2018.03 is the last release for the current generation of Amazon Linux and will be supported until June 30, 2020. Therefore, you have to come up with a migration plan.
Amazon Linux 2 comes with the same benefits as Amazon Linux, but it adds some new capabilities:
- long-term support: Amazon Linux 2 supports each LTS release for five years
- on-premises support: virtual machine images for on-premises development and testing are available
- systemd: replacing SystemVinit
- extras library: provides up-to-date versions of software bundles such as nginx
Let’s dive into some of the changes in more detail. At the end of the post, I will also outline some pitfalls I encountered when migrating our Free Templates for AWS CloudFormation to Amazon Linux 2.
Further reading: Release Notes, FAQs, AWS Blog Post, Announcement
The Amazon Linux delivers a continuous flow of updates that allow you to roll from one version of the Amazon Linux AMI to the most recent. A
yum update always moves your system to the latest Amazon Linux version. There were no versions of Amazon Linux available, only snapshots.
Amazon Linux 2 changes this. You will have Amazon Linux 2 versions that are supplied with updates for five years. Once a new Amazon Linux 2 LTS release becomes available, no breaking changes will be introduced by AWS for this release.
Amazon Linux uses SysVinit to bootstrap the Linux user space and to manage system processes after booting. This procedure is usually called init. One of the major drawbacks of
SysVinit is that it starts tasks serially, waiting for each to finish loading before moving on to the next. This can result in long delays during boot.
Amazon Linux 2 uses systemd as the init system.
systemd executes elements of its startup sequence in parallel, which is faster than the traditional serial approach from
systemd can also ensure that a service is running (e.g., it restarts a service if it crashed).
systemd is not just the name of the init system daemon but also refers to the entire software bundle around it, which includes:
journald: responsible for event logging (replaces syslog)
udevd: device manager for the Linux kernel, which handles the /dev directory and all user space actions when adding/removing devices
logind: manages user logins and seats in various ways.
I will not cover
logind in this post. You should not get in touch with them as a normal user like me. Keep in mind that networking configuration is not controlled by
networkd (also part of
systemd software bundle). Instead, networking configuration is controlled by cloud-init which is triggered by
systemd several times during boot.
cloud-init handles early initialization of an EC2 instance (also works with other vendors).
Further reading: systemd man page
Reading logs from journald
To read all system logs (journal in journald terminology), starting with the oldest entry, run
journalctl. The output is paged through
less by default. Which means you can scroll down / up an entry with the
UP arrow keys, or scroll a full page down/up with the
b keys. Press the
q key to quit. To reverse the order, run
To show only the most recent journal entries, and continuously print new entries, run
journalctl -f (like a
There are many ways to filter the output. Based on priority, run
journalctl -p err to get levels alert, crit, and err (using syslog log levels). Based on the unit, run
journalctl -u sshd to get all entries for
sshd. Check the further reading links for more information.
Keep in mind that some applications still write logs to
/var/log. Journald also forwards logs to
rsyslog which is configured (
/etc/rsyslog.conf) to write some of them to files in
Further reading: journalctl man page
Controlling systemd services
To start a service (unit in systemd terminology), you run:
systemctl start awslogsd.service
To make sure a service (unit in systemd terminology) is started during boot/reboot, you run:
systemctl enable awslogsd.service
There are many other commands. E.g., you can also reboot the system:
Further reading: systemctl man page
The Extras Library (aka Amazon Linux Extras Repository or Extras mechanism), provides a way to install up-to-date software bundles (topics in Amazon Linux 2 terminology) without impacting the stability of the rest of the operating system.
Extras Library is not covered by LTS!
To get a list of available topics, run:
$ amazon-linux-extras list
To install an topic, run
amazon-linux-extras install <topic> (e.g.,
amazon-linux-extras install ruby2.4).
If you install (or only enable) a topic, a new repository (plus two for sources and debuginfo) is configured in
I migrated Free Templates for AWS CloudFormation to Amazon Linux 2. In the following, I will outline the problems I was faced with and how I worked around them.
The awslogs agent was renamed
awslogs agent was renamed to
awslogsd but you still install it via
yum install awslogs.
You can start (activate in systemd terminology) awslogs with
systemctl start awslogsd.service (shortcut:
systemctl start awslogsd).
The awslogs agent does not support journald
awslogs agent cannot read logs directly from the journal.
journald fowards all logs to
rsyslog which is configured (
/etc/rsyslog.conf) to write some of the logs to files in
/var/log from where the awslogs agent can pick them up.
Where are the log files?
/var/log does not contain all system logs anymore.
If in doubt, you can access all system logs with
Ruby is missing
Ruby is no longer installed by default. This breaks
cfn-init if you want to install RubyGems.
You can install Ruby 2.0 with
yum install ruby or Ruby 2.4 with
amazon-linux-extras install ruby2.4.
netcat is missing
nc) is no longer installed by default.
You can install
yum install nmap-ncat, but this will install nmap based ncat which behaves differently (e.g., no
-z flag anymore). Learn more
Nginx package not available by default
nginx is no longer part of the default repository.
$ yum install nginx
To install nginx, use the new Amazon Linux Extras Repository
amazon-linux-extras install nginx1.12.
EPEL repository is missing
The EPEL repository (Extra Packages for Enterprise Linux) is no longer installed by default or available to install. The Extras Library replaces the EPEL repository but contains only a fraction of the packages which may causes troubles during your migration.
NAT and ECS optimized AMIs are missing
NAT and ECS optimized AMI are not available. You can replace your NAT instances with NAT Gateways to get around this problem. But for ECS workloads there is no easy workaround. I advise waiting for news from AWS regarding the ECS optimized AMI.
cfn-init is not integrated with the Extras Library
You can not install packages from the Extras Library with the package mechanism in
cfn-init is the way how you can install software onto EC2 instances managed by CloudFormation.
There can either run
amazon-linux-extras enable <topic> before running
cfn-init which than can install the package by using the package mechanism. Or you can use two config sets. The first config sets uses the command mechanism to enable the topic. The second config set uses the package mechanism to install the enabled package. You have to use two config sets because commands run after package installation. Here is an example:
Amazon Linux 2 is the new default for running Linux workloads on AWS. Amazon Linux 2 benefits from systemd, LTS, and a new extras library. There are a few pain points when migrating, most notably the missing EPEL repository. Besides that, you should spend some time to understand how systemd works, because that’s central in modern Linux operating systems.
It’s time to plan your migration from Amazon Linux now!