6 tips and tricks for AWS command-line ninjas

Andreas Wittig – 09 Jun 2016

The AWS Command Line Interface (CLI) allows you to manage AWS services. Using the CLI from your terminal interactively allows you to half-automate tasks and frees you from logging into the AWS Management Console. In addition integrating the CLI into shell scripts allows you to automate your infrastructure and the configuration of EC2 instances during the boot process.

This article covers typical hurdles when using the AWS CLI.

Command Completion

When using the CLI interactively within your terminal command completion is a killer feature you should not miss. When enabled Command Completion allows you to use the TAB key to complete commands. This will speed up your CLI usage significant.

The following steps are needed to enable Command Completion for bash on OS X:

echo "complete -C aws_completer aws" >> ~/.bash_profile
source ~/.bash_profile

The official documentation contains general instructions for other shells as well.

Filtering results of requests on server-side

By default, the CLI uses a page size of 1000 and retrieves all available items. If you need to request items from a list of more than 1000 items or if you want to speed up your commands it is a good idea to filter the results of your request on server-side.

Many describe-*and list-* commands support server-side filtering: --filter. For example it is possible to filter EC2 instances by instance type:

$ aws ec2 describe-instances --filter Name=instance-type,Values=t2.nano

Filtering output on client-side

Another useful feature of the CLI is filtering the output of any command on client-side: --query. The JMESPath query language is used for filtering.

The following example lists all VPCs within a region and filters the results by using a --query.

$ aws ec2 describe-vpcs --query "Vpcs[?VpcId == 'vpc-aaa22bbb'].CidrBlock"

You might need the CIDR of a VPC as a variable in your shell script. The following example shows how to achieve that. Formatting the output as text by adding the parameter --output text removes " character from the JSON result.

Andreas and Michael Wittig

Hej, Andreas & Michael here!

We launched the cloudonaut blog in 2015. Since then, we have published 325 articles: small tips and tricks, best practices, and service reviews. We enjoy writing about all things AWS a lot.

Do you like our blog posts and podcast episodes? Have you learned something new? Consider supporting us create in-depth and independent AWS content. Please help us with a monthly or one-time payment through GitHub Sponsors.

Start supporting us today!
CIDR=$(aws ec2 describe-vpcs --query "Vpcs[?VpcId == 'vpc-aaa22bbb'].CidrBlock" --output text)
echo $CIDR

Wait for …

When writing shell scripts by using the CLI there will be the need to wait for a specific condition from time to time. For example, after initiating an EBS snapshot your script might need to wait until the snapshot was completed. Waiting can be achieved with a polling loop and a describe-* command. But there is a simpler solution built into the CLI for this: aws <service> wait <condition>.

The following example contains a wait command that will block the script until the snapshot has been completed.

echo "Waiting for EBS snapshot"
aws ec2 wait snapshot-completed --snapshot-ids snap-aabbccdd
echo "EBS snapshot completed"

Assuming an IAM role

The CLI supports assuming an IAM role. Very handy if you need to switch between multiple AWS accounts with the help of cross-account roles.

All you need to do is to configure two profiles in ~/.aws/config: an IAM user and an IAM role profile.

[profile iam-user]
output = json
region = eu-west-1

[profile iam-role]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE>
source_profile = iam-user
output = json
region = eu-west-1

Only the IAM user needs security credentials stored in ~/.aws/credentials.

aws_access_key_id = ***
aws_secret_access_key = ***

Afterwards, you are able to assume the IAM role by adding --profile iam-role to your CLI commands.

Fine-tuning S3 config

The AWS CLI includes transfer commands for S3: cp, sync, mv, and rm. You are able to fine-tune these commands with special configuration.

For example if you need to sync a large number of small files to S3, the increasing the following values added to your ~/.aws/config config file will speed up the sync process.

[profile default]
s3 =
max_concurrent_requests = 100
max_queue_size = 10000
use_accelerate_endpoint = true

The official documentation contains detailed information about additional S3 configuration values.

Andreas Wittig

Andreas Wittig

I'm an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I'm writing this blog and all other projects together with my brother Michael.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

Feedback? Questions? Drop me a line: Email, Twitter, LinkedIn.

Briefcase icon
Hire me