🆕 Video Course Out Now: Rapid Docker on AWS

EFS with TLS behind a proxy

Andreas Wittig – 19 Jun 2019

Encryption of data at rest and in transit is the new normal. Or as Werner Vogels (Amazon, CTO) says: “Dance like nobody’s watching. Encrypt like everyone is.” The Amazon Elastic File System (EFS) supports both: encryption at rest and encryption in transit.

EFS Encryption In Transit

As often, the tooling for the cloud does not deal very well with the fact that there isn’t unrestricted Internet connectivity from every subnet. This article explains how to set up an encrypted connection from an EC2 instance - running in a subnet without a route to an Internet Gateway or NAT gateway - to EFS. That includes scenarios, where your EC2 instance is required to use a proxy for HTTP and HTTPS connections.

How does encryption in transit from EC2 to EFS work? First, the efs-utils tool configures and starts stunnel to establish a TLS connection between the instance and the file system. Next, the tool creates an NFS mount. The following figure illustrates the architecture.

Use efs-util to create EFS mount with TLS

How to mount an EFS file system with TLS behind a proxy or from a subnet without Internet connectivity? First of all, you need to install the efs-utils tool on your EC2 instance.

yum install -y amazon-efs-utils

Next, you need to mount your EFS file system. All you need is the ID of your file system and the following command. Note, the -o tls parameter tells the efs-util to establish a TLS connection to EFS.

mount -t efs -o tls fs-222121ea:/ efs

Unfortunately, the command will fail when your EC2 instance is not running in a subnet with a route to an Internet gateway or NAT gateway.

Cover of Amazon Web Services in Action

Level up, strengthen your AWS skills.

Our book Amazon Web Services in Action is a comprehensive introduction to computing, storing, and networking in the AWS cloud. You'll find clear, relevant coverage of all the essential AWS services, emphasizing best practices for security, high availability, and scalability. Get the first chapter for free!

mount.nfs4: Connection reset by peer
Failed to initialize TLS tunnel for fs-222121ea

Why is that? The efs-util uses stunnel to establish a TLS connection between your EC2 instance and EFS. By default, stunnel uses OCSP (Online Certificate Status Protocol ) to validate the certificate used by EFS. Doing so requires an Internet connection. Even worse, stunnel does not support an HTTP/HTTPS proxy for OCSP.

As of Jul 23, 2019, Amazon no longer enables OCSP by default.

But there is a workaround for the problem: disabling OCSP. To do so, open the file /etc/amazon/efs/efs-utils.conf with the editor of your choice.

vi /etc/amazon/efs/efs-utils.conf

Search for the configuration parameter stunnel_check_cert_validity and set the value from true to false.

#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#

[DEFAULT]
logging_level = INFO
logging_max_bytes = 1048576
logging_file_count = 10
# mode for /var/run/efs in octal
state_file_dir_mode = 750

[mount]
dns_name_format = {fs_id}.efs.{region}.amazonaws.com
stunnel_debug_enabled = false

# Validate the certificate hostname on mount. This option is not supported by certain stunnel versions.
stunnel_check_cert_hostname = true

# Use OCSP to check certificate validity. This option is not supported by certain stunnel versions.
stunnel_check_cert_validity = false

# Define the port range that the TLS tunnel will choose from
port_range_lower_bound = 20049
port_range_upper_bound = 20449

[mount-watchdog]
enabled = true
poll_interval_sec = 1
unmount_grace_period_sec = 30

Afterward, try to mount the EFS again.

mount -t efs -o tls fs-222121ea:/ efs

Works now!

In my opinion, it is fine to disable the certification validation as the risk of a man-in-the-middle attack is very low inside your VPC. The authenticity provided by TLS is not needed, because it is already provided by the VPC service.

AWS is aware of the problem. You might want to check out the GitHub issue discussing the stunnel limitation.

Andreas Wittig

Andreas Wittig

I’m an independent consultant, technical writer, and programming founder. All these activities have to do with AWS. I’m writing this blog and all other projects together with my brother Michael.

In 2009, we joined the same company as software developers. Three years later, we were looking for a way to deploy our software—an online banking platform—in an agile way. We got excited about the possibilities in the cloud and the DevOps movement. It’s no wonder we ended up migrating the whole infrastructure of Tullius Walden Bank to AWS. This was a first in the finance industry, at least in Germany! Since 2015, we have accelerated the cloud journeys of startups, mid-sized companies, and enterprises. We have penned books like Amazon Web Services in Action and Rapid Docker on AWS, we regularly update our blog, and we are contributing to the Open Source community. Besides running a 2-headed consultancy, we are entrepreneurs building Software-as-a-Service products.

We are available for projects.

You can contact me via Email, Twitter, and LinkedIn.

Briefcase icon
Hire me