EFS with TLS behind a proxy

Andreas Wittig – 19 Jun 2019

Encryption of data at rest and in transit is the new normal. Or as Werner Vogels (Amazon, CTO) says: “Dance like nobody’s watching. Encrypt like everyone is.” The Amazon Elastic File System (EFS) supports both: encryption at rest and encryption in transit.

EFS Encryption In Transit

As often, the tooling for the cloud does not deal very well with the fact that there isn’t unrestricted Internet connectivity from every subnet. This article explains how to set up an encrypted connection from an EC2 instance - running in a subnet without a route to an Internet Gateway or NAT gateway - to EFS. That includes scenarios, where your EC2 instance is required to use a proxy for HTTP and HTTPS connections.

How does encryption in transit from EC2 to EFS work? First, the efs-utils tool configures and starts stunnel to establish a TLS connection between the instance and the file system. Next, the tool creates an NFS mount. The following figure illustrates the architecture.

Use efs-util to create EFS mount with TLS

How to mount an EFS file system with TLS behind a proxy or from a subnet without Internet connectivity? First of all, you need to install the efs-utils tool on your EC2 instance.

yum install -y amazon-efs-utils

Next, you need to mount your EFS file system. All you need is the ID of your file system and the following command. Note, the -o tls parameter tells the efs-util to establish a TLS connection to EFS.

mount -t efs -o tls fs-222121ea:/ efs

Unfortunately, the command will fail when your EC2 instance is not running in a subnet with a route to an Internet gateway or NAT gateway.

Andreas and Michael Wittig

Please support our work!

We have published 327 articles, 41 podcast episodes, and 15 videos. It's all free and means a lot of work in our spare time.

Thanks to Alan Leech, Alex DeBrie, e9e4e5f0faef, Goran Opacic, jhoadley, Shawn Tolidano, Thorsten Hoeger, Todd Valentine, Vince Fulco, and all anonymous supporters for your help! We also want to thank all supporters who purchased a cloudonaut t-shirt. It gives us great pleasure to send our t-shirts all over the world.

With your help, we can continue to produce independent & high-quality content focused on AWS. Please support us!

Support us
mount.nfs4: Connection reset by peer
Failed to initialize TLS tunnel for fs-222121ea

Why is that? The efs-util uses stunnel to establish a TLS connection between your EC2 instance and EFS. By default, stunnel uses OCSP (Online Certificate Status Protocol ) to validate the certificate used by EFS. Doing so requires an Internet connection. Even worse, stunnel does not support an HTTP/HTTPS proxy for OCSP.

As of Jul 23, 2019, Amazon no longer enables OCSP by default.

But there is a workaround for the problem: disabling OCSP. To do so, open the file /etc/amazon/efs/efs-utils.conf with the editor of your choice.

vi /etc/amazon/efs/efs-utils.conf

Search for the configuration parameter stunnel_check_cert_validity and set the value from true to false.

#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#

[DEFAULT]
logging_level = INFO
logging_max_bytes = 1048576
logging_file_count = 10
# mode for /var/run/efs in octal
state_file_dir_mode = 750

[mount]
dns_name_format = {fs_id}.efs.{region}.amazonaws.com
stunnel_debug_enabled = false

# Validate the certificate hostname on mount. This option is not supported by certain stunnel versions.
stunnel_check_cert_hostname = true

# Use OCSP to check certificate validity. This option is not supported by certain stunnel versions.
stunnel_check_cert_validity = false

# Define the port range that the TLS tunnel will choose from
port_range_lower_bound = 20049
port_range_upper_bound = 20449

[mount-watchdog]
enabled = true
poll_interval_sec = 1
unmount_grace_period_sec = 30

Afterward, try to mount the EFS again.

mount -t efs -o tls fs-222121ea:/ efs

Works now!

In my opinion, it is fine to disable the certification validation as the risk of a man-in-the-middle attack is very low inside your VPC. The authenticity provided by TLS is not needed, because it is already provided by the VPC service.

AWS is aware of the problem. You might want to check out the GitHub issue discussing the stunnel limitation.

Andreas Wittig

Andreas Wittig

I launched cloudonaut.io in 2015 with my brother Michael. Since then, we have published hundreds of articles, podcast episodes, and videos. It’s all free and means a lot of work in our spare time. We enjoy sharing our AWS knowledge with you.
Have you learned something new by reading, listening, or watching our content? If so, we kindly ask you to support us in producing high-quality & independent AWS content. We look forward to sharing our AWS knowledge with you.

Support us

Feedback? Questions? You can reach me via Email, Twitter, or LinkedIn.