EFS with TLS behind a proxy

Andreas Wittig – 19 Jun 2019

Encryption of data at rest and in transit is the new normal. Or as Werner Vogels (Amazon, CTO) says: “Dance like nobody’s watching. Encrypt like everyone is.” The Amazon Elastic File System (EFS) supports both: encryption at rest and encryption in transit.

EFS Encryption In Transit

As often, the tooling for the cloud does not deal very well with the fact that there isn’t unrestricted Internet connectivity from every subnet. This article explains how to set up an encrypted connection from an EC2 instance - running in a subnet without a route to an Internet Gateway or NAT gateway - to EFS. That includes scenarios, where your EC2 instance is required to use a proxy for HTTP and HTTPS connections.

How does encryption in transit from EC2 to EFS work? First, the efs-utils tool configures and starts stunnel to establish a TLS connection between the instance and the file system. Next, the tool creates an NFS mount. The following figure illustrates the architecture.

Use efs-util to create EFS mount with TLS

How to mount an EFS file system with TLS behind a proxy or from a subnet without Internet connectivity? First of all, you need to install the efs-utils tool on your EC2 instance.

yum install -y amazon-efs-utils

Next, you need to mount your EFS file system. All you need is the ID of your file system and the following command. Note, the -o tls parameter tells the efs-util to establish a TLS connection to EFS.

mount -t efs -o tls fs-222121ea:/ efs

Unfortunately, the command will fail when your EC2 instance is not running in a subnet with a route to an Internet gateway or NAT gateway.

mount.nfs4: Connection reset by peer
Failed to initialize TLS tunnel for fs-222121ea

Why is that? The efs-util uses stunnel to establish a TLS connection between your EC2 instance and EFS. By default, stunnel uses OCSP (Online Certificate Status Protocol ) to validate the certificate used by EFS. Doing so requires an Internet connection. Even worse, stunnel does not support an HTTP/HTTPS proxy for OCSP.

As of Jul 23, 2019, Amazon no longer enables OCSP by default.

But there is a workaround for the problem: disabling OCSP. To do so, open the file /etc/amazon/efs/efs-utils.conf with the editor of your choice.

vi /etc/amazon/efs/efs-utils.conf

Search for the configuration parameter stunnel_check_cert_validity and set the value from true to false.

# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.

logging_level = INFO
logging_max_bytes = 1048576
logging_file_count = 10
# mode for /var/run/efs in octal
state_file_dir_mode = 750

dns_name_format = {fs_id}.efs.{region}.amazonaws.com
stunnel_debug_enabled = false

# Validate the certificate hostname on mount. This option is not supported by certain stunnel versions.
stunnel_check_cert_hostname = true

# Use OCSP to check certificate validity. This option is not supported by certain stunnel versions.
stunnel_check_cert_validity = false

# Define the port range that the TLS tunnel will choose from
port_range_lower_bound = 20049
port_range_upper_bound = 20449

enabled = true
poll_interval_sec = 1
unmount_grace_period_sec = 30

Afterward, try to mount the EFS again.

mount -t efs -o tls fs-222121ea:/ efs

Works now!

In my opinion, it is fine to disable the certification validation as the risk of a man-in-the-middle attack is very low inside your VPC. The authenticity provided by TLS is not needed, because it is already provided by the VPC service.

AWS is aware of the problem. You might want to check out the GitHub issue discussing the stunnel limitation.

Andreas Wittig

Andreas Wittig

I’m the author of Amazon Web Services in Action. I work as a software engineer, and independent consultant focused on AWS and DevOps.

You can contact me via Email, Twitter, and LinkedIn.

Briefcase icon
Hire me
Cover of Rapid Docker on AWS

New book: Rapid Docker on AWS

A rapid way to get your web application up and running on AWS. Made for web developers and DevOps engineers who want to dockerize their web applications and run their containers on Amazon Web Services. Prior knowledge of Docker and AWS is not required.

Buy icon
Buy now
Marbot Logo

Incident Management for Slack

Team up to solve incidents with our chatbot marbot. Never miss a critical alert. Escalate alerts from your AWS infrastructure among your team members. Strong integrations with all parts of your AWS infrastructure: CloudWatch, Elastic Beanstalk, RDS, EC2, ...

Slack icon
Try for free