EFS with TLS behind a proxy

Encryption of data at rest and in transit is the new normal. Or as Werner Vogels (Amazon, CTO) says: “Dance like nobody’s watching. Encrypt like everyone is.” The Amazon Elastic File System (EFS) supports both: encryption at rest and encryption in transit.

As often, the tooling for the cloud does not deal very well with the fact that there isn’t unrestricted Internet connectivity from every subnet. This article explains how to set up an encrypted connection from an EC2 instance - running in a subnet without a route to an Internet Gateway or NAT gateway - to EFS. That includes scenarios, where your EC2 instance is required to use a proxy for HTTP and HTTPS connections.

How does encryption in transit from EC2 to EFS work? First, the efs-utils tool configures and starts stunnel to establish a TLS connection between the instance and the file system. Next, the tool creates an NFS mount. The following figure illustrates the architecture.

How to mount an EFS file system with TLS behind a proxy or from a subnet without Internet connectivity? First of all, you need to install the efs-utils tool on your EC2 instance.

yum install -y amazon-efs-utils

Next, you need to mount your EFS file system. All you need is the ID of your file system and the following command. Note, the -o tls parameter tells the efs-util to establish a TLS connection to EFS.

mount -t efs -o tls fs-222121ea:/ efs

Unfortunately, the command will fail when your EC2 instance is not running in a subnet with a route to an Internet gateway or NAT gateway.

mount.nfs4: Connection reset by peer
Failed to initialize TLS tunnel for fs-222121ea

Why is that? The efs-util uses stunnel to establish a TLS connection between your EC2 instance and EFS. By default, stunnel uses OCSP (Online Certificate Status Protocol ) to validate the certificate used by EFS. Doing so requires an Internet connection. Even worse, stunnel does not support an HTTP/HTTPS proxy for OCSP.

As of Jul 23, 2019, Amazon no longer enables OCSP by default.

But there is a workaround for the problem: disabling OCSP. To do so, open the file /etc/amazon/efs/efs-utils.conf with the editor of your choice.

vi /etc/amazon/efs/efs-utils.conf

Search for the configuration parameter stunnel_check_cert_validity and set the value from true to false.

#
# Copyright 2017-2018 Amazon.com, Inc. and its affiliates. All Rights Reserved.
#
# Licensed under the MIT License. See the LICENSE accompanying this file
# for the specific language governing permissions and limitations under
# the License.
#

[DEFAULT]
logging_level = INFO
logging_max_bytes = 1048576
logging_file_count = 10
# mode for /var/run/efs in octal
state_file_dir_mode = 750

[mount]
dns_name_format = {fs_id}.efs.{region}.amazonaws.com
stunnel_debug_enabled = false

# Validate the certificate hostname  on mount. This option is not supported by certain stunnel versions.
stunnel_check_cert_hostname = true

# Use OCSP to check certificate validity. This option is not supported by certain stunnel versions.
stunnel_check_cert_validity = false

# Define the port range that the TLS tunnel will choose from
port_range_lower_bound = 20049
port_range_upper_bound = 20449

[mount-watchdog]
enabled = true
poll_interval_sec = 1
unmount_grace_period_sec = 30

Afterward, try to mount the EFS again.

mount -t efs -o tls fs-222121ea:/ efs

Works now!

In my opinion, it is fine to disable the certification validation as the risk of a man-in-the-middle attack is very low inside your VPC. The authenticity provided by TLS is not needed, because it is already provided by the VPC service.

AWS is aware of the problem. You might want to check out the GitHub issue discussing the stunnel limitation.