AWS SSM is a trojan horse: fix it now!

Recently, I held a security workshop together with a team of engineers. At some point, the team demonstrated how they use AWS Systems Manager (SSM) to run commands on a machine. What the team didn’t know: they enabled a backdoor that allows everyone with access to the AWS account to run commands on every EC2 instance as root. On top of it, they granted full read and write permissions to every S3 bucket and full write access to CloudWatch Logs. One team member nailed it: we installed a trojan horse! Read on to understand how the backdoor works and how you can close it.

How the trojan horse works

The three steps to enable the trojan horse:

1. SSM is a handy service to replace SSH, patch your OS, and much more. To use SSM, you have to install the SSM agent on your EC2 instances. Amazon Linux 2 comes with the SSM agent pre-installed and pre-started. The SSM agent runs with root privileges.
2. You also have to grant your EC2 instances permissions to talk to the SSM API. The docs suggest that you attach the managed policy AmazonEC2RoleforSSM to your IAM instance (using an IAM instance profile with an IAM role).
3. To use SSM, your IAM user or role also needs permissions. Very likely, you have those permissions thanks to managed policies like AdministratorAccess, PowerUserAccess, or AmazonSSMFullAccess.

You can now use SSM Run Commands or Session Manager to execute any command on any EC2 instance as root. Even better, all commands that are executed use the permissions of the EC2 instance where it runs on. Thanks to AmazonEC2RoleforSSM you can now read all the data from S3, or do you prefer to override some data stored on S3, or you might want to inject some nonsense logs? No problem.

Protect yourself: limit access to SSM

You have to be very careful about the following permissions which can be used to execute a command on an EC2 instance via the SSM agent:

• ssm:CreateAssociation
• ssm:CreateAssociationBatch
• ssm:RegisterTargetWithMaintenanceWindow
• ssm:RegisterTaskWithMaintenanceWindow
• ssm:SendCommand
• ssm:StartAutomationExecution
• ssm:StartSession
• ssm:UpdateMaintenanceWindowTarget
• ssm:UpdateMaintenanceWindowTask

Fine, so we need to restrict access to these actions to certain EC2 instances only. Unfortunately, the documentation of the resource-level permissions and conditions is incomplete. It is unclear whether it is possible to restrict all of the actions to certain EC2 instances. AWS support could not help as well. Unacceptable!

If you still want to use SSM, I recommend that you allow ssm:SendCommand and ssm:StartSession only for specific tags using permissions. You also have to ensure that tagging is restricted! Otherwise, the tag-based restriction is pointless.

Assuming that multiple workloads run in the same account and you want to restrict access to only some engineers I highly recommend to put each workload in a separate AWS account. It’s unlikely that you can separate the permission properly in the same AWS account.

Protect yourself: limit permissions of the SSM agent

Let’s look at the AmazonEC2RoleforSSM policy statements in more details. Remember, you attached the managed policy to your EC2 instance to allow the SSM agent to talk to the AWS API.

The first statement allows your EC2 instance to report data to SSM. But it is not restricted to a specific instance. You (or an attacker on the machine) can send inventory and compliance data not just in its name. You can do this for every instance id. I don’t see how this can help me to ensure compliance, sorry.

{
  "Effect": "Allow",
  "Action": [
    "ssm:PutInventory",
    "ssm:PutComplianceItems",
    "ssm:PutConfigurePackageResult",
    "ssm:UpdateAssociationStatus",
    "ssm:UpdateInstanceAssociationStatus",
    "ssm:UpdateInstanceInformation"
  ],
  "Resource": "*"
}

The next statement grants you access to write logs to and CloudWatch log group!

{
  "Effect": "Allow",
  "Action": [
    "logs:CreateLogGroup",
    "logs:CreateLogStream",
    "logs:DescribeLogGroups",
    "logs:DescribeLogStreams",
    "logs:PutLogEvents"
  ],
  "Resource": "*"
}

And finally, you can read and write from all S3 buckets.

{
  "Effect": "Allow",
  "Action": [
    "s3:GetBucketLocation",
    "s3:PutObject",
    "s3:GetObject",
    "s3:GetEncryptionConfiguration",
    "s3:AbortMultipartUpload",
    "s3:ListMultipartUploadParts",
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads"
  ],
  "Resource": "*"
}

I suggest that you do not use the AmazonEC2RoleforSSM. Instead, to make Run Commands and Session Manager work, the following policy grants enough permissions:

{
  "Effect": "Allow",
  "Action": [
    "ssmmessages:*",
    "ssm:UpdateInstanceInformation",
    "ec2messages:*"
  ],
  "Resource": "*"
}

Depending on the SSM documents you run, you might need to add additional permissions. I created an SSM example for the cfn-modules project.

Summary

Be careful when using any defaults. Especially when it comes to managed IAM policies provided by AWS. To restrict access to SSM and your EC2 instance, consider the following steps:

1. If you can, use multiple AWS accounts to separate workloads. As soon as they run in a single AWS account, it’s unlikely that you can separate SSM permissions properly.
2. Are you not using SSM? Stop and remove the pre-installed SSM agent from your EC2 instances.
3. Do not attach the managed policy AmazonEC2RoleforSSM to any of your EC2 instances.
4. If you use SSM, restrict your engineer’s permissions to SSM. Never grant access to ssm:*. And make sure to replace the managed policy AmazonEC2RoleforSSM with a policy customized to your needs.

Are you interested in an AWS security workshop or review? Get in touch: hello@cloudonaut.io.