One of the bigger challenges I have seen in managing AWS environments is how to handle the need for developers to log into a instance. While we try to build our infrastructure with tooling that negates the need to log in to an instance in about 95% of use cases, there is still the 5% where we need hands on a box.
There are a number of ways to do this, but most are either difficult to maintain in a dynamic environment (centralized authentication such as LDAP) or insecure (sharing the AWS key that was used to launch an instance). Most of the time it tends to be the latter, which is why I have been extremely excited to look at the SSH Backend for Vault.
There were a couple of gotcha’s, and I still need to figure out how to restrict login to specific boxes, but below is the steps I took to get this SSH Backend configured in our environment.
Preparing the Environment
Before setting up the SSH Backend, it helps to set some environmental variables first. Set the VAULT_ADDR to the URL of your Vault server (I like to include the “https://” so there is less to type) and set VAULT_TOKEN to the token you will be using to execute the commands.
export VAULT_ADDR=https://vault.example.com:8200
export VAULT_TOKEN=a38dc275-86d3-48bd-57ae-237a45d6663b
Once set, you can test your configuration using the curl command to get the health endpoint.
curl -s -k -X GET ${VAULT_ADDR}/v1/sys/health
{"initialized":true,"sealed":false,"standby":false,"server_time_utc":1477441389,"version":"0.6.2","cluster_name":"vault-cluster-2fbd0333","cluster_id":"d8056c7f-acbb-ae59-4ed4-3673f2d27d48"}
Initialize the SSH Backend
Once you have verified that everything is working, you can create and configure the SSH backend.
export PAYLOAD='{"type": "ssh", "description": "SSH Signer", "config": { "max_lease_ttl": "24h"}}'
curl -s -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d "${PAYLOAD}" ${VAULT_ADDR}/v1/sys/mounts/ssh-client-signer
This will create a new mount point with the max lease time of 24 hours. Since it doesn’t return anything, you can verify that it has been created with the mounts endpoint. As always, I recommend using jq for parsing the output.
curl -s -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/sys/mounts|jq .
Now that the backend is complete, you can either upload your own certificate authority, or let Vault create one for you. I’ve opted to let Vault create mine.
export PAYLOAD='{"generate_signing_key": true}'
curl -s -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d "${PAYLOAD}" ${VAULT_ADDR}/v1/ssh-client-signer/config/ca
This returns the public key. You can also get the public key and save it to a file at any time by running the command:
curl -s -k -X GET ${VAULT_ADDR}/v1/ssh-client-signer/public_key
Create Roles
Now that the backend is configured, the next step is to create roles for the backend. The following command will setup a role that will allow the certificate holder to log into a system as any user but defaults to ec2-user.
export PAYLOAD='{"allow_user_certificates": true, "allowed_users": "*", "default_extensions": [{"permit-pty": ""}], "key_type": "ca", "default_user": "ec2-user", "ttl": "30m0s"}'
curl -s -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d "${PAYLOAD}" ${VAULT_ADDR}/v1/ssh-client-signer/roles/aws
You can validate the role
curl -s -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/ssh-client-signer/roles?list=true|jq -r .
And you can get more information on the specific role
curl -s -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/ssh-client-signer/roles/aws|jq -r .
Setup Hosts to Use Vault
Once you have the backend configured, you will need to configure the host to use the it. This will require downloading the public key to each server and updating the sshd_config file to use the key. You can add the key directly to the ssh hosts by logging into the host and running the following (assuming VAULT_ADDR is set on the host):
curl -s -k -o /etc/ssh/trusted-user-ca-keys.pem ${VAULT_ADDR}/v1/ssh-client-signer/public_key
and then update the sshd_config file to use the new setting and restart SSHD.
printf "\nTrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem" >> /etc/ssh/sshd_config
service sshd restart
You can now request a key
export PAYLOAD='{"valid_principals": "ec2-user", "public_key": "'`cat ~/.ssh/id_rsa.pub`'"}'
curl -s -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d "${PAYLOAD}" ${VAULT_ADDR}/v1/ssh-client-signer/sign/test-role-ec2-user |jq -r .data.signed_key >id_rsa-cert.pub
Now you can log into the box
ssh -I id_rsa-cert.pub -i ~/.ssh/id_rsa ec2-user@172.1.1.10
You can check the validity of the key with the ssh-keygen command:
ssh-keygen -Lf ~/.ssh/signed-cert.pub
Other Commands
If you decide yo you need to remove a role, you can do it with the following:
curl -s -k -X DELETE -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/ssh-client-signer/roles/test-role-ec2-user
Comments