I’ve been putting off setting up the AWS backend in our Vault server for the last few months. I knew that I was going to need it eventually, but other priorities kept taking precedence so I have been pushing it off. This past week one of the application teams came to me with a requirement for needing to write a file to an S3 bucket.
Under normal circumstances, I would probably just go ahead and create an instance profile that could be applied to the system and the problem would be solved. The problem with this approach was that they wanted did not want other applications to be able to access it. Since we run containers, using instance profiles to control access would allow every container on that host access to the buckets.
Preparing the Environment
Setting up the AWS backend is pretty straight forward. To begin, you need to configure your environment to be able to interact with Vault.
export VAULT_ADDR=vault.example.com:8200
export VAULT_TOKEN=a38dc275-86d3-48bd-57ae-237a45d6663b
Once set, you can test your configuration using the curl command to go to the health endpoint.
% curl -k -X GET ${VAULT_ADDR}/v1/sys/health
{"initialized":true,"sealed":false,"standby":false,"server_time_utc":1477441389,"version":"0.6.2","cluster_name":"vault-cluster-2fbd0333","cluster_id":"d8056c7f-acbb-ae59-4ed4-3673f2d27d48"}
Initialize the AWS Backend
Once you have verified that the endpoint is working, you can create and configure the AWS Backend. Since we use multiple AWS accounts for each environment, I will mount different backends for each account.
curl -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d '{"type": "aws", "description": "AWS Backend", "config": {"default_lease_ttl": "360", "max_lease_ttl": "720"}}' ${VAULT_ADDR}/v1/sys/mounts/aws-prototype
This command sets up the aws-prototype backend with a default lease time of 5 minutes and a max lease time of 10 minutes. Since the post doesn’t return anything, you can verify it with the mounts endpoint. If you don’t have jq, I highly recommend you download it, as it makes viewing JSON output much easier
curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/sys/mounts|jq .
Configure AWS Backend
Once the mount is created, you will need to add AWS credentials to the backend. You will need to create an AWS user that has full IAM access so that it can create other users. We have automation that controls our IAM, but you can use a couple of IAM commands to set up the user you will need.
aws iam create-user --user-name HashiVault
aws iam attach-user-policy --user-name HashiVault --policy-arn arn:aws:iam::aws:policy/IAMFullAccess
aws iam create-access-key --user-name HashiVault
Once you create the access & secret keys, you can use them to configure the AWS backend.
curl -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d '{"access_key": "XXXXXXXXXXXXXXXXXXXX", "secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "region": "us-east-1"}' ${VAULT_ADDR}/v1/aws-prototype/config/root
After the backend is configured, you can start adding roles to Vault. For the S3 access that we want, we need to create a role that creates a user that has the right policy attached.
curl -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d '{"arn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"}' ${VAULT_ADDR}/v1/aws-prototype/roles/S3-Access
You can verify that it was created properly by curling the endpoint and getting the credentials back.
curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/aws-prototype/creds/S3-Access
Now that you are getting credentials, you can repeat the process for every account and/or role that you need to have setup.
Testing the S3 Access
I had a little problem testing my S3 access once I had everything configured. I wrote a quick little one-liner to get my creds and set them to the proper environmental variables.
CREDS=$(curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/aws-prototype/creds/S3-Access);export AWS_ACCESS_KEY_ID=$(echo $CREDS |jq -r .data.access_key);export AWS_SECRET_ACCESS_KEY=$(echo $CREDS |jq -r .data.secret_key)
When I tried to download a file, I received and error.
download failed: s3://mybucket/testfile.txt to ./testfile.txt An error occurred (InvalidArgument) when calling the GetObject operation: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4. You can enable AWS Signature Version 4 by running the command:
aws configure set s3.signature_version s3v4
I ran the command, but each typed I tried to run the aws s3 command, I received the same error. What I learned was that running the command updated my .aws/config file and added the following line to it:
[default]
s3 =
signature_version = s3v4
Since I already have a ‘[profile default]’ set with nothing in it, I moved the S3 bits up underneath that and removed the ‘[default]’ block and everything started working as expected.
Comments