One of the things that I have been wanting to do for some time is to add Gruntwork.io’s Terratest to my Terraform modules. This week I decided to invest some time in getting it running, and I have added it to my EFS Module. One of the learning items on my list was how to run Terratest via a Jenkins Job. Since we discourage installing anything our our Jenkins agents, I needed to figure out how to run Terratest in a docker container.
I started by looking on Docker Hub to see if anybody else had already created a container I could use. I found around 50 results when I searched for Terratest, but there were a number of concerns. My biggest concern was the lack of documentation / access to the source code. Without knowing what they installed on these nodes or even what version of Terraform they used, I was not really comfortable running them. I was able to at least find a couple of containers that had their Dockerfile available in the Overview section, so I could see what they were doing (rqtx, igordcsouza). Both of the Dockerfile examples had outdated versions of Terraform, so I decided since I would have to build it anyways that I should just go ahead and build my own.
My initial iteration started the Dockerfile from rqtx, with a little cleanup and a version update.
FROM golang:1.16
LABEL maintainer="AustinCloudGuru"
ARG tf_version=1.0.0
RUN apt-get update && apt-get install -y gnupg software-properties-common curl \
&& curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add - \
&& apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
&& apt-get update \
&& apt-get install terraform=${tf_version}
WORKDIR $GOPATH/src/app/test/
ENTRYPOINT ["go", "test"]
CMD ["-v"]
This worked well out of the box, but I wanted to be able to run commands other than go test -v on the container, especially in my test scripts, so I removed the ENTRYPOINT and CMD directives.
FROM golang:1.16
LABEL maintainer="AustinCloudGuru"
ARG tf_version=1.0.0
RUN apt-get update && apt-get install -y gnupg software-properties-common curl \
&& curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add - \
&& apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
&& apt-get update \
&& apt-get install terraform=${tf_version}
WORKDIR $GOPATH/src/app/test/
After building the container, I could run my tests with the docker run command.
docker run --rm -it \
-v $PWD:/go/src/app \
-v $(cd ~ && pwd)/.aws/credentials:/root/.aws/credentials \
-v $(cd ~ && pwd)/.aws/config:/root/.aws/config \
-e AWS_SHARED_CREDENTIALS_FILE=/root/.aws/credentials \
-e AWS_CONFIG_FILE=/root/.aws/credentials \
-e AWS_PROFILE=cdp-dev \
austincloud/terratest:latest go test -v
This all worked great on my local machine, so I added a stage to the Jenkinsfile to run the test (AWS creds are handled differently on my Jenkins server than locally).
stage('Run Tests') {
ansiColor('xterm') {
sh '''docker run --rm \
-v $PWD:/go/src/app \
-e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
-e AWS_DEFAULT_REGION \
-e AWS_REGION \
austincloud/terratest:latest go test -v'''
}
}
Everything worked as expected on the first run, but subsequent runs failed with an error. It turns out, the container runs as root and as a result the Terraform files (state, lock, etc) were created as the root user and couldn’t be deleted the agent user (ec2-user). After researching a few options, I decided to add a user to the container and to configure it to run run as that user.
FROM golang:1.16
LABEL maintainer="AustinCloudGuru"
ARG tf_version=1.0.0
ARG uid=1000
ARG gid=1000
ARG user=terratest
ARG group=terratest
ARG terratest_home=/terratest
RUN apt-get update && apt-get install -y gnupg software-properties-common curl \
&& curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add - \
&& apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
&& apt-get update \
&& apt-get install terraform=${tf_version} \
&& mkdir -p ${terratest_home} \
&& chown -R ${uid}:${gid} $terratest_home \
&& groupadd -g ${gid} ${group} \
&& useradd -d "$terratest_home" -u ${uid} -g ${gid} -m -s /bin/bash ${user}
USER ${user}
WORKDIR $GOPATH/src/app/test/
I updated my docker run command and ran it again locally to test.
docker run --rm -it \
-v $PWD:/go/src/app \
-v $(cd ~ && pwd)/.aws/credentials:/terratest/.aws/credentials \
-v $(cd ~ && pwd)/.aws/config:/terratest/.aws/config \
-e AWS_SHARED_CREDENTIALS_FILE=/terratest/.aws/credentials \
-e AWS_CONFIG_FILE=/terratest/.aws/credentials \
-e AWS_PROFILE=cdp-dev \
austincloud/terratest:latest go test -v
This time it failed, hanging on the connection to AWS. After investigating, I found that docker is mounting the .aws/* files as the root user and could not be accessed by my terratest user, even thought hey were in the home directory. So I create a .aws directory on the container and just mounted the whole folder.
FROM golang:1.16
LABEL maintainer="AustinCloudGuru"
ARG tf_version=1.0.0
ARG uid=1000
ARG gid=1000
ARG user=terratest
ARG group=terratest
ARG terratest_home=/terratest
RUN apt-get update && apt-get install -y gnupg software-properties-common curl \
&& curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add - \
&& apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
&& apt-get update \
&& apt-get install terraform=${tf_version} \
&& mkdir -p ${terratest_home}/.aws \
&& chown -R ${uid}:${gid} $terratest_home \
&& groupadd -g ${gid} ${group} \
&& useradd -d "$terratest_home" -u ${uid} -g ${gid} -m -s /bin/bash ${user}
USER ${user}
WORKDIR $GOPATH/src/app/test/
I ran the command again and all the Terraform files were created as UID 1000 (which is the same user Jenkins uses). I was also able to shorten the manual command by removing some of the AWS environment variables since they could now be found in the home directory of the user.
docker run --rm -it \
-v $PWD:/go/src/app \
-v $(cd ~ && pwd)/.aws:/terratest/.aws \
-e AWS_PROFILE=cdp-dev \
austincloud/terratest:latest go test -v
I’ve created containers using the latest patches of Terraform versions .13, .14, .15, and 1.0 and published them at Docker Hub. The source repository is available on GitHubhttps://github.com/AustinCloudGuru/docker-terratest if you are interested seeing exactly what is being done.
Happy Testing!
Comments