One of the projects that I have been working on for the last few weeks is to migrate our Jenkins slaves to use IAM roles to manage our AWS accounts rather than credentials. Policy dictates that we rotate our credentials every 90 days for machine accounts, so every three months teams have to rotate credentials for 3-10 AWS accounts. As you can imagine, this can be painful. Even worse, our password policies configured in AWS require that accounts get locked out at the end of 90 days, so if somebody forgets one account, it could break some builds. By switching to IAM roles, we can configure our slaves to use the role to manage AWS accounts and the individual teams can quit worring about password rotation.

Setting it up

The first step is to create the role for the jenkins slave to use. This role is pretty simple, and essentially only allows the slave node to assume the role in another account. If you have a role already attached to your slave for other access reasons, you can add the statement block to your existing role.

Create a text file called jenkins-slave.json and add the following policy to it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ec2.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Using your favorite orchestration tool (I normally use Terraform, but for the sake of this article I chose the AWS CLI), create the role.

aws iam create-role --role-name jenkins-slave --assume-role-policy-document file://jenkins-slave.json

Once the role is created, create a text file called jenkins-slave-policy.json and add the following text:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": [
                "arn:aws:iam::111111111111:role/jenkins",
                "arn:aws:iam::222222222222:role/jenkins"
            ]
        }
    ]
}

Create the policy:

aws iam create-policy --policy-name jenkins-slave --policy-document file://jenkins-slave.json

Once the policy is in place, it needs to be attached to the role that was created.

aws iam attach-role-policy --role-name jenkins-slave --policy-arn arn:aws:iam::111111111111:policy/jenkins-slave

Finally, create the instance profile and add the role to the instance profile.

Create the Instance Profile

aws iam create-instance-profile --instance-profile-name jenkins-slave
aws iam add-role-to-instance-profile --instance-profile-name jenkins-slave --role-name jenkins-slave

Create the Role Jenkins will Assume

With the Jenkins slave setup, the next thing to do is setup the role in the account that Jenkins is going to be managing. Start by creating a trust policy document called jenkins-trust.json. The principle should be the account number where the Jenkins slaves exist. We decided to also use the ExternalId so that different teams wouldn’t be able to accidentally run jobs in AWS accounts that are not theirs.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "7KAXZ44tHx8mKsPxZnCi2X6Yz"
        }
      }
    }
  ]
}

Create the role using the CLI.

aws iam create-role --role-name jenkins --assume-role-policy-document file://jenkins-trust.json

Create a textfile called jenkins-policy.json and add to it the permissions that you want Jenkins to be able to control. The example allows Jenkins full control over EC2, but nothing else.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:*",
                "iam:ListInstanceProfilesForRole",
                "iam:PassRole",
            ],
            "Resource": "*"
        }
    ]
}

Add the policy to AWS.

aws iam create-policy --policy-name jenkins --policy-document file://jenkins-policy.json

Attach the policy.

aws iam attach-role-policy --role-name jenkins --policy-arn arn:aws:iam::222222222222:policy/jenkins

Update your job to use STS:AssumeRole

Now that the roles are configured in both AWS accounts, the final step is to update the Jenkins jobs to use the role instead of the credentials. Start by creating a new secret text credential in your credential store and insert the ExternalID. Next, install the Pipeline: AWS Steps plugin on Jenkins. Once those are in place, you can use the example below as a template for updating your own jobs.

node {
	  properties([
      parameters([
          string(name: 'role_account', description: 'Account', trim: false)
      ])
    ])
    withCredentials(
        [
            string(credentialsId: 'ExternalId', variable: 'external_id')
        ]
    )
    {
        stage('hello AWS') {
            withAWS(role:'jenkins', roleAccount:"$role_account", externalId: "$external_id", duration: 900, roleSessionName: 'jenkins-session')
            {
                sh 'echo "hello KB">hello.txt'
                s3Upload acl: 'Private', bucket: 'myjenkins-test-12345', file: 'hello.txt'
                s3Download bucket: 'myjenkins-test-12345', file: 'downloadedHello.txt', path: 'hello.txt', force: true
                sh 'cat downloadedHello.txt'
             }
        }
    }
}

I’ve actually updated my Jenkins server and all of my AWS accounts to use this same methodology, eliminating the need for any user accounts in all but 1 of my AWS accounts and I no longer have to worry about rotating credentials.