Secure Cross Account Continuous Delivery Pipelines using AWS CodePipeline

An Akkodis client in the Lands industry has been an Akkodis Managed Services customer for many years, and Akkodis has been responsible for running the majority of Cloud workloads in AWS for the client since 2014.

As part of this ongoing engagement, Akkodis recognised a need for an uplift in DevOps capability for the client and delivered a secure cross account pipeline using AWS CodePipeline.

The solution focussed on providing faster and more reliable delivery pipelines, with the ability to scale the solution out across multiple AWS workloads and accounts.

The client is using Amazon Web Services as their Cloud platform for a number of workloads, with tooling and processes that have evolved over many years.

The evolution that has occurred over this time, both in the AWS landscape and in DevOps maturity, has resulted in a set of processes for building and managing workloads that do not always deliver speed, reliability and repeatability for the deployment of workloads.

It was identified that a standardised approach, using cloud-native services and best practices, would be provide a substantial uplift in capability and deliver additional value for the client.

Akkodis often recommends the use of Continuous Integration and Continuous Delivery (CI/CD) as best practice for building, deploying and operating workloads, and emphasises the use of end-to-end pipelines and automation.

Continuous Integration is the practice by which developers will integrate their code into a shared code repository on a regular basis, preferably multiple times per day. Each code check-in triggers an automated build process to verify the code and execute unit tests to identify problems as early as possible in the development cycle.

Continuous Delivery is the practice of writing software in short cycles, ensuring that it can be reliably released at any time, with deployment automated as much as possible. This technique in combination with Continuous Integration enables releasing software with greater speed and frequency

Through experience and insights from deep and varied customer engagements, Akkodis recommends the use of cloud-native services where possible to reduce the operational overheads of managing the DevOps tool chain and leverage the scalability and resilience those services offer.

AWS CodePipeline is AWS’s managed service offering for CI/CD and was chosen to implement delivery pipelines for this client, with AWS CodeCommit used for source control.

A cross account pipeline was developed using AWS CodePipeline, which is deployed and managed from the Tools account. A centralised approach provides an end-to-end process deploying a workload from Development through to Production, and addresses concerns around visibility and consistency.

To ensure that the foundations for creating pipelines in AWS CodePipeline were firmly in place, and to avoid a proliferation of duplicate resources, a number of common resources were created as part of a common stack. Supporting cross account roles were also created which allows AWS CodePipeline to assume those roles, and perform actions within the target accounts.

This process is illustrated as follows:

  1. A DevOps Engineer commits changes to a CodeCommit repository
  2. Commits to a CodeCommit repository trigger an execution of the pipelines in CodePipeline
  3. CodeBuild builds and packages the artifacts
  4. The artifacts are stored in S3 for use in subsequent stages (build once, deploy many)
  5. A deployment stage assumes a role in the Dev account, which allows CodePipeline to create/update stacks in CloudFormation
  6. A deployment stage assumes a role in the Test account, which allows CodePipeline to create/update stacks in CloudFormation
  7. Once testing has been completed and the appropriate Change Management process is followed, the deployment to UAT is approved.
  8. A deployment stage assumes a role in the UAT account, which allows CodePipeline to create/update stacks in CloudFormation
  9. Once testing has been completed and the appropriate Change Management process is followed, the deployment to Production is approved.
  10. The final deployment stage assumes a role in the Production account, which allows CodePipeline to create/update stacks in CloudFormation.

The common resources created in the Tools account, and which are used by all AWS CodePipeline pipelines, are:

  • A KMS key for encrypting build artifacts
  • The S3 bucket for storing build artefacts and the associated bucket policy
  • An IAM role for AWS CodePipeline
  • An IAM role for CodeBuild projects
  • SSM Parameters for the resource ARNs and role names, which a referenced from the CloudFormation templates using Dynamic References.

A CloudFormation templates repository was also created, containing a series of examples and baselines for the creation of pipelines. It includes for example a standard Configuration for CloudFormation-based workloads supporting dependencies such as “Include” Snippets, Nested Stacks and Lambda Functions, that in most cases can be used to create pipelines without any customisation.

Additional functionality in the examples and baselines include:

  • Support for using a single repository for both the workload and pipeline configuration, to reduce the proliferation of repositories
  • Self-updating of the pipeline from changes in pipeline configuration to the repository, including the ability to bootstrap from a basic configuration
  • Support for separation into two pipelines set up from the same configuration files: a Development pipeline driven from the “develop” branch and a main pipeline driven from the “main” branch, triggered after merging of an approved Pull Request
  • Support for multiple deployed instances of the workload for an environment, with pipelines for each set up from the same configuration files
  • Support for validation steps such as cfn-lint and cfn-nag for CloudFormation workloads
  • Support for deployment to CodeArtifact.

There are two main roles that are required in each workload account:

  1. The CrossAccountPipelineRole, which allows CodePipeline to assume a role in the target account and then a) download the build artifact from S3, b) Pass a deployment role to CloudFormation and c) Create or update a CloudFormation stack
  2. The CrossAccountDeploymentRole, which is the role CloudFormation uses to create/update the stack.

These roles are deployed via CloudFormation Stack Sets in the Management account to all workload accounts in the AWS Organization, enabling AWS CodePipeline to be used for any client AWS account, and is crucial in allow the solution to be scaled out across workloads.

The first workloads deployed using AWS CodePipeline have used a standard template for deploying CloudFormation stacks and uses the following stages (later workloads uses a standard template including a Validate stage):

  • Source
  • Build
  • DeployToDev
  • DeployToTest
  • ApproveDeployToUat
  • DeployToUat
  • ApproveDeployToProd
  • DeployToProd

On commit to the configured branch in the repository, the Source, Build, DeployToDev and DeployToTest stages are automatically executed:

UAT and Production deployments are gated behind Manual Approval, with Review and Approve steps:

The use of an end-to-end pipeline, which is capable of deploying to controlled environments (albeit with approval gates) means that additional controls need to be added to the CodeCommit repository and main branch to guard against accidental commits.

Two mechanisms are used to achieve this in the solution:

  1. Limiting pushes and merges to the main branch using an IAM policy, which is applied to standardised IAM roles:
{
	"Version": "2012-10-17",
	"Statement": [
		{
		   "Effect": "Deny",
		   "Action": [
				"codecommit:GitPush",
				"codecommit:DeleteBranch",
				"codecommit:PutFile",
				"codecommit:MergeBranchesByFastForward",
				"codecommit:MergeBranchesBySquash",
				"codecommit:MergeBranchesByThreeWay"
			],
			"Resource": "*",
			"Condition": {
				"StringEqualsIfExists": {
					"codecommit:References": [
						"refs/heads/main",
                       				"refs/heads/master"
					 ]
				},
				"Null": {
					"codecommit:References": "false"
				}
			}
		}
	]
}
  1. The addition of Approval templates to AWS CodeCommit repositories to ensure that one or more approvals are required prior to merging a Pull Request

With this new solution, workloads are able to be continuously integrated and automatically deployed to lower environments for testing, with deployments to controlled environments integrating with the client’s Change Management process.

The focus on scalability and reusability for new and existing workloads has meant that delivery pipelines can easily be adopted using standard templates and further customised to meet the needs of each workload, lowering the barrier for entry for adopting CI/CD best practices.

Most importantly, teams have a fast, reliable and repeatable way to deliver workloads to Production, reducing the time and complexity of changes to controlled environments.