Dynamic Pipeline Orchestration using CodePipeline, CodeBuild and Step Functions

Introduction

AWS CodePipeline helps in orchestrating CI/CD pipeline all the way from building and testing to deploying applications, from development to production environments.

However, one of the limitations of AWS CodePipeline is its ina…



Introduction

AWS CodePipeline helps in orchestrating CI/CD pipeline all the way from building and testing to deploying applications, from development to production environments.

However, one of the limitations of AWS CodePipeline is its inability to dynamically execute or skip any pipeline action.

This post demonstrates how to setup a CI/CD pipeline to dynamically orchestrate and execute actions based on inputs



AWS Services

This solution uses the following AWS services:

  • AWS CodeCommit – Source Control
  • AWS CodeBuild – Build System
  • AWS CodePipeline – Continuous Delivery
  • AWS Step Functions – Workflow automation using state machines



Solution Overview

  1. A change or commit to the code in the CodeCommit application repository triggers CodePipeline with the help of a CloudWatch event.
  2. The pipeline downloads the code from the CodeCommit repository, initiates the first build using CodeBuild and securely saves the built artifact to an S3 bucket.
  3. Once build succeeds, pipeline triggers a Step Function Workflow
  4. Step Function workflow uses artifact from previous build as input and iterates for each item in the input
  5. For each input, it triggers respective CodeBuild or Pass state using Choice state
  6. Step Functions will succeed once all invoked builds suceed



Architecure

CodePipeline CodeBuild StepFunctions Orchestration



Steps



CodeCommit

  1. Created a repo and have a file “sample_source.json” with below content

{"services":["service_A", "service_B"]}



CodeBuild



First CodeBuild

  • It copies source file and generates ABC.json which is passed as output artifact and consumed by next Step Function
  • This is optional when codebuild can generate a similar json file with list of services to be passed as input to step functions
  • BuildSpec



ServiceA and ServiceB CodeBuild



Step Functions

State Machine

State Machine json

  • The artifact passed as json file will be received as json input by step functions state machine
  • It receives the input using a “Map” state which supprts iteration
  • Within iterator, “Choice” state is used to validate the values and invoke respective “Task” or “Pass” state
  • The “Task” state in our scenario will invoke a CodeBuild task using “Optimized Integrations”
  • CodeBuild task supports only sync invocation which means the next state following CodeBuild execution has to wait until it finishes.
  • However, since the iterator uses a “Choice” state, each iteration is independent and happens in parallel
  • In this example, for “ServiceA” and “ServiceB”, CodeBuild is executed and “ServiceC” is set as a “pass” state



Note: Ensure enough permissions are granted for StepFunctions IAM role to make relevant service invocations

StepFunction IAM permissions for CodeBuild



CodePipeline

CodePipeline Definition

CodePipeline Definition

  • Once all the above steps are done, all we need to do is to create pipeline in CodePipeline
  • Create Source stage using any SCM, I used CodeCommit
  • Create a Build stage using existing “First Codebuild” build project
  • Create next “Step Sunctions” stage with following params
    • Action Provider: Step Functions
    • Input artifacts: Output Artifact from previous “First CodeBuild”
    • State machine ARN: Choose the recently created stateMachine ARN
    • Input type: File Path
    • Input: ABC.json (use whatever file is sent in output artifact from previous build)



Execution

  • Release a change to test the pipeline
  • In this example, if services list changes in “sample_source.json”, step functions will execute only those states



Conclusion

  • This solution demonstrated the invocation of Step Functions from CodePipeline and Step Functions execution executing builds in CodeBuild dynamically based upon the received input

Code Repository

My LinkedIn


Print Share Comment Cite Upload Translate
APA
omuthu | Sciencx (2024-03-28T09:50:03+00:00) » Dynamic Pipeline Orchestration using CodePipeline, CodeBuild and Step Functions. Retrieved from https://www.scien.cx/2021/11/25/dynamic-pipeline-orchestration-using-codepipeline-codebuild-and-step-functions/.
MLA
" » Dynamic Pipeline Orchestration using CodePipeline, CodeBuild and Step Functions." omuthu | Sciencx - Thursday November 25, 2021, https://www.scien.cx/2021/11/25/dynamic-pipeline-orchestration-using-codepipeline-codebuild-and-step-functions/
HARVARD
omuthu | Sciencx Thursday November 25, 2021 » Dynamic Pipeline Orchestration using CodePipeline, CodeBuild and Step Functions., viewed 2024-03-28T09:50:03+00:00,<https://www.scien.cx/2021/11/25/dynamic-pipeline-orchestration-using-codepipeline-codebuild-and-step-functions/>
VANCOUVER
omuthu | Sciencx - » Dynamic Pipeline Orchestration using CodePipeline, CodeBuild and Step Functions. [Internet]. [Accessed 2024-03-28T09:50:03+00:00]. Available from: https://www.scien.cx/2021/11/25/dynamic-pipeline-orchestration-using-codepipeline-codebuild-and-step-functions/
CHICAGO
" » Dynamic Pipeline Orchestration using CodePipeline, CodeBuild and Step Functions." omuthu | Sciencx - Accessed 2024-03-28T09:50:03+00:00. https://www.scien.cx/2021/11/25/dynamic-pipeline-orchestration-using-codepipeline-codebuild-and-step-functions/
IEEE
" » Dynamic Pipeline Orchestration using CodePipeline, CodeBuild and Step Functions." omuthu | Sciencx [Online]. Available: https://www.scien.cx/2021/11/25/dynamic-pipeline-orchestration-using-codepipeline-codebuild-and-step-functions/. [Accessed: 2024-03-28T09:50:03+00:00]
rf:citation
» Dynamic Pipeline Orchestration using CodePipeline, CodeBuild and Step Functions | omuthu | Sciencx | https://www.scien.cx/2021/11/25/dynamic-pipeline-orchestration-using-codepipeline-codebuild-and-step-functions/ | 2024-03-28T09:50:03+00:00
https://github.com/addpipe/simple-recorderjs-demo