Architecture for using codeBeamer managed Git repositories as source for AWS CodePipeline

The initiative for this architecture is to use third-party git repositories to use as a source for CodePipeline. In this case, the third-party git repository is codebeamer's managed git repository.

To set up the solution, you complete the following steps:

1. Launch the AWS CloudFormation stack to provision resources.

2. Deploy a sample CodePipeline and test the custom action type.

Note : For implementing the steps, you can use CloudShell or you can launch an ec2 instance. Cloudshell is preferable

• Go to the codebeamer web application running in the ec2 instance and navigate to the SCM repositories tab.

• If you haven't give any public ssh keys before, the ssh option in clone and push will be striked out, which will be seen alike to the image below

1. Clone this repository. The repository contains all cloudformation templates needed. These resources help uplift the logic for connecting to the Git repository, which for this post is codeBeamer managed git repository.

2. First launch the cloudformation stack that creates private vpc and subnets that can be used to deploy codebuild and lambda in that private vpc

   Click on the link below to create the stack. Open link in new tab for more convenience

   codepipeline-codebeamer-integration-pvt-vpc

   This will create a stack named "codepipeline-codebeamer-integration-pvt-vpc". Wait till the creation of stack.

   After the creation of stack, note down the values of pubPrivateVPCID, privateSubnet1ID and privateSubnet2ID from the outputs section of the stack.

   

3. Upload the Lambda function code(lambda_function.py) to any S3 bucket in the same Region where the stack is being deployed. To create a new S3 bucket, use the following code or create it via AWS management console(make sure to provide a unique name):

export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export S3_BUCKET_NAME=codepipeline-git-custom-action-${ACCOUNT_ID} 
aws s3 mb s3://${S3_BUCKET_NAME} --region <your region>

Then zip the contents of the function and upload to the S3 bucket (substitute the appropriate bucket name):

export ZIP_FILE_NAME="codepipeline_git.zip"
export S3_BUCKET_NAME="codepipeline-codebeamer-integration-canada-2054864"
zip -jr ${ZIP_FILE_NAME} ./lambda_function.py && \
aws s3 cp codepipeline_git.zip \
s3://${S3_BUCKET_NAME}/${ZIP_FILE_NAME}

Check whether the file is uploaded to the bucket

4. Run the following AWS CLI command to deploy the third-party Git source solution stack:

   Change the parameters according to the resources created by you.

    export vpcId="vpc-123456789"
    export subnetId1="subnet-12345"
    export subnetId2="subnet-54321"
    export GIT_SOURCE_STACK_NAME="demo-thirdparty-codepipeline-git-source"
    aws cloudformation create-stack \
    --stack-name ${GIT_SOURCE_STACK_NAME} \
    --template-body file://$(pwd)/third_party_git_custom_action.yaml \
    --parameters ParameterKey=SourceActionVersion,ParameterValue=1 \
    ParameterKey=SourceActionProvider,ParameterValue=CustomSourceForCB \
    ParameterKey=GitPullLambdaSubnet,ParameterValue=${subnetId1}\\,${subnetId2} \
    ParameterKey=GitPullLambdaVpc,ParameterValue=${vpcId} \
    ParameterKey=LambdaCodeS3Bucket,ParameterValue=${S3_BUCKET_NAME} \
    ParameterKey=LambdaCodeS3Key,ParameterValue=${ZIP_FILE_NAME} \
    --capabilities CAPABILITY_IAM

Wait until the stack is being created.

A custom source action type is now available on the account where you deployed the stack. You can check this on the CodePipeline console by attempting to create a new pipeline. You can see your source type listed under Source provider.

We now deploy a sample pipeline and test the custom action type using the template demo_sample_pipeline_custom.yml

You can run the following AWS CLI command to deploy the CloudFormation stack:

Note: Please provide the codeBeamer's Git repository http clone url to HTTP_URL environment variable that you have access to, remove the "http://" and change the ip in http url to the public DNS value

For example Original http url : http://ip-172-31-39-39.sa-east-1.compute.internal:8080/cb/git/sdv-demo-1-repo .Change it to ec2-15-228-168-130.sa-east-1.compute.amazonaws.com:8080/cb/git/sdv-demo-1-repo

Change the GitWebHookIpAddress parameter to the value of IPv4 CIDR range of your codeBeamer ec2 instance. You can get the value by selecting your codebeamer instance in management console. In details tab, scroll down to get subnet ID and click that. The subnet details will be opened in the new tab

Copy the IPv4 CIDR range from the subnet details

Change the CodebeamerUsername & CodebeamerPassword parameters accordingly

Run the following AWS CLI command to deploy the stack

export HTTP_URL="ec2-15-228-168-130.sa-east-1.compute.amazonaws.com:8080/cb/git/sdv-demo-1-repo"
export SAMPLE_STACK_NAME="third-party-codepipeline-git-source-test"
export CodebeamerUsername="<your codebeamer username>"
export CodebeamerPassword="<your codebeamer password>"
aws cloudformation create-stack \
--stack-name ${SAMPLE_STACK_NAME} \
--template-body file://${pwd}/demo_sample_pipeline_custom.yaml \
--parameters ParameterKey=Branch,ParameterValue=main \
ParameterKey=GitUrl,ParameterValue=${HTTP_URL} \
ParameterKey=SourceActionVersion,ParameterValue=1 \
ParameterKey=SourceActionProvider,ParameterValue=CustomSourceForCB \
ParameterKey=CodePipelineName,ParameterValue=sampleCodePipeline \
ParameterKey=CodebeamerUsername,ParameterValue=${CodebeamerUsername} \
ParameterKey=CodebeamerPassword,ParameterValue=${CodebeamerPassword} \
ParameterKey=GitWebHookIpAddress,ParameterValue=172.31.32.0/20 \
--capabilities CAPABILITY_IAM

The pipeline will be executed after the creation of stack. It invokes the codebuild that pulls the repository from our codebeamer webapp

Now we are going to make the codepipeline to trigger automatically whenever the we push modified code to the codebeamer's managed repo

We need to set post receive git hook to trigger the codepipeline event. For that we need to ssh into the codebeamer's instance.

In this blog, I am gonna use VS Code as the IDE for modifying codebeamer's managed git repository. You can use whichever IDE you prefer.

Note : If you are planning to pull the repo residing in codebeamer, commit changes and push using VS Code, you should have git installed in your local computer.

SSH into the codebeamer instance and change user to "codebeamer" with the help of the below command and give your password

su codebeamer

Now run the command to get into codebeamer home directory and list the files and folders present in it

cd ~/
ls

You should see a folder named CB or CB-***linux.bin or both, it depends on your codebeamer installation. If you have CB, move inside repo or use CB-***linux.bin. Both are same

Run the below command

cd CB/repository/git/
ls

Inside this git repo, you should see your managed git repository residing. Create a log file here. We are going to use this log file for debugging the hook if anything goes wrong

touch post-receive-log.log

Now move inside the repository directory for which you wish to create hook and trigger pipeline and move to hook folder. Here it is sdv-demo-1-repo-fork-2

cd sdv-demo-1-repo-fork-2/.git/hooks/

If you list the objects, you should see a file called post-receive. That file is the hook file

Open the post-receive file and change the contents as given below, You can use nano editor

nano post-receive

#!/bin/bash

WEBHOOK_URL="<YOUR CODEPIPELINE WEBHOOK HERE>"
LOG_FILE="/home/codebeamer/CB/repository/git/post-receive-log.log"

# Extract the branch or commit information from the hook event
#branch=$(git rev-parse --symbolic --abbrev-ref $1)
#commit=$(git rev-parse $2)

# Create the JSON payload with the branch and commit details
payload="{\"ref\":\"$branch\",\"commit\":\"$commit\"}"

# Get the branch name and commit ID from the standard input
read old_rev new_rev ref_name

# Extract the branch name from the ref_name
branch_name=$(basename "$ref_name")

# Only trigger the pipeline for a specific branch (optional)
if [[ $branch_name != "main" ]]; then
  echo "Skipping pipeline trigger for branch $branch_name"
  exit 0
fi

# Redirect both standard output and standard error to the log file
exec >> $LOG_FILE 2>&1

# Trigger the CodePipeline using cURL with the webhook URL
response=$(curl -sSL -w "%{http_code}" -o /dev/null -X POST -H "Content-Type: application/json" -d "$payload" $WEBHOOK_URL 2>&1)

echo "Webhook URL: $WEBHOOK_URL"
echo "Response: $response"

if [[ "$response" == "200" ]]; then
  echo "CodePipeline trigger request sent successfully"
else
  echo "Failed to send CodePipeline trigger request. HTTP response code: $response"
fi

In the WEBHOOK_URL parameter, change it to your codepipeline webhook url that you have already saved (You can find it in the cloudformation template output)

If you have the log file in any other place and has a different name, change that too in LOG_FILE parameter.

Save and close the file.

Here all the steps are done in VS Code.

If you prefer terminal, you can do it. For help, click here

Open the VS Code -> navigate to source control -> click Clone repository (See the picture below)

Now clone the repository by providing the url in the prompt above and open it once cloned. The URL should be,

http://<CodebeamerUsername>:<CodebeamerPassword>@<codeBeamer instance's public IPv4 DNS>:8080/cb/git/<repository name>

Open the desired file, make changes and save it. Here I am changing the README.md file. After saving, you should see the notification in source control icon. Click that icon

You can see the changes done to your file listed there.

You can view what exactly you have changed by clicking the "M" icon under "Changes"

When you are ready for commit, click three dots under source code control -> Commit -> Commit all (Refer the picture below)

Give "Yes" if you are prompted like the below image

Give a commit message in the prompt above and hit enter

Now click "Sync Changes" under source control

You will be asked whether to push and pull the branch, click "Yes" for that

Check whether the change has been pushed to your codebeamer project by going to the web app -> your project -> SCM repository -> Your repository -> Changes

If the push is successfully accomplished, check whether the codepipeline is triggered

The pipeline should show the status In Progress.

After few minutes, it changes to Succeeded status and you see the Git Commit message on the source stage.

You can also view the Git Commit message by choosing the execution ID of the pipeline, navigating to the timeline section, and choosing the source action. You should see the Commit message and Commit ID that correlates with the Git repository.

If the CodePipeline fails, check the Lambda function logs for the function.

If the Lambda logs has the CodeBuild build ID, then check the CodeBuild run logs for that build ID for errors.

1. Delete the CloudFormation stacks that you created.

2. Additionally, empty the S3 bucket and delete it.