"Automating CI/CD with Multi-Branch Jenkins Pipelines, Git, Docker, and Kubernetes"
"A Step-by-Step Approach to Streamlining Development and Deployment with Scalable Automation Tools"
✨ Continuous Integration and Continuous Deployment (CI/CD) pipelines are essential in modern software development to ensure the seamless delivery of high-quality applications. 🚀 This project leverages Jenkins Multi-Branch Pipelines to automate the entire CI/CD process for a scalable web application. 🌐
The setup handles multiple stages for feature branches and pull requests, ensuring strict code quality standards. ✅ Once the code passes quality checks, the main branch pipeline automates versioning, containerization, and deployment. 🛠️
I will directly jump into what this project will do. 🎯 It focuses on automating the entire development workflow with two pipelines:
1️⃣ One for checking the quality of the code
2️⃣ Another for deploying code changes to the production environment
These pipelines will:
Trigger on pull requests and merged changes 🔄
Handle versioning with semantic versioning and conventional commits 📦
Deploy using Docker and Kubernetes 🐳☸️
Key Workflow Steps:
Code Quality Pipeline:
Triggered by pull requests, this pipeline:
Checks out the code.
Installs dependencies with Poetry.
Runs tests to ensure no new issues.
Release Pipeline:
Triggered by merges to the main branch, this pipeline:
Manages versioning with semantic versioning.
Builds and tags Docker images.
Deploys to Kubernetes, ensuring seamless updates.
This CI/CD process automates checks and deployments, improving efficiency and reducing errors.
Tools and Setup:
Here's a quick look at the tools and setup I used to get this project up and running:
Jenkins: The backbone of the pipeline, automating everything from code quality checks to deployment.Docker: Containerized the Flask app, so it runs consistently no matter where it’s deployed.Docker Hub: Stored the Docker images, making them easy to pull and deploy.EKS (Elastic Kubernetes Service): Handled the deployment and scaling of the app on AWS.AWS: Provided all the cloud resources, like EC2 instances, to keep things running.GitHub: Managed the source code and triggered the pipeline with pull requests and merges.Poetry: Took care of Python dependencies, so the app always had the right packages.Git: Used for version control and tracking all code changes.VS Code: My go-to code editor for writing and debugging the app.
This setup made the whole process smooth, from writing code to deploying it, and kept everything organized and efficient.
Open up your Jenkins server and set up a new item.
"Give the item a name, select a folder to organize your jobs, and click 'ok'."
"Give it a display name and Save."
"Now that we've created the Advanced project folder, let's move on to creating our first Code Quality pipeline within that folder."
"Click on 'New Item', give it a name, select 'Multibranch Pipeline', and then click 'OK'"
"Now, configure the pipeline. Make sure to add your GitHub credentials and provide the repository URL."
"Next, configure branch discovery by selecting the strategy to exclude branches filed as PRs, then set up pull request discovery for both origin and forks with the appropriate strategies."
"Provide the Jenkinsfile path and set the pipeline to run at an interval of every 2 minutes and 'Save'. "You can adjust the interval as needed."
"Pipeline Execution: Jenkins will trigger the pipeline every 2 minutes, checking for updates or changes in the repository or branches."
"After the pipeline is triggered, the Code Quality Pipeline runs, checking the code for quality issues and ensuring that all tests pass. Our pipeline has successfully completed with no issues."
"Below is the Jenkinsfile for our Code Quality Pipeline.”
pipeline {
agent any
stages {
stage('environment variables') {
steps {
sh 'printenv'
sh 'ls -la'
}
}
stage('Pull Request Number') {
when {
changeRequest target: 'main'
}
steps {
echo "PR: ${CHANGE_ID}"
}
}
stage('Setup') {
steps {
sh '''
# Install Poetry and add to PATH
curl -sSL https://install.python-poetry.org | python3 -
export PATH="/var/lib/jenkins/.local/bin:$PATH"
poetry install --no-root
'''
}
}
stage('Test') {
steps {
sh '''
export PATH=/var/lib/jenkins/.local/bin:$PATH
poetry run pytest
'''
} }
}
}
This Jenkins pipeline automates a series of tasks to check and ensure the quality of code in a project. It works in a few stages:
- environment variables:
- Prints environment variables (printenv) and lists files in the directory (ls -la).
- Pull Request Number:
- Runs only when a pull request targets the main branch. It prints the pull request ID (CHANGE_ID).
- Setup:
- Installs Poetry (a Python package manager), updates the system’s PATH, and installs project dependencies with poetry install.
- Test:
- Runs the project’s tests using pytest via Poetry, ensuring the code works as expected.
Each stage serves a specific purpose, from checking the environment to running tests after setting up dependencies.
Modify the Code Quality Pipeline Configuration:
We will configure it to exclude the main branch from triggering the pipeline. This ensures that the pipeline only runs for feature branches or pull requests, and not for the main branch, which doesn't need to be checked repeatedly.
Push the Changes to the Remote Repository:
Once your changes are committed to the feature1 branch, use the git push origin feature1 command to push the branch to your remote GitHub repository. This ensures that your updates are available on GitHub and ready for review.
Jenkins will trigger the pipeline automatically
"If you go to GitHub, you will see that a Pull Request has been created for the feature1 branch. This pull request is now ready for review and can be merged into the main branch once approved."
After creating a pull request,If we visit our Jenkins server, we will notice that the pipeline is triggered automatically for the pull request. Jenkins will then proceed to execute the configured checks and tests for the changes in the pull request.
"After merging the pull request, the changes will be integrated into the main branch. Once the changes are successfully merged, our Release Pipeline will be triggered automatically to handle versioning, building, and deployment processes.
Now, let's configure our second Release Pipeline to automate this process for production deployment."
"Go to your Jenkins Dashboard and select New Item to create a new pipeline for the release process."
"Then, give your item a name (e.g., 'release '), select Pipeline as the project type, and click OK to proceed with the configuration."
"Next, configure the GitHub Hook Triggered option for GitSCM polling. Make sure you have already added the necessary GitHub Webhooks to enable Jenkins to automatically trigger the pipeline whenever changes are pushed to your repository."
"Then, provide your repository URL, add your GitHub credentials, and select the branch to monitor. In this case, choose the main branch."
"Now, expand the Additional Behaviour dropdown and select Advanced Clone Behaviour to configure additional cloning options."
"Tick the Fetch Tags option, set the Branch Name to main in this case, and provide the Script Path as Jenkinsfile-Release to ensure all relevant tags are fetched and the correct release pipeline script is used for your repository. and Hit Save"
"Before moving further, let me show you the configuration for my Release Pipeline."
pipeline { agent any environment { IMAGE_NAME = 'suresh53/flask-app' IMAGE_TAG = "${IMAGE_NAME}:${env.GIT_COMMIT}" KUBECONFIG = credentials('kubeconfig-cred') AWS_ACCESS_KEY_ID = credentials('aws-access-key') AWS_SECRET_ACCESS_KEY = credentials('aws-secret-key') GH_TOKEN = credentials('github') POETRY_HOME = "/var/lib/jenkins/.local" } stages { stage("Check for Git Tag") { steps { script { def tag = sh(returnStdout: true, script: "git tag --contains").trim() env.GIT_TAG = tag ?: '' echo "GIT_TAG is set to: ${env.GIT_TAG}" env.IMAGE_TAG_RELEASE = "${IMAGE_NAME}:${GIT_TAG}" } } } stage('Setup') { steps { script { sh ''' # Install Poetry curl -sSL https://install.python-poetry.org | POETRY_HOME=$POETRY_HOME python3 - # Add poetry to PATH and install dependencies export PATH="$POETRY_HOME/bin:$PATH" poetry install --no-root ''' } } } stage('Create Release') { when { expression { return env.GIT_TAG == "" } } steps { script { sh ''' # Ensure poetry is in PATH export PATH="$POETRY_HOME/bin:$PATH" # Debug: Check poetry installation echo "Poetry version:" poetry --version # Run semantic release poetry run semantic-release version poetry run semantic-release publish ''' } } } stage("Build and Deploy") { when { expression { return env.GIT_TAG != "" } } stages { stage('Docker Login') { steps { withCredentials([usernamePassword(credentialsId: 'DockerhubCred', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD')]) { sh 'echo ${PASSWORD} | docker login -u ${USERNAME} --password-stdin' echo 'Login successfully' } } } stage('Build') { steps { sh 'docker build -t ${IMAGE_TAG} -t ${IMAGE_TAG_RELEASE} .' echo "Docker image build successfully" sh 'docker image ls' } } stage('Push Image') { steps { sh 'docker push --all-tags ${IMAGE_NAME}' echo "Docker image push successfully" } } stage('Deploy to kubenrentie') { steps { sh ''' kubectl apply -f deployment.yaml kubectl apply -f service.yaml kubectl set image deployment/flask-app flask-app=${IMAGE_TAG} kubectl rollout status deployment/flask-app kubectl get deployment flask-app kubectl get service flask-app-service ''' } } } } } }Here's a simplified explanation of each step:
Set Environment Variables:- It defines variables for Docker image, Kubernetes configuration, AWS credentials, and GitHub token for accessing services.
Check for Git Tag:- This stage checks if there’s a Git tag associated with the commit. If a tag is found, it sets up an image tag accordingly.
Setup:- installs Poetry (a Python dependency manager), adds it to the system PATH, and installs project dependencies.
Create Release:- If there’s no Git tag, this step runs the semantic-release tool to generate and publish a new version.
Build and Deploy (if there is a Git tag):Docker Login: Logs into Docker Hub using stored credentials.
Build: Builds a Docker image using the application’s Dockerfile and tags it with the Git commit
and release tag.
Push Image:Pushes the built Docker image to Docker Hub.Deploy to Kubernetes: Applies Kubernetes configuration files and updates the app's image on the cluster to the newly built version.
Note:
Release Creation: If no Git tag is associated with the commit, the pipeline triggers the Create Release stage, where the semantic-release tool is used to automatically generate a new version and publish it.
Build and Deploy: If a Git tag is present, the pipeline will trigger the Build and Deploy stages. This includes logging into Docker Hub, building a Docker image, pushing the image to Docker Hub, and deploying the app to Kubernetes.
Ensure Semantic Versioning During Commit.
Make sure to follow semantic versioning when committing your code. This means:
Commit messages should include a version number in the format vMAJOR.MINOR.PATCH.
For each commit, update the version based on the changes made:
- MAJOR version when you make incompatible API changes.
2. MINOR version when you add functionality in a backward-compatible manner.
3. PATCH version for backward-compatible bug fixes. This ensures consistent versioning and helps automate the release process
"Push your code with a commit message. Here, we are pushing the code from the feature2 branch to GitHub."
"If you go to your Jenkins server, you will see that the feature2 branch has successfully built."
"Now, let's see what happens when we merge the pull request."
Release Pipeline Triggered: After merging the pull request, the release pipeline is automatically triggered and assigned version 0.1.2.
Build and Deploy Steps Begin:
The pipeline proceeds with building the Docker image and deploying the new version.
Docker Image Build: The new Docker image for version 0.1.2 is created.Push Image to Docker Hub: The image is pushed to Docker Hub.Deploy to Kubernetes: Kubernetes deployment is updated with the new image.Monitor Rollout Status: Ensures successful rollout of the new version.Deployment Confirmation: Verifies the deployment is successful.
This confirms that the application is live and successfully deployed with the latest version.
Source Code:
Reference :