UPES-CSDV3001

Jenkins pipelines

Jenkins Pipelines are used to define, manage, and automate the entire lifecycle of software delivery processes (a.k.a. CI/CD pipelines). These pipelines provide a structured way to define the steps and stages involved in building, testing, deploying, and delivering software applications.

Jenkins Pipelines enable teams to automate repetitive tasks, and ensure consistent and reliable software releases.

Motivation

Let’s take the Docker images build process as an example…

In our development workflow, whenever we wanted to create a new version of our application, we had to manually run the docker build command to create a Docker image. This process involved remembering the right set of build arguments, manually managing images, and we had no any mechanism to track the build history.

With Jenkins Pipelines, we can automate the build process, which makes it consistent and reproducible process. In addition, Jenkins provides a clear view of the build status, logs, and any errors encountered during the build.

Create your first pipeline

The Jenkinsfile

A Jenkins pipeline is defined in a file usually called Jenkinsfile, stored as part of the code repository. In this file you instruct Jenkins on how to build, test, and deploy your application by specifying a series of stages, steps, and configurations.

There are two main types of syntax for defining Jenkins pipelines in a Jenkinsfile: Declarative Syntax and Scripted Syntax.

The Jenkinsfile typically consists of multiple stages, each of which performs a specific steps, such as building the code as a Docker image, running tests, or deploying the software to Kubernetes cluster.

Let’s create a declarative pipeline that builds aa docker image for the Roberta app.

  1. In your repository, in branch main, create a file called build.Jenkinsfile in the root directory as the following template:
pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh 'ls'
                sh 'echo building...'
            }
        }
    }
}

The Jenkinsfile you’ve provided is written in Declarative Pipeline syntax. Let’s break down each part of the code:

  1. Commit and push your changes.

Configure the pipeline in Jenkins

  1. From the main Jenkins dashboard page, choose New Item.
  2. Enter the project name (e.g. RobertaBuild), and choose Pipeline.
  3. Check GitHub project and enter the URL of your GitHub repo.
  4. Under Build Triggers check GitHub hook trigger for GITScm polling.
  5. Under Pipeline choose Pipeline script from SCM.
  6. Choose Git as your SCM, and enter the repo URL.
  7. If you don’t have yet credentials to GitHub, choose Add and create Jenkins credentials.
    1. Kind must be Username and password
    2. Choose informative Username (as github or something similar)
    3. The Password should be a GitHub Personal Access Token with the following scope: 
      repo,read:user,user:email,write:repo_hook

      Click here to create a token with this scope.

    4. Enter github as the credentials ID.
    5. Click Add.
  8. Under Branches to build enter main as we want this pipeline to be triggered upon changes in branch main.
  9. Under Script Path write the path to your build.Jenkinsfile defining this pipeline.
  10. Save the pipeline.
  11. Test the integration by add a sh step to the build.Jenkinsfile, commit & push and see the triggered job.

Well done! You’ve implemented an automated build pipeline for the Roberta app.

Pipeline Execution

Let’s discuss the execution stages when your pipeline is running:

  1. Job scheduling: When you trigger a pipeline job, Jenkins schedules the job on one of its available agents (also known as nodes). Agents are the machines that actually execute the build steps. In our case, the pipline runs on the Jenkins server machine itself, known as the built-in node. This is very bad practice, and we will change it soon. Each agent can have one or more executors, which are worker threads responsible for running jobs concurrently.
  2. Workspace creation: Jenkins creates a workspace directory on the agent’s file system. This directory serves as the working area for the pipeline job.
  3. Checkout source code: Jenkins checks out the source code into the workspace.
  4. Pipeline execution: Jenkins executes your pipeline script step-by-step.

The Build phase

The Build phase builds the app source code and store the build artifact somewhere, make it ready to be deployed. In our case, a docker image is our build artifact, but in general, there are many other build tools that can be used in different programming languages and contexts (e.g. maven, npm, gradle, etc…)

Guidelines

We now want to complete the build.Jenkinsfile pipeline, such that on every run of this job, a new docker image of the app will be built and stored in container registry (DockerHub or ECR).

stage('Build') {
   steps {
       sh '''
            docker login ...
            docker build ...
            docker tag ...
            docker push ...
       '''
   }
}

The Deploy phase

Let’s create another new Jenkins pipeline that deploys the image we’ve just built to a Kubernetes cluster.

We would like to trigger the Deploy pipeline after every successful running of the Build pipeline.

  1. In the app repo, create another Jenkinsfile called deploy.Jenkinsfile. In this pipeline we will define the deployment steps for the roberta app: 
    pipeline {
 agent any
    
 stages {
     stage('Deploy') {
         steps {
             // complete this code to deploy to real k8s cluster
             sh '# kubectl apply -f ....'
         }
     }
 }
}
  2. In the Jenkins dashboard, create another Jenkins Pipeline (can be named RobertaDeploy), fill it similarly to the Build pipeline, but don’t trigger this pipeline as a result of a GitHub hook event (why?).

We now want that every successful Build pipeline running will automatically trigger the Deploy pipeline. We can achieve this using the following two steps:

  1. In build.Jenkinsfile, add the Pipeline: Build function that triggers the Deploy pipeline:
stage('Trigger Deploy') {
    steps {
        build job: '<deploy-job-name>', wait: false, parameters: [
            string(name: 'ROBERTA_IMAGE_URL', value: "<full-url-to-docker-image>")
        ]
    }
}

Where:

  1. In the deploy.Jenkinsfile define a string parameter that will be passed to this pipeline from the Build pipeline:
pipeline {
    agent ..
    
    # add the below line in the same level an `agent` and `stages`:
    parameters { string(name: 'ROBERTA_IMAGE_URL', defaultValue: '', description: '') }

    stages ...
}

Test your simple CI/CD pipeline end-to-end.

The Build and Deploy phases - overview

Self-check questions

Enter the interactive self-check page

Exercises

:pencil2: Clean the build artifacts from Jenkins server

Use the post directive and the docker image prune command to cleanup the built Docker images from the disk.

:pencil2: Fine tune the your pipelines

Review some additional pipeline features, as part of the options directive. Add the options{} clause with the relevant features for the Build and Deploy pipelines.

:pencil2: Clean the workspace after every build

Jenkins does not clean the workspace by default after a build. Jenkins retains the contents of the workspace between builds to improve performance by avoiding the need to re-fetch and recreate the entire workspace each time a build runs.

Cleaning the workspace can help ensure that no artifacts from previous builds interfere with the current build.

Configure stage('Clean Workspace') stage to clean the workspace before or after a build.

:pencil2: Security vulnerability scanning

Integrate snyk image vulnerability scanning into your build-deploy pipline.

Guidelines

snyk ignore --id=<ISSUE_ID>

:pencil2: Backup pipeline

Create a new Jenkins pipeline and code the corresponding Jenkinsfile, to periodically backup the Jenkins server in S3.

:pencil2: Shared libraries

https://www.jenkins.io/blog/2017/02/15/declarative-notifications/