Let’s review the following Git workflow:
main
branch into their feature branch.dev
branch, and push to remote.main
.main
branch can be deployed to production environment directly after the merge.From the above Git workflow, we can conclude that a Pull Request is a crucial point for testing and review code changes before they are merged into the main
branch, and potentially deployed to production systems.
Thus, it’s common practice to let Jenkins perform an extensive testing on an opened Pull Requests.
So far we’ve seen how pipeline can be built and run around a single Git branch (e.g. main
or dev
).
Now we would like to create a new pipeline which will be triggered on every PR that is created in GitHub.
For that we will utilize Jenkins multi-branch pipeline.
main
branch, create the pr-testing.Jenkinsfile
as follows:
pipeline {
agent any
stages {
stage('Unittest') {
steps {
echo "testing"
}
}
stage('Lint') {
steps {
echo "linting"
}
}
stage('Functional test') {
steps {
echo "testing"
}
}
}
}
pr-testing
.main
create a new branch called sample_feature
and change some code lines. Push the branch to remote.sample_feature
into main
.main
branchWe also would like to protect the main
branch from being merged by non-tested branch.
main
branch as follows:
continuous-integration/jenkins/pr-merge
check done by Jenkins.Your main
branch is now protected and no code can be merged to it unless the PR is reviewed by other team member and passed all automatic tests done by Jenkins.
Automated testing is a very broad topic. In this section we will lightly cover 3 types of testing: unittests, code linting, functional testing. But the are many more…
Unit testing is a software testing technique where individual components of code are tested in isolation to ensure they behave as expected, helping to identify and fix bugs before the code is built.
In Python, unittest
is a testing framework that allows developers to write and run unit tests.
In the roberta
directory, you are given directory called tests
. This is a common name for the directory containing all unittests files. The directory contains a file called test_cache.py
which implements unittest for the caching functionality of the app (the server caches the recent requested texts and their results).
Run the unittest locally, check that all tests are passed:
cd roberta
python3 -m pytest --junitxml results.xml tests
Note: Usually the unittest
package is used to execute unit tests in Python, but here we used the pytest
package (has to be installed by pip install pytest
). We did it because pytest
allows to export the test results in a .xml
file format that can be presented by Jenkins in a human friendly format.
Integrate the unittesting in the pr-testing.Jenkinsfile
under the Unittest stage.
You can add the following post
step to display the tests results in the readable form:
post {
always {
junit allowEmptyResults: true, testResults: 'results.xml'
}
}
Pylint is a static code analyser for Python. Pylint analyzes your code without actually running it. It checks for syntax errors, enforces a coding standard, and can make suggestions about how the code could be refactored.
pylint
using pip
..pylintrc
file which is a configuration file used to customize and enforce coding standards, styles, and static analysis checks.pylint --generate-rcfile > .pylintrc
python3 -m pylint *.py
How many warnings do you get? If you need to ignore some unuseful warning, add it to disable
list in the [MESSAGES CONTROL]
section in your .pylintrc
file.
pr-testing.Jenkinsfile
under the Lint stage.steps {
sh 'python3 -m pylint -f parseable --reports=no *.py > pylint.log'
}
post {
always {
sh 'cat pylint.log'
recordIssues (
enabledForFailure: true,
aggregatingResults: true,
tools: [pyLint(name: 'Pylint', pattern: '**/pylint.log')]
)
}
}
Functional testing evaluates whether the app works as intended. It involves testing the application against some pre-defined scenario (usually the common scenario of your customers), and ensure that the software meets its functional objectives.
Implement functional testing in the pr-testing.Jenkinsfile
under the Functional test stage.
# Sentence: Intel is happy and excited to launch the new generation of processors
# Expect positive labels: optimism, excitement
curl localhost:8081/analyze?text=Intel%20is%20happy%20and%20excited%20to%20launch%20the%20new%20generation%20of%20processors
# Sentence: This is terrible movie
# Expect negative labels: disappointment, disgust, fear
curl localhost:8081/analyze?text=This%20is%20terrible%20movie.
# Sentence: This is terrible movie
# Expect neutral labels: neutral
curl localhost:8081/analyze?text=This%20is%20a%20neutral%20statement
Note: In general, running services within the Jenkins agent, as we did in the above testing, is not a good idea. It’s better to deploy the application under test (AUT) in test environments.
Use parallel
directive to run the test stages in parallel, while failing the whole build when one of the stages is failed.
While Snyk scans vulnerabilities in the container OS level, it’s also necessary to scan your Python code and the dependencies you use.
Integrate safety and bandit into the pr-testing.Jenkinsfile
pipeline.