Difference between revisions of "Continuous Integration: Releases Jenkins Pipeline"
Manuele.simi (Talk | contribs) (→Build Commits Report) |
Manuele.simi (Talk | contribs) (→Build Commits Report) |
||
Line 247: | Line 247: | ||
Here's an example of build commits report: | Here's an example of build commits report: | ||
− | [[File: | + | [[File:Jenkins-build-commits-report.png|600px]] |
Revision as of 02:07, 26 December 2019
Contents
gCubeRelease Pipeline Project
In gCube we use a Pipeline to trigger the builds of jobs forming a gCube Release. The pipeline project is available at: https://jenkins.d4science.org/job/gCube-Release/
Parameters
Triggers
No triggers are defined because the pipeline is expected to be manually launched by the Release Manager:
It can be changed according to the release needs and the availability of a sufficient number of dedicate agents in Jenkins.
Git
The pipeline definition is maintained in a Git repository. This section connects the project to the Git repository.
Jenkins Pipeline Definition
Git Repository
The definition of the gCube release pipeline is maintained in this Git Repository: https://code-repo.d4science.org/gCubeCI/gCubeRelease
Requirements on Jenkins
- Jenkins ver. 2.164.2 or newer
- Pipeline Plugin
- Pipeline: Basic Steps
- Pipeline: Maven
- Kubernetes Plugin (for the YAML parser)
- NodeLabelParameter Plugins (for the Label's job parameter)
- Jenkins configured with a JDK named 'OpenJDK 8'
- One or more Jenkins agents labeled as 'pipeline-agent'
- a global variable named MAVEN_CONFIG_FOLDER that points to the base folder to use for the maven local repositories
Requirements on Downstream Jobs
In order to customize the build environment, the pipeline requires that each downstream job is configured with the following 3 parameters:
- gcube_settings (type: String Parameter): the filename of the settings.xml for maven
- local_repo (type: String Parameter): the name of the folder to use as maven local repository
- exec_label (type: Label): the label assigned to the nodes to use for the executions
Here's a screenshot for the parameters and their default values:
TBP
Also, the build command of each job must be the following:
--settings $MAVEN_CONFIG_FOLDER/$gcube_settings -Dmaven.repo.local=$MAVEN_CONFIG_FOLDER/$local_repo dependency:tree deploy
Basic Structure
Jenkinsfile defines a Declarative Pipeline with the Groovy DSL. This is the typical structure of a declarative pipeline:
The Pipeline’s code (written in the Groovy language) models the entire build process of a gCube Release.
This is the stub of the gCubeRelease pipeline available in Git.
import org.yaml.snakeyaml.Yaml // manage options and settings here (not shown) // load and parse the release file String releaseURL = "<git repo url>/releases/gcube-${gCube_release_version}.yaml" def text = releaseURL.toURL().getText() def jsonConfig = new Yaml().load(text) // pipeline pipeline { // run only on agents with the label agent { label 'CD' } // expected input parameters parameters { choice(name: 'Type', choices: ['SNAPSHOT-DRY-RUN', 'SNAPSHOT', 'RELEASE-DRY-RUN', 'RELEASE-STAGING', 'RELEASE'], description: 'The type of artifacts the build is expected to generate') string(name: 'gCube_release_version', defaultValue: 'x.y.z', description: 'The number of the gCube release to build. Sample values: 4.14.1, 4.15, etc.') } stages { stage('preliminary steps') { //prepare the environment for the builds steps { //execute steps here if needed } } // the maven-parent needs to be built (once) at each execution stage('build maven-parent') { steps { echo build(job: 'maven-parent', wait: true, parameters: [[$class: 'StringParameterValue', name: 'gcube_settings', value: "${maven_settings_file}"], [$class: 'StringParameterValue', name: 'local_repo', value: "${maven_local_repo_path}"], [$class: 'LabelParameterValue', name: 'exec_label', label: "CD", nodeEligibility: [$class: 'AllNodeEligibility']] ]).result echo "Done with maven-parent" } } stage('build components') { steps { script { jsonConfig.gCube_release.Components.each { group_name, component_list -> stage("build ${group_name} components") { buildComponents items: component_list?.collect { "${it.name}" }, "${maven_settings_file}", "${maven_local_repo_path}" echo "Done with ${group_name} components" } } } } } } // post-build actions post { always { echo 'This will always run' } success { echo 'This will run only if successful' } failure { echo 'This will run only if failed' } unstable { echo 'This will run only if the run was marked as unstable' } changed { echo 'This will run only if the state of the Pipeline has changed' echo 'For example, if the Pipeline was previously failing but is now successful' } } } def buildComponents(args, maven_settings_file, maven_local_repo_path) { if (args.items) { parallel args.items?.collectEntries { name -> ["${name}": { if (name && !"NONE".equalsIgnoreCase(name)) build(job: name, parameters: [[$class: 'StringParameterValue', name: 'gcube_settings', value: "${maven_settings_file}"], [$class: 'StringParameterValue', name: 'local_repo', value: "${maven_local_repo_path}"], [$class: 'LabelParameterValue', name: 'exec_label', label: "CD", nodeEligibility: [$class: 'AllNodeEligibility']] ]) } ] } } }
Reference Documentation
Jenkins Pipeline Configuration
Each gCube release requires a configuration file (called release file) written in the YAML format.
The file must be placed in the /release folder of the Git repository and named as gcube-<version>.yaml (e.g. gcube-4.14.5.yaml). The file must report the gcube release version and the list of Jenkins jobs to build grouped in logical groups.
This is a sample release file with 3 groups (SmartGears, Enabling, Data):
gCube_release: Version: 4.14.5 Components: SmartGear: - name: maven-parent version: 1.1.0 - name: gcube-bom version: 1.4.0 - name: maven-smartgears-bom version: 1.1.0 - name: authorization-client version: 1.1.0 - name: gxRest version: 1.1.2 Enabling: - name: information-system-bom version: 1.1.0 - name: information-system-model version: 1.1.0 - name: resource-registry-api version: 1.1.0 - name: resource-registry-client version: 1.1.0 Data: - NONE
Do note that if a logical group is added, a correspondent stage must be added to the pipeline definition.
Executing the release pipeline with the previous release file results in something like the following on the Jenkins interface:
Build Flow
There are two levels of execution in the gCube pipeline:
- stages: stages are executed sequentially, one after the other;
- steps inside each state: steps belonging to the same stage are executed in parallel.
These two levels are mapped in the pipeline configuration as follows:
- stages correspond to the logical groups
- steps execute the list of jobs inside each group.
For instance, in the following pseudo-config:
group A | |-job1 |-job2 |-job3 |-job4 group B | |-job5 |-job6 |-job7
- A is executed before B
- job1,job2,job3,job4 are executed in parallel
- job5,job6,job7 are executed in parallel if all the jobs in A succeed (do not fail or become unstable).
Build Commits Report
If the pipeline execution succeeds, it sends to the release manager a build commits report.
Here's an example of build commits report:
Back to the CI guide.