Latest revision as of 11:22, 30 May 2017

Introduction

This document describes the procedures and tools used for the integration and release of Maven-based gCube components. After an initial overview of the overall solution, the document narrow its focus to the integration of Maven components in ETICS.

This document is also an operational guide for developers with all information needed to properly configure a Maven-based gCube component in ETICS to successfully integrate with other components in gCube. It provides also very quick-to-read checklist with the minimal set of things to know and to do in order to let things work in ETICS.

Glossary

• Maven component: a gCube component which uses Maven ad building tool
• Legacy component: a gCube component (most probably developed before iMarine project) which uses the classical building system based on ant
• Mavenizer: a tool developed within gCube project able to create a pom.xml file

gCube Maven Repositories

gCube Releases

• Contains artifacts officially released.
• URL: http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases/
• access: only release manager
• only release version, no re-deploy
• updated by etics as last step of the release process

gCube Staging

• Contains artifacts produced during the release cycle.
• URL: http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-staging/
• access: release manager, developers
• only release versions, redeploy allowed
• updated by:
  • etics nightly builds and on-demand builds for SmartGears release

gCube Staging gCore

• Contains artifacts produced during the release cycle.
• URL: http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-staging-gcore/
• access: release manager, developers
• only release versions, redeploy allowed
• updated by:
  • etics nightly builds and on-demand builds for gCore release

gCube Staging All

• Contains artifacts produced during the release cycle.
• URL: http://maven.research-infrastructures.eu/nexus/content/groups/gcube-staging-all/
• access: release manager, developers
• only release versions, redeploy allowed
• virtual repository that merges gCube Staging and gCube Staging gCore

gCube Snapshots

• Contains snapshot versions of of gCube components. It is used mainly during the development phase to make public and available for integration not yet released software
• URL: http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-snapshots/
• access: release manager, developers
• only snapshot versions, redeploy allowed
• updated by:
  • developers from their local environment
  • etics on-demand builds for HEAD configurations built within gcube.HEAD
  • etics daily builds of gcube.HEAD

gCube Externals

• Contains patched versions of third party components. Those components cannot be considered part of gCube system
• URL: http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-externals/
• access: release manager

Overall strategy

Figure below catches the main components, and their interactions, that are involved in the gCube' software integration, release and distribution.

The central element of the gCube release and distribution strategy is the gCube Releases Repository. It is a classical Maven repository where all released versions of all gCube components are stored.The gCube Releases Repository is the main distribution site for gCube Software:

• Developers configure their development machine to use gCube Releases to resolve and fetch dependencies (a) for gCube components their are developing.
• the Software Gateway service uses gCube Releases as back-end (among, potentially, other third party Maven repositories (c) ) making software stored there available for dynamic deployment within the D4Science Infrastructure.

Another central role is played by ETICS: the integration system used by gCube. It is responsible for the publication (d) in gCube Releases of artifacts that comes out from the integration of gCube components. In particular, actions performed by ETICS are:

• checkout (e), compilation and testing of gCube Maven components from the gCube' source code repository
• validation and certification of artifacts. This ensure that components published in gCube Releases satisfy the quality level defined in the gCube project
• mavenization of legacy components. This make also gCube components that builds with ant available for dependency resolution in Maven
• publication of artifacts in gCube Releases

An identical approach is used also for integration of development version of components. Main differences in this case are:

• gCube Snapshots is used instead of gCube Releases
• also Developers can commit a snapshot version of their component directly from their development machine
• development versions are never deployed in the D4Science production infrastructure (but they are used in the testing infrastructure)

Bringing Maven to ETICS: the maven-builder

ETICS builds take place on a remote machine on a "empty" environment. In order to properly build Maven components it is needed Maven to be installed and configured on the building machine. All those tasks are performed by an ETICS component created for this purpose called maven-builder.

Briefly, this is what maven-builder provides:

• downloads and install Maven tool on the building machine
• downloads and install the jdk and Ant tool on the building machine
• provides the customized maven' settings file gcube-default-settings.xml which contains credentials to access gCube Snapshot repository
• overrides the default Maven's local repository location (which is not writable by ETICS)
• defines env variables for gCube repositories
• replaces original mvn command with the Maven Proxy in the PATH (see section Maven Proxy)
• provides mavenizer tool and put it in the PATH (see section Mavenizer)

In detail, maven-builder exports following environment variables:

MAVEN_GCUBE_RELEASES:	http://maven.research-infrastructures.eu:8081/nexus/content/repositories/gcube-releases/
MAVEN_GCUBE_SNAPSHOTS:	http://maven.research-infrastructures.eu:8081/nexus/content/repositories/gcube-snapshots/
MAVEN_GCUBE_STAGING:	http://maven.research-infrastructures.eu:8081/nexus/content/repositories/gcube-staging/
MAVEN_LOCAL_REPOSITORY:	${moduleDir}/localrepository
MAVEN_SETTINGS:	${moduleDir}/etc/gcube-default-settings.xml
PATH:	${moduleDir}/bin:$PATH

maven-builder is a pre-requisite to build any gCube component in ETICS. All maven-based components must have a direct or indirect dependency on maven-builder.

To simplify dependency specification in your component, for all basic components on which almost all other gCube components depends, a dependency on maven-builder has been set. This allows, for most of components, to avoid a direct dependency on maven-builder.

If your components does not functionally depends on any of gcf, legacy-ws-core, 'maven-parent-*, you will have to explicitly set a dependency on maven-builder.

Maven Proxy

When running Maven in ETICS' integration builds, a set of properties and plugins have to be properly configured in order to let Maven behave as expected. To achieve this result transparently for Developers, it has been decided to have a proxy script for Maven. It is called mvn (like the original Maven command) and the maven-builder replace the original mvn with the proxy script in the PATH. The final result is that when mvn is called from inside a build command of an ETICS configuration, the mvn proxy is invoked instead.

Actions performed by mvn proxy are, in order:

1. sets settings.xml file and path for maven local repository
2. checks that pom and ETICS versions are coherent
3. runs versions plugin replacing dependencies' versions in pom.xml files to force Maven to use dependencies stored in the local repository instead of download version required from remote repositories.
4. set -Dmaven.deploy.skip property that avoid publication of artifacts on gCube Releases if not the last build
5. call original mvn

Step 3) is required because we want to ensure a real integration between components: if a component A depends on component B, we want that B is the one compiled in the context of the same build and not another one compiled previously and uploaded on gCube repository. Please note that this does not apply for externals: Maven Central repository is enabled.

Mavenizer

 README: As of September 2015 (last update of this wiki), all gCube components are using Maven. The Mavenizer is still supported in ETICS but not longer used.

Mavenizer is a tool created with the intent of making gCube legacy components compatible with new maven components. It is able to generate maven artifacts starting from outcomes of compilation. It is meant to be used by ETICS during its builds to mavenize legacy components with a twofold benefit:

• being able to have those components in gCube Maven Repositories
• make legacy components available for Maven components in the local Maven repository for compile-time dependency resolution purposes

It provides two commands that must be added in build commands of each legacy configuration: mavenize and mavenize_sa.

mavenize

This command take as input an artifact that can be .jar, .gar, .war, writes a pom.xml on-the-fly for that artifact and deploys it in the gCube's Maven Repository.

mavenize command determines (if not provided from command line - see below) Maven coordinates groupId:artifactId:version for the artifact it is going to create from the profile.xml file of the component. The algorithm is:

• groupId is equal to org.gcube.<ServiceClass>, where <ServiceClass> is the string found in the profile with "-" and "." replaced by "" and all in lowercase
• artifactId is equal to the <PackageName> value found in the profile.xml for the <Software> section corresponding to the artifact that is going to be mavenized;
• version is equal to the <version> value found in the profile.xml for the <Software> section corresponding to the artifact that is going to be mavenized;

The snippet below shows the input parameters accepted by mavenize script.

gabriele@laptop-trasferte:~/tmp$ mavenize --help
USAGE: /home/gabriele/projects/ENG.iMarine/workspace/maven-builder/bin/mavenize [flags]
flags:
  -f,--file:  artifact file to mavenize. If not specified, a search will be done in the etics module home trying to find a suitable artifact (default: '')
  -p,--profile:  profile.xml file from where take information about the artifact (default: '')
  -g,--groupid:  profile.xml file from where take information about the artifact (default: '')
  -a,--artifactid:  profile.xml file from where take information about the artifact (default: '')
  -v,--version:  profile.xml file from where take information about the artifact (default: '')
  -n,--[no]dryrun:  no act. Perform just a simulation (default: false)
  -l,--loglevel:  log level. Accepted values are 0, 1, 2 (default: 1)
  -h,--[no]help:  show this help (default: false)

All parameters are optional. If a parameter is not provided, mavenize assumes that files are in the standard location for gCube components:

• if --file is not provided, by default a jar/gar/war will be searched in the ${prefix} directory. If multiple artifacts are found in ${prefix} the command fails
• if --profile is not specified, by default the file in ${moduleDir}/etc/profile.xml will be used. If it is not present, the command fails
• if --groupid, --artifactid and/or --version are specified, their values will override values found in the profile.xml

To trigger execution of mavenize on a given legacy component, it is enough to have an invocation to mavenize in the component's ETICS build configuration in target prepublish or postpublish (as shown in figure below).

Note: call to mavenize should be already present in all legacy components since it has been added automatically by ETICS administrators.

mavenize_sa

This command is specific to execute specific actions on legacy servicearchive components. In particular it:

• updates the profile.xml contained in the servicearchive adding, for each package, the maven coordinates of that package following the same convention described in mavenize
• for each doc component found in the servicearchive, creates a jar artifacts and deploy it on the gCube Maven repositories with the same coordinates of the package to which it refers with the classifier javadoc
• transform the servicearchive package into a maven artifact that has the same coordinates of the first package found in the profile.xml and deploy it on gCube Maven repositories with the classifier servicearchive

The mavenize_sa accept as input parameter the servicearchive artifact to process. If not provided, it will search for a .tar.gz file into the standard location where ETICS creates packages.

gabriele@laptop-trasferte:~/tmp$ mavenize_sa --help
USAGE: /home/gabriele/projects/ENG.iMarine/workspace/maven-builder/bin/mavenize_sa [flags]
flags:
  -s,--sa:  servicearchive file containing the profile to update (default: '')
  -n,--[no]dryrun:  no act. Perform a simulation (default: false)
  -l,--loglevel:  log level. Accepted values are 0, 1, 2 (default: 1)
  -h,--[no]help:  show this help (default: false)

To trigger execution of mavenize_sa on a given legacy servicearchive, it is enough to have an invocation to mavenize_sa in the component's ETICS build configuration in target prepublish

Note: a call to mavenize_sa should be already present in all legacy servicearchive since it has been added automatically by ETICS administrators.

Further Reading

• the presentation Bringing Maven to gCube taken at 1st TCOM event: http://bscw.research-infrastructures.eu/bscw/bscw.cgi/d244165/WP7_Bringing_Maven_to_gCube.pdf
• the document Bringing Maven to gCube that explains in detail the usage of Maven in gCube, not only in ETICS but also in the development phase: http://bscw.research-infrastructures.eu/bscw/bscw.cgi/d244327/Project%20Apash.pdf