Revision as of 22:07, 2 May 2012

Introduction

This document describes the procedures and tools used for the integration and release of Maven-based gCube components. After an initial overview of the overall solution, the document narrow its focus to the integration of Maven components in ETICS.

This document is also an operational guide for developers with all information needed to properly configure a Maven-based gCube component in ETICS to successfully integrate with other components in gCube. It provides also very quick-to-read checklist with the minimal set of things to know and to do in order to let things work in ETICS.

Glossary

• Maven component: a gCube component which uses Maven ad building tool
• Legacy component: a gCube component (most probably developed before iMarine project) which uses the classical building system based on ant
• Mavenizer: a tool developed within gCube project able to create a pom.xml file

Checklist for Developers

For Maven components configurations:

• be sure your component has a dependency on maven-builder
  • since also gCore depends on maven-builder, if your component depends on gCore, no action is required
• if your Maven component has a parent (usually maven-parent) be sure you have a dependency in your component towards the needed parent's version
• Maven's goal executed in build commands must be deploy. This will allows artifacts of your component to be automatically published on gCube repository at every build

For legacy components configurations:

• be sure mavenize is called in postpublish target
• be sure that the artifact generated by the component (usually .jar, .gar, .war) is properly copied in ${prefix}

gCube Maven Repositories

gCube Releases

• Contains artifacts officially released.
• URL: http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases/
• access: only release manager
• only release version, no re-deploy

gCube Snapshots

• Contains snapshot versions of of gCube components. It is used mainly during the development phase to make public and available for integration not yet released software
• URL: http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-snapshots/
• access: release manager, developers
• only snapshot versions, redeploy allowed

gCube Externals

• Contains patched versions of third party components. Those components cannot be considered part of gCube system
• URL: http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-externals/
• access: release manager

Overall strategy

Figure below catches the main components, and their interactions, that are involved in the gCube' software integration, release and distribution.

The central element of the gCube release and distribution strategy is the gCube Releases Repository. It is a classical Maven repository where all released versions of all gCube components are stored.The gCube Releases Repository is the main distribution site for gCube Software:

• Developers configure their development machine to use gCube Releases to resolve and fetch dependencies (a) for gCube components their are developing.
• the Software Gateway service uses gCube Releases as back-end (among, potentially, other third party Maven repositories (c) ) making software stored there available for dynamic deployment within the D4Science Infrastructure.

Another central role is played by ETICS: the integration system used by gCube. It is responsible for the publication (d) in gCube Releases of artifacts that comes out from the integration of gCube components. In particular, actions performed by ETICS are:

• checkout (e), compilation and testing of gCube Maven components from the gCube' source code repository
• validation and certification of artifacts. This ensure that components published in gCube Releases satisfy the quality level defined in the gCube project
• mavenization of legacy components. This make also gCube components that builds with ant available for dependency resolution in Maven
• publication of artifacts in gCube Releases

An identical approach is used also for integration of development version of components. Main differences in this case are:

• gCube Snapshots is used instead of gCube Releases
• also Developers can commit a snapshot version of their component directly from their development machine
• development versions are never deployed in the D4Science production infrastructure (but they are used in the testing infrastructure)

Bringing Maven to ETICS: the maven-builder

ETICS builds take place on a remote machine on a "empty" environment. In order to properly build Maven components it is needed Maven to be installed and configured on the building machine. All those tasks are performed by an ETICS component created for this purpose called maven-builder.

Briefly, this is what maven-builder provides:

• downloads and install Maven tool on the building machine
• downloads and install the jdk and Ant tool on the building machine
• provides the customized maven' settings file gcube-default-settings.xml which contains credentials to access gCube Snapshot repository
• overrides the default Maven's local repository location (which is not writable by ETICS)
• defines env variables for gCube repositories
• replaces original mvn command with the Maven Proxy in the PATH (see section Maven Proxy)
• provides mavenizer tool and put it in the PATH (see section Mavenizer)

In detail, maven-builder exports following environment variables:

MAVEN_GCUBE_RELEASES:	http://maven.research-infrastructures.eu:8081/nexus/content/repositories/gcube-releases/
MAVEN_GCUBE_SNAPSHOTS:	http://maven.research-infrastructures.eu:8081/nexus/content/repositories/gcube-snapshots/
MAVEN_GCUBE_STAGING:	http://maven.research-infrastructures.eu:8081/nexus/content/repositories/gcube-staging/
MAVEN_LOCAL_REPOSITORY:	${moduleDir}/localrepository
MAVEN_SETTINGS:	${moduleDir}/etc/gcube-default-settings.xml
PATH:	${moduleDir}/bin:$PATH

maven-builder is a pre-requisite to build any gCube component in ETICS. All maven-based components must have a direct or indirect dependency on maven-builder.

To simplify dependency specification in your component, for all basic components on which almost all other gCube components depends, a dependency on maven-builder has been set. This allows, for most of components, to avoid a direct dependency on maven-builder.

If your components does not functionally depends on any of gcf, legacy-ws-core, 'maven-parent-*, you will have to explicitly set a dependency on maven-builder.

Maven Proxy

When running Maven in ETICS' integration builds, a set of properties and plugins have to be properly configured in order to let Maven behave as expected. To achieve this result transparently for Developers, it has been decided to have a proxy script for Maven. It is called mvn (like the original Maven command) and the maven-builder replace the original mvn with the proxy script in the PATH. The final result is that when mvn is called from inside a build command of an ETICS configuration, the mvn proxy is invoked instead.

Actions performed by mvn proxy are, in order:

1. sets settings.xml file and path for maven local repository
2. checks that pom and ETICS versions are coherent
3. runs versions plugin replacing dependencies' versions in pom.xml files to force Maven to use dependencies stored in the local repository instead of download version required from remote repositories.
4. set -Dmaven.deploy.skip property that avoid publication of artifacts on gCube Releases if not the last build
5. call original mvn

Step 3) is required because we want to ensure a real integration between components: if a component A depends on component B, we want that B is the one compiled in the context of the same build and not another one compiled previously and uploaded on gCube repository. Please note that this does not apply for externals: Maven Central repository is enabled.

Mavenizer

Mavenizer is a tool created with the intent of making gCube legacy components compatible with new maven components. It is able to generate a pom.xml on-the-fly from information available on component's profile.xml and ETICS configuration and upload artifacts on gCube repositories. It is meant to be used by ETICS during its builds to mavenize legacy components with a twofold benefit:

• being able to have those components in gCube Maven Repositories
• make legacy components available for Maven components in the local Maven repository for compile-time dependency resolution purposes

The snippet below shows the input parameters accepted by mavenize script.

[gabriele@gabriele-work maven-builder]$ bin/mavenize --help
USAGE: bin/mavenize [flags]
flags:
  -g,--groupId:  the groupId to use in the mvn commandline (default: 'org.gcube')
  -k,--packaging:  the type of the artifact. Default is 'jar'. (default: 'jar')
  -n,--artifactId:  the artifactId to use in the mvn commandline. If not specified, it will be the etics module's name (default: '')
  -v,--version:  the version to use in the mvn commandline. If not specified, it will be the etics module version (default: '')
  -f,--file:  artifact file to mavenize. If not specified, a search will be done in the etics module home trying to find a suitable artifact (default: '')
  -d,--[no]forcedeploy:  force deployment of artifact on remote repository (default: false)
  -s,--[no]dryrun:  no act. Perform just a simulation (default: false)
  -l,--loglevel:  log level. Accepted values are 0, 1, 2 (default: 1)
  -h,--[no]help:  show this help (default: false)

All parameters are optional; if a parameter is not provided, the mavenizer tries to guess the value searching the information in component's profile.xml and/or ETICS configuration (depending on the type fo parameter).

To trigger execution of mavenizer on a given legacy component, it is enough - in most of cases - to have an invocation to mavenize in the component's ETICS build configuration in target postpublish (as shown in figure below).

Note: call to mavenize should be already present in all legacy components since it has been added automatically by ETICS administrators.

ETICS Configurations

This section guides developer in the creation of ETICS configurations for both legacy and maven components.

Legacy components

Legacy components builds with ant and will use the Mavenizer tool in order to synthesize maven artifacts starting from build outcomes and deploy them both in the local repository on the building machine (in order to make them available for maven-based components that require them) and on the gCube Snapshots repository (to make them available in development environment).

Following checks have to been passed by the configuration to be considered valid:

• each not-servicearchive component must have a call to mavenize in its build commands. It is suggested to add it in the prepublish or postpublish target of the build commands. For further information of mavenize command, see section Mavenizer
• each servicearchive component must have a call to mavenize_sa in its build commands. It is required to be in the prepublish target. For further information of mavenize_sa command, see section Mavenizer
• each component should depend (directly or indirectly) by the maven-builder. See section maven-builder for further information

Maven components

Following checks must be passed by a configuration to be considered valid:

• each component should depend (directly or indirectly) by the maven-builder. See section maven-builder for further information
• build command must invoke Maven with goal deploy (e.g. mvn deploy). This allow deploy of artifact directly from ETICS builds
• install target of build commands should contain instructions to copy the profile.xml of the component in the ${prefix} directory. This will allow the registration of the profile at the Software Gateway at the end of the build. It is good practice to copy also the outcome of the compilation (usually jar files) in ${prefix} in order to include them in packages generated by ETICS

Typical build commands for a maven-based component should look like the one in the picture below:

Further Reading

• the presentation Bringing Maven to gCube taken at 1st TCOM event: http://bscw.research-infrastructures.eu/bscw/bscw.cgi/d244165/WP7_Bringing_Maven_to_gCube.pdf
• the document Bringing Maven to gCube that explains in detail the usage of Maven in gCube, not only in ETICS but also in the development phase: http://bscw.research-infrastructures.eu/bscw/bscw.cgi/d244327/Project%20Apash.pdf