Difference between revisions of "Developing gCube Maven Components"

From Gcube Wiki
Jump to: navigation, search
(Services)
(How to request creation of a new proxy repository in Nexus)
Line 161: Line 161:
 
'''Note''': if a component uses a public Maven repository directly, it will work for building but would fail in deployment phase. In fact, Software Gateway is only able to download software accessible from Nexus.
 
'''Note''': if a component uses a public Maven repository directly, it will work for building but would fail in deployment phase. In fact, Software Gateway is only able to download software accessible from Nexus.
  
==== How to request creation of a new proxy repository in Nexus ====
+
==== Request new proxy repository in Nexus ====
 
Proxy Repositories on Nexus are created by the Release Manager. To request a new proxy a new ticket must be created.
 
Proxy Repositories on Nexus are created by the Release Manager. To request a new proxy a new ticket must be created.
  
Line 194: Line 194:
  
 
'''Note''': it is possible that software stored on proxy repositories is not included in search results on Nexus. This happens for those repositories that does not have indexes (or have indexes in a format that Nexus is not able to download). Anyway, once Nexus has downloaded artifacts from the original repository for the first time (e.g. because a compilation of a pom that uses the proxy repository), creates its own indexes and those artifacts will become available also for local search.
 
'''Note''': it is possible that software stored on proxy repositories is not included in search results on Nexus. This happens for those repositories that does not have indexes (or have indexes in a format that Nexus is not able to download). Anyway, once Nexus has downloaded artifacts from the original repository for the first time (e.g. because a compilation of a pom that uses the proxy repository), creates its own indexes and those artifacts will become available also for local search.
 
  
 
=== Dependency Versions ===
 
=== Dependency Versions ===

Revision as of 16:08, 18 July 2012

Common

Maven Coordinates

Maven components' coordinates must follow these conventions:

  • their groupId must be of the following form:
org.gcube.<class>
where class identifies a functionally correlated set of components and helps a range of users (developers, administrator) to discover components in various UIs/reports used within the system and in tools. Logically, the class corresponds to the gCube class of the component's, as specified in its gCube Software Profile. However, more specific, or else cross-class groups may be agreed upon among developers. In all cases, the group should align with official Maven guidelines. By agreement, for example, components in gCube class DataAccess declare a groupId of org.gcube.data.access and those in gCube class DataTransfer declare a groupId of org.gcube.data.transfer. On the other hand, also by agreement, components in gCube class InformationSystem declare a groupId of org.gcube.informationsystem.
  • their artifactId must follow Maven conventions (e.g. avoid capital letters or punctuation other than hyphens).

Parent

All gCube components must inherit from maven-parent. This ensure and promotes compliance with project-wide requirements, from the enforcement of minimal Java and Maven versions to generation and packaging of Javadoc documentation and component sources.

<parent>
 <artifactId>maven-parent</artifactId>
 <groupId>org.gcube.tools</groupId>
 <version>1.0.0</version>
 <relativePath />
</parent>

Service Archive (SA)

Whenever they are associated with a gCube Software Archive (SA), they must produce it as a secondary tar.gz artefact with classifier servicearchive. Note that the SA of Maven components does not need to include Javadoc documentation (as this documentation is produced and published directly as a Maven artefact);

  • for each deployable artefact they produce (primary or secondary), they must include the Maven coordinates of the arteact in the corresponding package section of the gCube Software Profile. Specifying package dependencies in the Software Profile, on the other hand, is no longer required;
  • whenever they are associated with a gCube Software Archive (SA), they must produce it as a secondary tar.gz artefact with classifier servicearchive. Note that the SA of Maven components does not need to include Javadoc documentation (as this documentation is produced and published directly as a Maven artefact);

Dependencies

Service and libraries that depend on other gCube components (Maven-based or Ant-based) can find them in the gCube Maven Repositories: gcube-snapshots and gcube-releases.

gCube Maven Repositories

As a one-off task, developers must configure these repositories in the ~/.m2/settings.xml file of their local Maven installation, as follows:

<settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
 
	<servers>
		<server>
			<id>gcube-snapshots</id>
			<username>gcube-user</username>
			<password>maven</password>
		</server>
	</servers>
 
 
	<profiles>
		<profile>
			<id>gcube</id>
			<repositories>
				<repository>
					<id>gcube-snapshots</id>
					<name>gCube Snapshots</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-snapshots</url>
					<releases>
						<enabled>false</enabled>
					</releases>
					<snapshots>
						<enabled>true</enabled>
					</snapshots>
				</repository>
				<repository>
					<id>gcube-releases</id>
					<name>gCube Releases</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases</url>
					<snapshots>
						<enabled>false</enabled>
					</snapshots>
					<releases>
						<enabled>true</enabled>
					</releases>
				</repository>
				<repository>
					<id>gcube-externals</id>
					<name>gCube Externals</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-externals</url>
					<snapshots>
						<enabled>false</enabled>
					</snapshots>
					<releases>
						<enabled>true</enabled>
					</releases>
				</repository>
			</repositories>
 
			<pluginRepositories>
				<pluginRepository>
					<id>gcube-snapshots</id>
					<name>gCube Snapshots</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-snapshots</url>
					<releases>
						<enabled>false</enabled>
					</releases>
					<snapshots>
						<enabled>true</enabled>
					</snapshots>
				</pluginRepository>
				<pluginRepository>
					<id>gcube-releases</id>
					<name>gCube Releases</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases</url>
					<snapshots>
						<enabled>false</enabled>
					</snapshots>
					<releases>
						<enabled>true</enabled>
					</releases>
				</pluginRepository>
				<pluginRepository>
					<id>gcube-externals</id>
					<name>gCube Externals</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-externals</url>
					<snapshots>
						<enabled>false</enabled>
					</snapshots>
					<releases>
						<enabled>true</enabled>
					</releases>
				</pluginRepository>
			</pluginRepositories>
 
		</profile>
	</profiles>
 
	<activeProfiles>
		<activeProfile>gcube</activeProfile>
	</activeProfiles>
</settings>


Note:Developers must not configure these repositories in their POMs, only in settings.xml.

This is to:

  • keep their projects independent from future changes in repository locations;
  • avoid interference with ETICS builds, where access to these repositories may comprise correct system integration;


Third-party Repositories

Using the the settings.xml aforementioned, any dependency declared in pom.xml that can be found in one of gCube repositories or in Maven Central will be resolved at build time and deployment time without any further configuration. Frequently it may happens that a gCube component needs software stored on a public Maven repository not in on of those repositories. In this case, Maven provides an easy solution for declaring additional repositories for dependency resolution by adding a <repositories> section in the pom.xml of the project. E.g.:

<repositories>
    <repository>
    	<id>[repository-id]</id>    	<url>[repository URL]</url>    </repository>
</repositories>

Though technically possible to add any Maven repository in this way, in order to preserve the quality of gCube software it is required to check trustworthiness of its dependencies disallowing usage of certain external software. To enforce such a control over public Maven repositories, gCube components are allowed to use external repository not directly but only through a proxy repository (a.k.a. shadow repository) created on CNR's Nexus Repository Portal (http://maven.research-infrastructures.eu/nexus/).

List of all repositories in Nexus Portal is available at http://maven.research-infrastructures.eu/nexus/index.html#view-repositories

Note: if a component uses a public Maven repository directly, it will work for building but would fail in deployment phase. In fact, Software Gateway is only able to download software accessible from Nexus.

Request new proxy repository in Nexus

Proxy Repositories on Nexus are created by the Release Manager. To request a new proxy a new ticket must be created.

Details of ticket are:

  1. open a ticket of type "Task" in iMarine TRAC (at https://issue.imarine.research-infrastructures.eu/)
  2. ticket's description must report:
    • URL of repository to add
    • organization which hosts the repository
    • which artifacts are needed from that repository
  3. Type is Task
  4. Milestone field should be set to the gCube release in which the component is expected to be released
  5. Component field must report the gCube component that needs the external repository
  6. Blocking should list (if any) other tickets blocked by the addition of this Maven repository
  7. Task Category is software
  8. assign the ticket to the Release Manager

An example of ticket for requesting a new repository mirror is reported in the figure below.

Ticket for requesting a new Mirror Repository

Release Manager will process the ticket checking that no issues are foreseen from the usage of software stored in the repository. If request is accepted, a new proxy repository is created and the repository's URL will be communicated publicly on gCube's developers mailinglist (a.k.a. TCOM mailinglist).

Knowing the mirror repository URL, developer can use it adding following lines in pom.xml. E.g.

<repositories>
    <repository>
    	<id>gcube-ext-osgeo</id>    	<url>http://maven.research-infrastructures.eu/nexus/content/repositories/osgeo/</url>    </repository>
</repositories>

Note: it is possible that software stored on proxy repositories is not included in search results on Nexus. This happens for those repositories that does not have indexes (or have indexes in a format that Nexus is not able to download). Anyway, once Nexus has downloaded artifacts from the original repository for the first time (e.g. because a compilation of a pom that uses the proxy repository), creates its own indexes and those artifacts will become available also for local search.

Dependency Versions

ETICS builds ensure correct integration of gCube components, regardless of the versions their dependencies. It also ensures that components are released with fixed versions for their dependencies, i.e. ensures build reproducibility.

Accordingly, developers can be liberal in the choice of dependency versions. i.e. rely on version ranges to:

  • minimise the requirements for version management;
  • be readily informed of any mis-alignment with their dependencies;

As we start adopting Maven, the lower-bound of version ranges will be a SNAPSHOT version. For example, a dependency on the streams library can be specified as follows:

<dependency>
    <groupId>org.gcube.data.access</groupId>
    <artifactId>streams</artifactId>
    <version>[1.0.0-SNAPSHOT,2.0.0)</version>
</dependency>

With this dependency, a component will be soon notified of a non-retro compatible change that, contrary to conventions, streams may have introduced in, say, version 1.3.0 (typically the day after at the latest). The notification will be delivered in the IDE, assuming the availability of Maven integration (e.g. m2e in Eclipse). The component is in fact free to open the range further if it wants to be aligned with its dependencies across major versions.


Common Dependencies

Many gCube components depend on gcf, which is now available as a Maven component since version 1.4. At the time of writing, the component has not been released yet. A dependency on it is then specified as follows:

<dependency>
    <groupId>org.gcube.core</groupId>
    <artifactId>gcf</artifactId>
    <version>[1.4.0-SNAPSHOT,2.0.0)</version>
    <scope>provided</scope>
</dependency>

The scope provided indicates that these dependency will be satisfied in the development environment (the gHN) and should not be included in GAR and SA archives.

Some components depend on the libraries available in all gHNs, though they do not depend on gcf itself. The subset of these libraries that are required for compilation and integration testing have been coalesced in a single Jar, distributed as a Maven component called ghn-core-runtime. A dependency on this Jar can be specified as follows:

<dependency>
    <groupId>org.gcube.distribution</groupId>
    <artifactId>ghn-core-runtime</artifactId>
    <version>1.0.0</version>
    <scope>provided</scope>
</dependency>

Note: a component that depends on gcf indirectly depends on gun-core-runtime.

Another common dependency is on the gCube gRS2, an Ant component that the build infrastructure make available in Maven repositories. A dependency on this component can be specified as follows:

<dependency>
    <groupId>org.gcube.execution</groupId>
    <artifactId>grs2library</artifactId>
    <version>[1.2.0-SNAPSHOT,2.0.0)</version>
</dependency>

Libraries

The streams library is a Maven component in gCube class DataAccess. We will use it as example of a library component in Maven.

Profile

The gCube Software profile of the library is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ID />
  <Type>Service</Type>
  <Profile>
    <Description>Embedded Domain-Specific Language for Stream Transformations</Description>
    <Class>DataAccess</Class>
    <Name>streams</Name>
     <Version>1.0.0</Version>
     <Packages>
       <Software>
         <Name>streams</Name>
         <Version>1.0.0-SNAPSHOT</Version>
         <MavenCoordinates>           <groupId>org.gcube.data.access</groupId>           <artifactId>streams</artifactId>           <version>1.0.0-SNAPSHOT</version>         </MavenCoordinates>         <Files>
           <File>streams-1.0.0-SNAPSHOT.jar</File>
         </Files>
       </Software>
     </Packages>
  </Profile>
</Resource>

Notice that:

  • the profile includes a single package and this package corresponds to the main build artefact of the component;
  • the whole profile and the package have usual gCube coordinates;
  • the package name is aligned with the Maven artifactId of the component;
  • the package version is aligned with the Maven version of the component;
  • the package includes MavenCoordinates that can be directly copied and pasted from the POM;
  • the package does not specify dependencies;
  • the package points to the main artefact of the Maven build;


The alignment between profile and POM can be exploited to simplify the management of the profile across different component versions. In particular, streams uses POM variables top define its profile:

<?xml version="1.0" encoding="UTF-8"?>
<Resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ID />
  <Type>Service</Type>
  <Profile>
    <Description>${description}</Description>
     <Class>DataAccess</Class>
     <Name>${artifactId}</Name>
     <Version>1.0.0</Version>
     <Packages>
       <Software>
         <Name>${artifactId}</Name>
         <Version>${version}</Version>
         <MavenCoordinates>
            <groupId>${groupId}</groupId>
            <artifactId>${artifactId}</artifactId>
             <version>${version}</version>
         </MavenCoordinates>
	 <Files>
            <File>${build.finalName}.jar</File>
         </Files>
       </Software>
     </Packages>
</Profile>
</Resource>

Note that the variables must be resolved (i.e. the profile is interpolated) before the profile can be used within the system. In particular, the main destination of the profile is the gCube Software Archive (SA) which packages the component for registration within the system.

Service Archive

streams is responsible for generating its own SA as part of its Maven build. This requires dedicated logic in the POM but has the advantage that:

  • SA validity can be verified locally;
  • no SA configurations are required in ETICS;
  • the required build logic can be easily reused across components;

streams uses the Maven Assembly Plugin to generate its own SA, with an approach that makes optimal use of Maven variable interpolation in static files such as README, svnpath.txt, MAINTAINERS, changelog.xml, etc. In particular, the Assembly plugin takes care of variable interpolation in the profile and other static files. The approach is best illustrated with a reference to the [sources] of the streams library.


As we discuss below, streams has the final requirement to produce an interpolated profile during ETICS builds, outside the context of the SA. streams meets this requirement with the Maven Resources Plugin. Again, consult the [sources] of the streams library to reuse this approach.

Services

The tree-manager service is a Maven component in gCube class DataAccess.

Project Structure

Differently from a library, the service is associated with a number of distinguished build artefacts:

  • the service implementation (a Jar);
  • the stubs automatically generated from the WSDL interfaces (a Jar);
  • the archive required for deployment in gCore (a Gar);
  • the archive required for registration in the system (a SA);

Producing these artefacts requires sharing a number of static and dynamic files (from profiles to WSDLs to pre-processed WSDLs). The usual Maven approach of mapping the artefacts onto different Maven projects is then genuinely difficult. It is instead more practical to use a single multi-module Maven project. The tree-manager uses this module structure:

  tree-manager  (parent module)------- (produces) --- (POM only)
         |
         |----- tree-manager-service   (service impl module)------- (produces) --- jar, gar, SA
         |                       |
         |                       | (depends on)
         |                       |
         |                      \/
         |----- tree-manager-stubs     (service stubs module)------- (produces) --- jar

As customary, the parent does not produce any artefact (is a POM-only project). It declares these coordinates in its POM:

<groupId>org.gcube.data.access</groupId>
<artifactId>tree-manager</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>pom</packaging>

The parent physically includes the sub-modules;

<modules>
  <module>tree-manager-stubs</module>
  <module>tree-manager-service</module>
</modules>

Finally, the parent inherits from maven-parent:

<parent>
 <artifactId>maven-parent</artifactId>
 <groupId>org.gcube.tools</groupId>
 <version>1.0.0</version>
 <relativePath />
</parent>

The submodules inherit from the parent:

<parent>
  <artifactId>tree-manager</artifactId>
  <groupId>org.gcube.data.access</groupId>
  <version>1.0.0-SNAPSHOT</version>
  <relativePath>..</relativePath>
</parent>

and share its version and groupId. Their coordinates are then minimal:

<artifactId>tree-manager-service</artifactId>
<artifactId>tree-manager-stubs</artifactId>

The tree-manager-service module produces a Jar of its own implementation, as well the GAR and the SA of the whole services as secondary artefacts with classifiers of, respectively, gar and service archive.

The tree-manager-stubs module produces a Jar of its own implementation.

Finally, the parent module coordinates the build of its sub-modules and centralises the declaration of common dependencies and configuration. Refer to the [project sources] to reuse this approach.

Generating Stubs and Gars

The tree-manager-stubs uses the gCube Service Plugin to generate its classes from WSDL interfaces.

Similarly, the tree-manager-service uses the plugin to generate its secondary GAR artefact.

The configuration of the plugin is conveniently shared in the parent module.

Refer to the [project sources] of the streams for details.

Profile

The gCube Software profile (profile.xml) of a development version of the service is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Resource>
	<ID></ID>
	<Type>Service</Type>
	<Profile>
		<Class>DataAccess</Class>
		<Name>tree-manager-service</Name>
		<Version>1.0.0</Version>
		<Packages>
			<Main>
				<Name>tree-manager-service</Name>
				<Version>1.0.0</Version>
				<MavenCoordinates>					<groupId>org.gcube.data.access</groupId>					<artifactId>tree-manager-service</artifactId>					<version>1.0.0-SNAPSHOT</version>				</MavenCoordinates>				<GARArchive>tree-manager-service-1.0.0-SNAPSHOT.gar</GARArchive>
				<PortType>
					<Name>gcube/data/tm/binder</Name>
				</PortType>
				<PortType>
					<Name>gcube/data/tm/reader</Name>
				</PortType>
				<PortType>
					<Name>gcube/data/tm/writer</Name>
				</PortType>
			</Main>
			<Software>
				<Name>tree-manager-stubs</Name>
				<Version>1.0.0</Version>
				<MavenCoordinates>					<groupId>org.gcube.data.access</groupId>					<artifactId>tree-manager-stubs</artifactId>					<version>1.0.0-SNAPSHOT</version>				</MavenCoordinates>				<Type>library</Type>
				<Files>
					<File>tree-manager-stubs-1.0.0-SNAPSHOT.jar</File>
				</Files>
			</Software>
		</Packages>
	</Profile>
</Resource>

Notice that:

  • the profile includes two packages and this package corresponds to the main build artefacts of the submodules;
  • the whole profile and the packages have usual gCube coordinates;
  • package names are aligned with the Maven artifactIds of the submodules;
  • package versions are aligned with the Maven versions of the submodules;
  • package include MavenCoordinatess that can be directly copied and pasted from the POM of the submodules;
  • packages do not specify dependencies;
  • packages points to the main artefact of the Maven build of the submodules;

Notice that, unlike libraries, services cannot easily use Maven variables in their profiles (as these artefacts are used for different purposes by different modules). Accordingly, developers needs to maintain versions used in profiles across releases. In particular, they need to:

  • replace SNAPSHOT versions with release versions in release branches;
  • advance SNAPSHOT versions in HEAD;

Service Archive

tree-manager-service generates the service SA as part of its Maven build. This requires dedicated logic in the POM of the module and its parent.

As shown for libraries, the service uses the Maven Assembly Plugin to generate its own SA, with an approach that makes optimal use of Maven variable interpolation in static files such as README, svnpath.txt, MAINTAINERS, changelog.xml, etc. In particular, the Assembly plugin takes care of variable interpolation for these files. Refer to the [project sources] for details.

Portlets