Difference between revisions of "Continuous Integration procedure"

From Gcube Wiki
Jump to: navigation, search
(Build Activity)
 
(7 intermediate revisions by 2 users not shown)
Line 3: Line 3:
  
 
== Introduction ==
 
== Introduction ==
Continuous Integration is a well known procedure in software development that aims to reduce time to delivery software improving its quality. Continuous Integration consists in a set of practices and polices that should be enforced as widely as possible among the development team. The most important practices to follow are:
+
Continuous Integration is a well known procedure in software development that aims to reduce time to deliver software improving its quality. Continuous Integration consists of a set of practices and polices that should be enforced as widely as possible among the development team. The most important practices to follow are:
 
* keep '''source code in repository''', preferably a Version Control System (e.g. [http://www.nongnu.org/cvs/ CVS], [http://subversion.tigris.org/ SVN])
 
* keep '''source code in repository''', preferably a Version Control System (e.g. [http://www.nongnu.org/cvs/ CVS], [http://subversion.tigris.org/ SVN])
* automate builds '''on daily basis''' (or even more frequently).
+
* automate builds '''on daily basis''' (or even more frequently)
 
* automated builds '''should include all source code''' (even database scripts!). The idea is that anyone can, standing in front of a virgin machine, check out source code from the repository, build it and have a running system on his machine
 
* automated builds '''should include all source code''' (even database scripts!). The idea is that anyone can, standing in front of a virgin machine, check out source code from the repository, build it and have a running system on his machine
* builds are ''self-testing''. Source code includes tests that are autmoatically executed at each build. Testing include not only unit-testing, but also functional and deployment testing for instance
+
* builds are ''self-testing''. Source code includes tests that are automatically executed at each build. Testing include not only unit-testing, but also functional and deployment testing for instance
 
* developers '''commit on mainline of source repository as frequent as possible''', keeping the mainline builds successfully
 
* developers '''commit on mainline of source repository as frequent as possible''', keeping the mainline builds successfully
  
  
 
== Continuous Integration of gCube System ==
 
== Continuous Integration of gCube System ==
In next sections, most important best practices adopted in gCube System in order to speed-up software delivery and assure an high software quality are listed.
+
All gCube source code is maintained in centralized SVN Repository. Further details on this can be found in the section [[gcube subversion repository|gCube Subversion Repository]]. Development guidelines ask developers to provide one (or more) test-suites to test modules they develop.  
  
 
+
Automatic builds of the entire gCube System software are scheduled on the [[Integration Infrastructure|Integration Infrastructure]] on daily basis. Two versions of software are built:
=== Source Code Repository ===
+
All gCube and gCore source code is maintained in centralized SVN Repository. Further details on this can be found in the section [[gcube subversion repository|gCube Subversion Repository]].
+
 
+
=== Self-testing code ===
+
JRA guidelines ask developers to provide one (or more) test-suites to test modules they develop. Provided test-suites are executed during [[Functional Testing|Functional Testing]]
+
 
+
=== Automated Builds ===
+
Automatic builds of the entire gCube System software are scheduled on the [[Building Infrastructure|Building Infrastructure]] on daily basis. Two versions of software are built:
+
 
* gCube Development Configuration (org.gcube.HEAD): the latest version of gCube source code is built to quickly reveal and fix integration problems  
 
* gCube Development Configuration (org.gcube.HEAD): the latest version of gCube source code is built to quickly reveal and fix integration problems  
 
* gCube Release Candidate Configuration: if there is an ongoing [[Major/Minor Release Cycle procedure|Release Cycles]] or [[Maintenance Release Cycle procedure|Maintenance Cycles]], candidate configuration is built to solve all integration problems
 
* gCube Release Candidate Configuration: if there is an ongoing [[Major/Minor Release Cycle procedure|Release Cycles]] or [[Maintenance Release Cycle procedure|Maintenance Cycles]], candidate configuration is built to solve all integration problems
Line 28: Line 20:
  
 
==== Development Configurations ====
 
==== Development Configurations ====
In ETICS, the org.gcube project each component has a "special" development configuration called: ''org.gcube.component-name.HEAD''. This configuration points, at any time, to the latest source code available on the svn repository. From a practical point of view, this means that a valid *.HEAD configuration has the ''tag'' field pointing to svn repository's trunk and has not the revision number. Development configurations also exist for all subsystems and for the whole project. The latter is named '''org.gcube.HEAD'''.
+
In [[ETICS|ETICS]], each component has a "special" configuration called: ''org.gcube.<component-name>.HEAD''. This configuration points, at any time, to the latest source code available on the SVN repository. To be valid, any valid *.HEAD configuration has the ''tag'' field pointing to svn repository's trunk and has not a revision number set. Development configurations also exist for all subsystems and for the whole project. The latter is named '''org.gcube.HEAD'''.
  
 
<pre>
 
<pre>
Line 49: Line 41:
  
 
==== Integration Configurations ====
 
==== Integration Configurations ====
When a [[Major/Minor Release Cycle procedure|Release Cycles]] or a [[Maintenance Release Cycle procedure|Maintenance Cycles]] starts, components to be included in the upcoming version are released creating new ETICS configuration called ''integration configuration''. The most remarkable difference with Development Configurations is the presence, in Integration Configurations, of the SVN revision number in the field '''tag''' that point at a specific commit in the SVN repository. This assure source code attached to configuration will not change during the time. Like Development Configurations, they also exists at component, subsystem and project level.
+
When a [[Major/Minor Release Cycle procedure|Release Cycle]] or a [[Maintenance Release Cycle procedure|Maintenance Cycle]] start, components to be included in the upcoming version are delivered creating new ETICS configuration called ''integration configuration''. The most remarkable difference with Development Configurations is the presence, in Integration Configurations, of the SVN revision number in the field '''tag''' that point at a specific commit in the SVN repository. This assure source code attached to configuration will not change during the time. Like Development Configurations, they also exists at component, subsystem and project level.
  
 
<pre>
 
<pre>
 
Sample Integration Configuration
 
Sample Integration Configuration
  
&nbsp;&nbsp;ETICS name: org.gcore.gcontainer.gcontainer.1.0.0-0  
+
&nbsp;&nbsp;ETICS name: org.gcore.gcontainer.gcontainer.1-0-0
 
&nbsp;module name: org.gcore.gcontainer.gcontainer
 
&nbsp;module name: org.gcore.gcontainer.gcontainer
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;version: 1.0.0-0  
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;version: 1.0.0-0  
Line 70: Line 62:
 
The main goal of Build Activity is to produce software artefacts from source code potentially spread across repositories maintained within the consortium. Resulting artefacts are the basis for subsequent activities of the release procedure such as testing, packaging and distribution to the user communities.
 
The main goal of Build Activity is to produce software artefacts from source code potentially spread across repositories maintained within the consortium. Resulting artefacts are the basis for subsequent activities of the release procedure such as testing, packaging and distribution to the user communities.
  
Building Activity aims at verifying that code compiles correctly and that components are integrated at API level. Building is done with the aid of the [[SA3 Tools Etics|ETICS Command-Line]] tools that, by exploiting the model stored at the remote ETICS database, for each configuration automatically fetches the corresponding code and executes the specified build commands.
+
Building Activity aims at verifying that code compiles correctly and that components are integrated at API level. Building is done with the aid of the [[ETICS#Command-Line_Client|ETICS Command-Line]] tools that, by exploiting the model stored at the remote ETICS database, for each configuration automatically fetches the corresponding code and executes the specified build commands.
  
 
Building Activity takes as "parameter" the ETICS project configuration to build. Typically ''org.gcube.HEAD'' configuration is used for development builds, instead ''org.gcube.[VERSION_NUMEBR]'' is used for release cycle builds.
 
Building Activity takes as "parameter" the ETICS project configuration to build. Typically ''org.gcube.HEAD'' configuration is used for development builds, instead ''org.gcube.[VERSION_NUMEBR]'' is used for release cycle builds.
  
Builds are executed by a Build Server in the [[Building Infrastructure|Building Infrastructure]]. When a build is scheduled, these are the steps executed on the build server that will perform the build:
+
Builds are executed by a Build Server in the [[Integration Infrastructure|Integration Infrastructure]]. When a build is scheduled, these are the steps executed on the build server that will perform the build:
# source code is downloaded from D4Science-II code repository(ies) executing '''etics-checkout''' command;
+
# source code is downloaded from gCube code repository(ies) executing '''etics-checkout''' command;
 
# some statistics on source files types are collected. Later on, they will be available in [[BTRT|BTRT]];
 
# some statistics on source files types are collected. Later on, they will be available in [[BTRT|BTRT]];
 
# source code is built executing ( '''etics-build''' command);
 
# source code is built executing ( '''etics-build''' command);
 
# artefacts, checkout and build logs, ETICS reports, plugin reports are published in the repository;
 
# artefacts, checkout and build logs, ETICS reports, plugin reports are published in the repository;
# ''sa3-certification'' task is launched on the published build;
+
# ''distribution'' task is launched on the published build;
 
+
 
+
ETICS allows (if properly configurated) execution of a number of plugins during '''etics-build''' execution. Currently two plugins are executed during SA3 builds: [http://checkstyle.sourceforge.net/ '''checkstyle'''] and [http://findbugs.sourceforge.net/ '''findbugs''']. The former scans software sources looking for javadoc and reports the health status of javadoc documentation, the latter scans source code looking for patterns that could reveal tyipical bugs.
+
 
+
==== Time Table ====
+
To make gCube (and gCore) components integration "continuous", software is automatically built on regular basis (typically daily).
+
* org.gcube.HEAD and org.gcore.HEAD configurations are built once a day (usually  at midnight) by a Build Server sitting in [[SA3 Building Infrastructure|Building Infrastructure]]. Artefacts and reports are available at this BTRT page: http://grids16.eng.it/BuildReport/browse/Recent_Builds/org.gcube.HEAD and http://grids16.eng.it/BuildReport/browse/Recent_Builds/org.gcore.HEAD
+
* during a Release Cycle, software for that release is built twice a day usually at midnight and at midday
+
 
+
  
Summarizing, these are all scheduled builds running in Building Infrastructure:
 
  
{| cellspacing="0" cellpadding="3" border="1" align="center" style="text-align: center; background-color:#FFFFFF"
+
ETICS allows (if properly configured) execution of a number of plugins during '''etics-build''' execution. Currently two plugins are executed during builds: [http://checkstyle.sourceforge.net/ '''checkstyle'''] and [http://findbugs.sourceforge.net/ '''findbugs''']. The former scans software sources looking for javadoc and reports the health status of javadoc documentation, the latter scans source code looking for patterns that could reveal typical bugs.
|- style="background-color:#F2F2F2"
+
! '''built configuration'''
+
! '''timing'''
+
! '''average time to build'''
+
! '''notes'''
+
|-
+
|style="background-color:#F2F2F2"| org.gcube.[RELEASE_NUMBER]
+
| everyday at 00:00 CET and 11:00 CET
+
| 3h:52m
+
| active only during release cycles
+
|-
+
|style="background-color:#F2F2F2"| org.gcube.HEAD
+
| everyday at 00:00 CET
+
| 4h:15m
+
| during release cycle this build is executed in cascade to the org.gcube.[RELEASE_NUMBER] build. In these cases it will start around 04:00 CET
+
|-
+
|style="background-color:#F2F2F2"| org.gcore.HEAD
+
| everyday around 04:00 CET
+
| 0h:3m
+
| this build in executed in cascade to org.gcube.HEAD
+
|}
+
  
 
== External Links ==
 
== External Links ==
 
* [http://martinfowler.com/articles/continuousIntegration.html Continuous Integration by Martin Fowler]
 
* [http://martinfowler.com/articles/continuousIntegration.html Continuous Integration by Martin Fowler]

Latest revision as of 23:58, 3 November 2015

"Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly." - Martin Fowler


Introduction

Continuous Integration is a well known procedure in software development that aims to reduce time to deliver software improving its quality. Continuous Integration consists of a set of practices and polices that should be enforced as widely as possible among the development team. The most important practices to follow are:

  • keep source code in repository, preferably a Version Control System (e.g. CVS, SVN)
  • automate builds on daily basis (or even more frequently)
  • automated builds should include all source code (even database scripts!). The idea is that anyone can, standing in front of a virgin machine, check out source code from the repository, build it and have a running system on his machine
  • builds are self-testing. Source code includes tests that are automatically executed at each build. Testing include not only unit-testing, but also functional and deployment testing for instance
  • developers commit on mainline of source repository as frequent as possible, keeping the mainline builds successfully


Continuous Integration of gCube System

All gCube source code is maintained in centralized SVN Repository. Further details on this can be found in the section gCube Subversion Repository. Development guidelines ask developers to provide one (or more) test-suites to test modules they develop.

Automatic builds of the entire gCube System software are scheduled on the Integration Infrastructure on daily basis. Two versions of software are built:

  • gCube Development Configuration (org.gcube.HEAD): the latest version of gCube source code is built to quickly reveal and fix integration problems
  • gCube Release Candidate Configuration: if there is an ongoing Release Cycles or Maintenance Cycles, candidate configuration is built to solve all integration problems


Development Configurations

In ETICS, each component has a "special" configuration called: org.gcube.<component-name>.HEAD. This configuration points, at any time, to the latest source code available on the SVN repository. To be valid, any valid *.HEAD configuration has the tag field pointing to svn repository's trunk and has not a revision number set. Development configurations also exist for all subsystems and for the whole project. The latter is named org.gcube.HEAD.

Valid development configuration
 
 ETICS name:	org.gcube.annotation-management.abe.HEAD
 module name:	org.gcube.annotation-management.abe
 version:	1.1.0-0
 vcsroot:	http://svn.research-infrastructures.eu/public/d4science/gcube
 tag:	        trunk/annotation-management/Annotation_Back-End
 download from:	http://eticssoft.web.cern.ch/eticssoft/repository${projectName}/${moduleName}/[...]
 profiles:	
 created:	Thursday, April 24, 2008 1:10:36 PM
 modified:	Friday, May 8, 2009 1:29:46 PM
 id:	        99c1e2be-2872-4e91-93bf-ef1d14524520
 locked:	        false

Development configurations are expected to change very quickly (usually several times per day); also the API that each component exposes can change over time. The continuous build of these configurations is the usefulness of these builds, enabling developers to discover build problems difficult to spot on a local, unsynchronized environment.

Integration Configurations

When a Release Cycle or a Maintenance Cycle start, components to be included in the upcoming version are delivered creating new ETICS configuration called integration configuration. The most remarkable difference with Development Configurations is the presence, in Integration Configurations, of the SVN revision number in the field tag that point at a specific commit in the SVN repository. This assure source code attached to configuration will not change during the time. Like Development Configurations, they also exists at component, subsystem and project level.

Sample Integration Configuration

  ETICS name: org.gcore.gcontainer.gcontainer.1-0-0
 module name: org.gcore.gcontainer.gcontainer
     version: 1.0.0-0 
   * VCSROOT: http://svn.research-infrastructures.eu/public/d4science/gcube
         tag: trunk/PATH/TO/MY/COMPONENT@857 
* repository: http://eticssoft.web.cern.ch/eticssoft/repository
        path: ${projectName}/${moduleName}/${version}/${platformName}/${packageName}-${version}-${age}.tar.gz

Note: fields marked with '*' are common to all configurations of the same 
      module and cannot be overridden at configuration level

Build Activity

The main goal of Build Activity is to produce software artefacts from source code potentially spread across repositories maintained within the consortium. Resulting artefacts are the basis for subsequent activities of the release procedure such as testing, packaging and distribution to the user communities.

Building Activity aims at verifying that code compiles correctly and that components are integrated at API level. Building is done with the aid of the ETICS Command-Line tools that, by exploiting the model stored at the remote ETICS database, for each configuration automatically fetches the corresponding code and executes the specified build commands.

Building Activity takes as "parameter" the ETICS project configuration to build. Typically org.gcube.HEAD configuration is used for development builds, instead org.gcube.[VERSION_NUMEBR] is used for release cycle builds.

Builds are executed by a Build Server in the Integration Infrastructure. When a build is scheduled, these are the steps executed on the build server that will perform the build:

  1. source code is downloaded from gCube code repository(ies) executing etics-checkout command;
  2. some statistics on source files types are collected. Later on, they will be available in BTRT;
  3. source code is built executing ( etics-build command);
  4. artefacts, checkout and build logs, ETICS reports, plugin reports are published in the repository;
  5. distribution task is launched on the published build;


ETICS allows (if properly configured) execution of a number of plugins during etics-build execution. Currently two plugins are executed during builds: checkstyle and findbugs. The former scans software sources looking for javadoc and reports the health status of javadoc documentation, the latter scans source code looking for patterns that could reveal typical bugs.

External Links