Revision as of 18:53, 10 January 2012

This section introduces the gCube Building Infrastructure. It consists of all machines, networks and software involved in integration, testing, validation and releasing of gCube System's source code.

The Big Picture

The figure below shows the most relevant components of the infrastructure and their interactions with other infrastructure components and/or external elements.It is clear from the figure that the central component of the infrastructure is a repository(ies) that keep all data of interest for integration, testing, validation and releasing of gCube. All other components interact in someway with the central repository creating, modifying and accessing data.

Few notes on the Fabric Layer

The Building infrastructure is entirely hosted at E-IIS facilities. In order to realize this infrastructure on its datacenter, ENG (namely the Eng's R&D Laboratory) exploited two kind of resources in its own:

• a Storage Area Network (SAN) where all repositories' data is kept
• a cloud-like virtualized environment (managed by KVM + OpenNebula) that hosts infrastructure's virtual machines

The latter point implies that all software in D4S Building Infrastructure run in virtual machines that can be easily started, destroyed, rearranged on different physical machines. The extreme flexibility of this fabric configuration allows the Building Infrastructure to be:

• reliable: storage facilities are out-of-the-box redundant and the ease of manage virtual machines guarantee high availability and performances
• scalable: virtualization technology allows to set-up virtual machines and run them in minutes. In case of need resources can be added to infrastructure in few minutes

Description of infrastructure components

This section provide a short description for each component of D4S Building Infrastructure. Description of what components do, their technical specification and interactions with other internal/external components will be provided.

D4S SA3 Repository

A D4science SA3 Repository is a container for all outcomes related to D4Science software build and test. The main categorization of data in D4S SA3 Repositories is made by ETICS Configurations. Each configuration contains 4 sub-repositories:

• build-repository: contains artefacts and logs for all builds made in the context of the configuration;
• functional-testing repository: contains all reports of functional testing performed against software in that configuration;
• deployment-testing repository: contains all reports of deployment testing performed against builds in that configuration;
• vre-deployment-testing repository: contains all reports of vre-deployment tests performed in the context of that configuration;

In the figure below the typical structure of a d4s-sa3-repository is shown.

Physically, a repository is implemented as a hierarchic directories structure. Actually the physical implementation matches the logical organization discussed above. As mentioned, the filesystem in which repositories exists is hosted on ENG's SAN infrastructure and, hence, data is stored in reliable and redundant way.

Multiple Repositories

Theoretically, there is no need for multiple repositories: all data could sits in a single, huge "D4S_SA3_REPOSITORY". Indeed, from a practical point of view, having multiple repositories is a much more flexible and usable solution. So far, three different repositories have been created: "Diligent_Repository", "D4Science_Old_Builds_Repository", "D4Science_Recent_Builds_Repository". Contextually to D4Science-II project starting, the "D4Science2_Repository" is about to be created.

Clearly, all others infrastructure components (Build Servers, BTRT, Fetcher Server) are designed to work with multiple repositories.

Repository Access

Currently, the only way to access to data from outside ENG network is using the BTRT application.

Build Server

Build Server is the infrastructure component where D4Science software is actually built. Build Server is a virtual machine properly configured to run D4Science builds. In particular it has an installation of etics-client to interact with ETICS System, svn commands, java 5 environment, libraries to run checkstyle and findbugs plugins. Build Server is exploited to run automatic build of development and integration configurations, as well as to run manual builds.

Further information on how Build Servers are exploited, please see Continuous Build and Integration section.

Multiple Build Servers

Build server is not a physical machine, indeed it is a virtual machine. Relying on virtual machines flexibility, Build Server instances can be started and destroyed on demand. Thus, in the infrastructure, multiple build servers can exist at the same time and this can be exploit to run builds of different configurations simultaneously (very useful during a release cycle).

BTRT Server

A BTRT Server is simply a (virtual) machine hosting an instance of BTRT application. Essentially is a linux machine with a Tomcat installation. Like for Build Servers, also BTRT Servers can exists in multiple instances at the same time; this will result in multiple running instances of BTRT application.

For further information on BTRT, its features and purposes, please refer to BTRT wiki page.

Fetch Server

Functional and Deployment tests on built software are not carried out by ENG on its infrastructure, instead they are executed on the D4Science Testing Infrastructure hosted at 4DSoft. In order to make reports generated during tests available through BTRT they have to be downoladed and saved on a d4s-sa3-repository (and, in some cases, processed). The Fetch Server is a very simple server that, using cron jobs, automatically looks for new reports at given urls, download and publish them on a d4s-sa3-repository.

In order to avoid update anomalies, it is strictly required that in the infrastructure only one Fetch Server exist and is active.