Difference between revisions of "Data Transfer Facilities"
From Gcube Wiki
Andrea.manzi (Talk | contribs) |
Andrea.manzi (Talk | contribs) (→Main Components) |
||
| Line 32: | Line 32: | ||
:this family of components guarantees VO/VRE Administrators the possibility to transfer data among Data Sources and Data Storages. It can be exploited as well by any client or web services to implements data movement between infrastructure nodes by relying as well on the facilities offered by the [[Result Set components]] | :this family of components guarantees VO/VRE Administrators the possibility to transfer data among Data Sources and Data Storages. It can be exploited as well by any client or web services to implements data movement between infrastructure nodes by relying as well on the facilities offered by the [[Result Set components]] | ||
;the [[Result Set components]] | ;the [[Result Set components]] | ||
| + | :this family of components provide a common data transfer mechanism that aims to establish high throughput point to point on demand communication. | ||
Revision as of 14:16, 30 April 2012
Overview
The implementation of a reliable data transfer mechanisms between the nodes of the iMarine Data e-Infrastructure is one of the main objectives when dealing with large set of multi-type datasets distributed across different repositories, like the one made available by the iMarine EA Community of Practice.
To promote an efficient and optimized consumption of these data resources, a number of components have been designed to meet the data transfer requirements.
This document outlines the design rationale and high-level architecture of such components.
Key Features
The components part of the subsystem provide the following main key features:
- reliable data transfer between Infrastructure Data Sources and Data Storages
- by exploiting the uniform access interfaces provided by gCube and standard transfer protocols
- structured and unstructured Data Transfer
- it guarantees both Tree based and File based transfer to cover all possible use-cases
- transfers to local nodes for data staging
- data staging for particular use cases can be enabled on each node of the infrastructure
- advanced transfer scheduling and transfer optimization
- a dedicated gCube service responsible for data transfer scheduling combined to transfer optimization at the level of protocols and Access interfaces.
- transfer statistics availability
- transfers are logged by the system and made available to interested consumers.
Main Components
- the Data Transfer Scheduler & Agent components
- this family of components guarantees VO/VRE Administrators the possibility to transfer data among Data Sources and Data Storages. It can be exploited as well by any client or web services to implements data movement between infrastructure nodes by relying as well on the facilities offered by the Result Set components
- the Result Set components
- this family of components provide a common data transfer mechanism that aims to establish high throughput point to point on demand communication.
