Difference between revisions of "Data Transfer 2"
(→Design) |
(→Architecture) |
||
Line 31: | Line 31: | ||
=== Architecture=== | === Architecture=== | ||
− | The architecture of Data Transfer is a simple client server model. The client contacts the server asking for its capabilities, submits a transfer requests and monitors its status/outcome. The server side is designed exploiting the plugin design pattern, so that it can be easily extended without introducing hard dependencies between the architecture's components. | + | The architecture of Data Transfer is a simple client server model. The client contacts the server asking for its capabilities in order to decide the best efficient transfer way between the two sides, submits a transfer requests through the selected transfer channel and monitors its status/outcome. The server side is designed exploiting the plugin design pattern, so that it can be easily extended without introducing hard dependencies between the architecture's components. |
== Deployment == | == Deployment == |
Revision as of 17:07, 9 September 2016
Contents
Overview
The implementation of a reliable data transfer mechanisms between the nodes of a gCube-based Hybrid Data Infrastructure is one of the main objectives when dealing with large set of multi-type datasets distributed across different repositories.
To promote an efficient and optimized consumption of these data resources, a number of components have been designed to meet the data transfer requirements.
This document outlines the design rationale, key features, and high-level architecture, the options for their deployment and as well some use cases.
Key features
The components belonging to this class are responsible for:
- Point to Point transfer
- direct transfer invocation to a gCube Node
- reliable data transfer between Infrastructure Data Sources and Data Storages
- by exploiting the uniform access interfaces provided by gCube and standard transfer protocols
- automatic transfer optimization
- by exploiting best available transfer options between invoker and target nodes
- advanced and extensible post transfer processing
- plugin - oriented implementation to serve advanced use case
Design
Philosophy
In a Hybrid Data e-Infrastructure, transferring data between nodes is a crucial feature. Given the heterogeneous nature of datasets, applications, networks capabilities and restrictions, efficient data-transfers can be problematic. Negotiation of best-efficient-available-transfer method is essential in order to achieve a global efficiency, while an extensible set of post operations lets application serve specific needs among common ones. We wanted to provide a light-weight solution that could fit the evolving nature of the infrastructure and its communities needs.
Architecture
The architecture of Data Transfer is a simple client server model. The client contacts the server asking for its capabilities in order to decide the best efficient transfer way between the two sides, submits a transfer requests through the selected transfer channel and monitors its status/outcome. The server side is designed exploiting the plugin design pattern, so that it can be easily extended without introducing hard dependencies between the architecture's components.