Difference between revisions of "Data Transfer 2"

From Gcube Wiki
Jump to: navigation, search
(Architecture)
(Architecture)
Line 31: Line 31:
 
=== Architecture===
 
=== Architecture===
  
The architecture of Data Transfer is a simple client server model. The client contacts the server asking for its capabilities in order to decide the best efficient transfer way between the two sides, submits a transfer requests through the selected transfer channel and monitors its status/outcome. The server side is designed exploiting the plugin design pattern, so that it can be easily extended without introducing hard dependencies between the architecture's components.
+
The architecture of Data Transfer is a simple client server model, respectively implemented by ''data-transfer-library'' and ''data-transfer-service''.
  
The picture below describes the overall architecture.
+
The service is a simple REST server, implemented as a gCube SmartGear web application.
 +
It is designed exploiting the plugin design pattern, so that it can be easily extended without introducing hard dependencies between the architecture's components.
 +
It also exposes interfaces aimed to :
 +
* get node's capabilities in terms of available plugins and supported transfer method;
 +
* submit and monitor transfer requests
 +
 
 +
The client is a java library that :
 +
*contacts the server asking for its capabilities in order to decide the best efficient transfer way between the two sides;
 +
*submits a transfer request through the selected transfer channel
 +
*monitors its status/outcome.
 +
 
 +
Optionally, a transfer request can specify an ordered set of plugin invocation, that are executed after the transfer has been completed. These execution are monitored by the client as part of the atomic transfer process.
 +
 
 +
The picture summarize the described architecture.
  
 
[[Image:DT2_Architecture.png|frame|center|Data Transfer Architecture]]
 
[[Image:DT2_Architecture.png|frame|center|Data Transfer Architecture]]

Revision as of 17:03, 9 September 2016

Overview

The implementation of a reliable data transfer mechanisms between the nodes of a gCube-based Hybrid Data Infrastructure is one of the main objectives when dealing with large set of multi-type datasets distributed across different repositories.

To promote an efficient and optimized consumption of these data resources, a number of components have been designed to meet the data transfer requirements.

This document outlines the design rationale, key features, and high-level architecture, the options for their deployment and as well some use cases.

Key features

The components belonging to this class are responsible for:

Point to Point transfer
direct transfer invocation to a gCube Node
reliable data transfer between Infrastructure Data Sources and Data Storages
by exploiting the uniform access interfaces provided by gCube and standard transfer protocols
automatic transfer optimization
by exploiting best available transfer options between invoker and target nodes
advanced and extensible post transfer processing
plugin - oriented implementation to serve advanced use case

Design

Philosophy

In a Hybrid Data e-Infrastructure, transferring data between nodes is a crucial feature. Given the heterogeneous nature of datasets, applications, networks capabilities and restrictions, efficient data-transfers can be problematic. Negotiation of best-efficient-available-transfer method is essential in order to achieve a global efficiency, while an extensible set of post operations lets application serve specific needs among common ones. We wanted to provide a light-weight solution that could fit the evolving nature of the infrastructure and its communities needs.

Architecture

The architecture of Data Transfer is a simple client server model, respectively implemented by data-transfer-library and data-transfer-service.

The service is a simple REST server, implemented as a gCube SmartGear web application. It is designed exploiting the plugin design pattern, so that it can be easily extended without introducing hard dependencies between the architecture's components. It also exposes interfaces aimed to :

  • get node's capabilities in terms of available plugins and supported transfer method;
  • submit and monitor transfer requests

The client is a java library that :

  • contacts the server asking for its capabilities in order to decide the best efficient transfer way between the two sides;
  • submits a transfer request through the selected transfer channel
  • monitors its status/outcome.

Optionally, a transfer request can specify an ordered set of plugin invocation, that are executed after the transfer has been completed. These execution are monitored by the client as part of the atomic transfer process.

The picture summarize the described architecture.

Data Transfer Architecture

Deployment

Large Deployment

Small Deployment

Use Cases

Well suited use cases

Less well suited use cases