Difference between revisions of "Data Transfer APIs"
From Gcube Wiki
Andrea.manzi (Talk | contribs) (→Data Transfer Agent Service) |
(→gRS2 Broker) |
||
| (11 intermediate revisions by one other user not shown) | |||
| Line 1: | Line 1: | ||
Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. | Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. | ||
| − | The document outlines the design | + | The document outlines the design rationale for those APIs. |
==Overview== | ==Overview== | ||
| Line 11: | Line 11: | ||
| align="center" style="background:#f0f0f0;"|'''Framework''' | | align="center" style="background:#f0f0f0;"|'''Framework''' | ||
|- | |- | ||
| − | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Library Data Transfer Agent Library]||The CL for the Data Transfer Agent operations||Java||Local||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | + | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Library Data Transfer Agent Library]||The CL proxy for the [[#Data_Transfer_Agent_Service |Data Transfer Agent Service]] operations||Java||Local||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] |
|- | |- | ||
| [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Service Data Transfer Agent Service]|| Structured/Unstructured transfer operations among data Source/Storages||gCube Web Service||SOAP||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_WSRF_API_Framework_Specification gCore] | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Service Data Transfer Agent Service]|| Structured/Unstructured transfer operations among data Source/Storages||gCube Web Service||SOAP||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_WSRF_API_Framework_Specification gCore] | ||
| Line 25: | Line 25: | ||
* '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | * '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | ||
* '''Key Features''': | * '''Key Features''': | ||
| − | :* '' | + | :* ''synchronous/asynchronous transfers over data transfer service operations'' |
| − | :* '' | + | :* ''transfer monitoring'' |
| − | :* '' | + | :* ''transfer outcomes to clients'' |
* [[Data_Transfer_Agent#Data_Transfer_Agent_Library|'''Documentation''']] | * [[Data_Transfer_Agent#Data_Transfer_Agent_Library|'''Documentation''']] | ||
| Line 37: | Line 37: | ||
* '''Key Features''': | * '''Key Features''': | ||
:* ''transfer of local files from an external/internal client to a remote GHN'' | :* ''transfer of local files from an external/internal client to a remote GHN'' | ||
| − | :* | + | :* ''file transfer from a remote Data Source to a remote GHN using standard protocols ( HTTP, FTP, BitTorrent)'' |
| − | :* ''file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard | + | :* ''file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocols ( HTTP, FTP, BitTorrent)'' |
:* ''tree based data transfer from a remote Data Source to a remote tree-based data storage'' | :* ''tree based data transfer from a remote Data Source to a remote tree-based data storage'' | ||
| − | :* ''transfer statistics | + | :* ''transfer statistics publication trough messaging'' |
* [[Data_Transfer_Agent#Data_Transfer_Agent_Service|'''Documentation''']] | * [[Data_Transfer_Agent#Data_Transfer_Agent_Service|'''Documentation''']] | ||
| Line 49: | Line 49: | ||
* '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_HTTP_API_Framework_Specification HTTP Front End] | * '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_HTTP_API_Framework_Specification HTTP Front End] | ||
* '''Key Features''': | * '''Key Features''': | ||
| + | :* ''results are exposed as an HTTP endpoint'' | ||
| + | :* ''results are in XML format'' | ||
| + | :* ''server response is compressed in order to achieve performance benefits'' | ||
| + | :* ''the records are retrieved iteratively - gradually'' | ||
| + | :* ''number of results can be customized (max parameter)'' | ||
| + | :* ''files are stored locally on the server (garbage collection)'' | ||
* Wiki Doc: | * Wiki Doc: | ||
Latest revision as of 19:15, 2 July 2012
Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. The document outlines the design rationale for those APIs.
Contents
Overview
APIs
| Name | Description | Type | Protocol | Framework |
| Data Transfer Agent Library | The CL proxy for the Data Transfer Agent Service operations | Java | Local | CL |
| Data Transfer Agent Service | Structured/Unstructured transfer operations among data Source/Storages | gCube Web Service | SOAP | gCore |
| gRS2 Broker | This component exposes the gCube Resultset as an http endpoint | Java Servlet | REST | HTTP Front End |
Data Transfer Agent Library
- Description: The CL for the Data Transfer Agent operations
- Type: Java
- Protocol: Local
- Framework: CL
- Key Features:
- synchronous/asynchronous transfers over data transfer service operations
- transfer monitoring
- transfer outcomes to clients
Data Transfer Agent Service
- Description: Structured/Unstructured transfer operations among data Source/Storages
- Type: gCube Web Service
- Protocol: SOAP
- Framework: gCore
- Key Features:
- transfer of local files from an external/internal client to a remote GHN
- file transfer from a remote Data Source to a remote GHN using standard protocols ( HTTP, FTP, BitTorrent)
- file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocols ( HTTP, FTP, BitTorrent)
- tree based data transfer from a remote Data Source to a remote tree-based data storage
- transfer statistics publication trough messaging
gRS2 Broker
- Description: This component exposes the gCube Resultset as an http endpoint
- Type: Java Servlet
- Protocol: REST
- Framework: HTTP Front End
- Key Features:
- results are exposed as an HTTP endpoint
- results are in XML format
- server response is compressed in order to achieve performance benefits
- the records are retrieved iteratively - gradually
- number of results can be customized (max parameter)
- files are stored locally on the server (garbage collection)
- Wiki Doc:
