Difference between revisions of "Data Transfer APIs"
From Gcube Wiki
(→gRS2 Broker) |
|||
| (25 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. | Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. | ||
| − | The document outlines the design | + | The document outlines the design rationale for those APIs. |
==Overview== | ==Overview== | ||
| Line 11: | Line 11: | ||
| align="center" style="background:#f0f0f0;"|'''Framework''' | | align="center" style="background:#f0f0f0;"|'''Framework''' | ||
|- | |- | ||
| − | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Library Agent Library]||||Java||Local||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | + | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Library Data Transfer Agent Library]||The CL proxy for the [[#Data_Transfer_Agent_Service |Data Transfer Agent Service]] operations||Java||Local||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] |
|- | |- | ||
| − | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Service Data Transfer Agent Service]|| | + | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Service Data Transfer Agent Service]|| Structured/Unstructured transfer operations among data Source/Storages||gCube Web Service||SOAP||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_WSRF_API_Framework_Specification gCore] |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
|- | |- | ||
| [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#gRS2_Broker gRS2 Broker]||This component exposes the gCube Resultset as an http endpoint||Java Servlet||REST||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_HTTP_API_Framework_Specification HTTP Front End] | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#gRS2_Broker gRS2 Broker]||This component exposes the gCube Resultset as an http endpoint||Java Servlet||REST||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_HTTP_API_Framework_Specification HTTP Front End] | ||
| Line 24: | Line 20: | ||
==Data Transfer Agent Library== | ==Data Transfer Agent Library== | ||
| − | * '''Description''': | + | * '''Description''': The CL for the Data Transfer Agent operations |
* '''Type''': Java | * '''Type''': Java | ||
* '''Protocol''': Local | * '''Protocol''': Local | ||
* '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | * '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | ||
* '''Key Features''': | * '''Key Features''': | ||
| − | * [Data_Transfer_Agent|'''Documentation'''] | + | :* ''synchronous/asynchronous transfers over data transfer service operations'' |
| + | :* ''transfer monitoring'' | ||
| + | :* ''transfer outcomes to clients'' | ||
| + | * [[Data_Transfer_Agent#Data_Transfer_Agent_Library|'''Documentation''']] | ||
| + | |||
==Data Transfer Agent Service== | ==Data Transfer Agent Service== | ||
| − | * '''Description''': | + | * '''Description''': Structured/Unstructured transfer operations among data Source/Storages |
* '''Type''': gCube Web Service | * '''Type''': gCube Web Service | ||
* '''Protocol''': SOAP | * '''Protocol''': SOAP | ||
* '''Framework''': gCore | * '''Framework''': gCore | ||
* '''Key Features''': | * '''Key Features''': | ||
| − | + | :* ''transfer of local files from an external/internal client to a remote GHN'' | |
| − | + | :* ''file transfer from a remote Data Source to a remote GHN using standard protocols ( HTTP, FTP, BitTorrent)'' | |
| − | + | :* ''file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocols ( HTTP, FTP, BitTorrent)'' | |
| − | * ''' | + | :* ''tree based data transfer from a remote Data Source to a remote tree-based data storage'' |
| − | + | :* ''transfer statistics publication trough messaging'' | |
| − | * ''' | + | * [[Data_Transfer_Agent#Data_Transfer_Agent_Service|'''Documentation''']] |
| − | + | ||
| − | * '' | + | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | * ''' | + | |
| − | + | ||
| − | * ''' | + | |
| − | * [Data_Transfer_Agent|'''Documentation'''] | + | |
==gRS2 Broker== | ==gRS2 Broker== | ||
* '''Description''': This component exposes the gCube Resultset as an http endpoint | * '''Description''': This component exposes the gCube Resultset as an http endpoint | ||
| Line 57: | Line 49: | ||
* '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_HTTP_API_Framework_Specification HTTP Front End] | * '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_HTTP_API_Framework_Specification HTTP Front End] | ||
* '''Key Features''': | * '''Key Features''': | ||
| − | * | + | :* ''results are exposed as an HTTP endpoint'' |
| + | :* ''results are in XML format'' | ||
| + | :* ''server response is compressed in order to achieve performance benefits'' | ||
| + | :* ''the records are retrieved iteratively - gradually'' | ||
| + | :* ''number of results can be customized (max parameter)'' | ||
| + | :* ''files are stored locally on the server (garbage collection)'' | ||
| + | * Wiki Doc: | ||
Latest revision as of 19:15, 2 July 2012
Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. The document outlines the design rationale for those APIs.
Contents
Overview
APIs
| Name | Description | Type | Protocol | Framework |
| Data Transfer Agent Library | The CL proxy for the Data Transfer Agent Service operations | Java | Local | CL |
| Data Transfer Agent Service | Structured/Unstructured transfer operations among data Source/Storages | gCube Web Service | SOAP | gCore |
| gRS2 Broker | This component exposes the gCube Resultset as an http endpoint | Java Servlet | REST | HTTP Front End |
Data Transfer Agent Library
- Description: The CL for the Data Transfer Agent operations
- Type: Java
- Protocol: Local
- Framework: CL
- Key Features:
- synchronous/asynchronous transfers over data transfer service operations
- transfer monitoring
- transfer outcomes to clients
Data Transfer Agent Service
- Description: Structured/Unstructured transfer operations among data Source/Storages
- Type: gCube Web Service
- Protocol: SOAP
- Framework: gCore
- Key Features:
- transfer of local files from an external/internal client to a remote GHN
- file transfer from a remote Data Source to a remote GHN using standard protocols ( HTTP, FTP, BitTorrent)
- file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocols ( HTTP, FTP, BitTorrent)
- tree based data transfer from a remote Data Source to a remote tree-based data storage
- transfer statistics publication trough messaging
gRS2 Broker
- Description: This component exposes the gCube Resultset as an http endpoint
- Type: Java Servlet
- Protocol: REST
- Framework: HTTP Front End
- Key Features:
- results are exposed as an HTTP endpoint
- results are in XML format
- server response is compressed in order to achieve performance benefits
- the records are retrieved iteratively - gradually
- number of results can be customized (max parameter)
- files are stored locally on the server (garbage collection)
- Wiki Doc:
