Data Transfer APIs

Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. The document outlines the design rationale for those APIs.

Overview

APIs

Name	Description	Type	Protocol	Framework
Data Transfer Agent Library	The CL proxy for the Data Transfer Agent Service operations	Java	Local	CL
Data Transfer Agent Service	Structured/Unstructured transfer operations among data Source/Storages	gCube Web Service	SOAP	gCore
gRS2 Broker	This component exposes the gCube Resultset as an http endpoint	Java Servlet	REST	HTTP Front End

Data Transfer Agent Library

• Description: The CL for the Data Transfer Agent operations
• Type: Java
• Protocol: Local
• Framework: CL
• Key Features:

• synchronous/asynchronous transfers over data transfer service operations
• transfer monitoring
• transfer outcomes to clients

• Documentation

Data Transfer Agent Service

• Description: Structured/Unstructured transfer operations among data Source/Storages
• Type: gCube Web Service
• Protocol: SOAP
• Framework: gCore
• Key Features:

• transfer of local files from an external/internal client to a remote GHN
• file transfer from a remote Data Source to a remote GHN using standard protocols ( HTTP, FTP, BitTorrent)
• file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocols ( HTTP, FTP, BitTorrent)
• tree based data transfer from a remote Data Source to a remote tree-based data storage
• transfer statistics publication trough messaging

• Documentation

gRS2 Broker

• Description: This component exposes the gCube Resultset as an http endpoint
• Type: Java Servlet
• Protocol: REST
• Framework: HTTP Front End
• Key Features:

• results are exposed as an HTTP endpoint
• results are in XML format
• server response is compressed in order to achieve performance benefits
• the records are retrieved iteratively - gradually
• number of results can be customized (max parameter)
• files are stored locally on the server (garbage collection)

• Wiki Doc: