Data Transfer APIs

From Gcube Wiki
Jump to: navigation, search

Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. The document outlines the design rationale for those APIs.

Overview

APIs

Name Description Type Protocol Framework
Data Transfer Agent Library The CL proxy for the Data Transfer Agent Service operations Java Local CL
Data Transfer Agent Service Structured/Unstructured transfer operations among data Source/Storages gCube Web Service SOAP gCore
gRS2 Broker This component exposes the gCube Resultset as an http endpoint Java Servlet REST HTTP Front End

Data Transfer Agent Library

  • Description: The CL for the Data Transfer Agent operations
  • Type: Java
  • Protocol: Local
  • Framework: CL
  • Key Features:
  • synchronous/asynchronous transfers over data transfer service operations
  • transfer monitoring
  • transfer outcomes to clients

Data Transfer Agent Service

  • Description: Structured/Unstructured transfer operations among data Source/Storages
  • Type: gCube Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Key Features:
  • transfer of local files from an external/internal client to a remote GHN
  • file transfer from a remote Data Source to a remote GHN using standard protocols ( HTTP, FTP, BitTorrent)
  • file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocols ( HTTP, FTP, BitTorrent)
  • tree based data transfer from a remote Data Source to a remote tree-based data storage
  • transfer statistics publication trough messaging

gRS2 Broker

  • Description: This component exposes the gCube Resultset as an http endpoint
  • Type: Java Servlet
  • Protocol: REST
  • Framework: HTTP Front End
  • Key Features:
  • results are exposed as an HTTP endpoint
  • results are in XML format
  • server response is compressed in order to achieve performance benefits
  • the records are retrieved iteratively - gradually
  • number of results can be customized (max parameter)
  • files are stored locally on the server (garbage collection)
  • Wiki Doc: