Difference between revisions of "Data Access and Storage APIs"

From Gcube Wiki
Jump to: navigation, search
(→‎SPD API)
(→‎SPD Plugin API)
 
(19 intermediate revisions by 5 users not shown)
Line 1: Line 1:
−
Data Access and Storage APIs support client access and back-end access to a range of structured and unstructured data sources available in the Data e-Infrastructure. APIs include remote Web Service APIs, local client APIs, data model implementations, auxiliary APIs, and development frameworks for service plugins.  
+
Data Access and Storage APIs support both client and back-end access to a range of structured and unstructured data sources available in the Data e-Infrastructure. APIs include remote Web Service APIs, local client APIs, data model implementations, auxiliary APIs, and development frameworks for service plugins.  
  
−
This document lists all the available APIs in this areas along with their key features, pointing to external documentation for details on their design and usage.  
+
This document lists all the available APIs in these areas along with their key features, pointing to external documentation for details on their design and usage.  
  
 
==API Overview==
 
==API Overview==
Line 28: Line 28:
 
| [[#View Manager API|View Manager]]||Views over data sources accessible via [[#Tree Manager API|Tree Manager]] endpoints||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
| [[#View Manager API|View Manager]]||Views over data sources accessible via [[#Tree Manager API|Tree Manager]] endpoints||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
|-
 
|-
−
| [[#View Manager Client API|View Manager Client]]||Client proxies for [[#View Manager API|View Manager]] endpoints||Java||Java||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#View Manager Client API|View Manager Client]]||Client proxies for [[#View Manager API|View Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|-
 
|-
−
| [[#Storage Manager API|Storage Manager]]||TODO||Java||Java||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#Storage Manager API|Storage Manager]]||Access and storage of unstructured bytestreams, or files implementation||Java||Local||none
 
|-
 
|-
 
| [[#SPD API|Species Products Discovery (SPD)]]||Read access to pluggable sources of biodiversity data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
| [[#SPD API|Species Products Discovery (SPD)]]||Read access to pluggable sources of biodiversity data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
Line 38: Line 38:
 
| [[#SPD Client API|SPD Client]]||Client proxies for [[#SPD API|SPD]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
| [[#SPD Client API|SPD Client]]||Client proxies for [[#SPD API|SPD]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|}
 
|}
 +
 +
 +
 +
The picture below offers an immediate view of the area and where each component is placed and which other interfaces:
 +
 +
 +
[[File:DataManagementAPIs.png]]
  
 
==Tree Manager API==
 
==Tree Manager API==
Line 162: Line 169:
  
 
==Storage Manager API==
 
==Storage Manager API==
−
* '''Description''': TODO
+
* '''Description''': A client library for remote storage of files. The library allows clients to download, upload, remove, add, and list files. The file storage is based on MongoDB technology
 
* '''Type''': Java
 
* '''Type''': Java
−
* '''Protocol''': Local
+
* '''Protocol''': Custom
 
* [[Storage_Management_NEW|'''Documentation''']]
 
* [[Storage_Management_NEW|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:*''TODO''
+
:*''structured file storage: Clients can create folder hierarchies''
−
:*''TODO''
+
:*''secure file storage: File access is authenticated, every files can have private, group, or public access rights''
 +
:*''scalable file storage: files are stored in chunks and chunks are distributed across clusters of servers based on the workload of individual servers;''
 +
:*''fault-tolerant file storage: files are asynchronously replicated across cluster servers for data recovery and redundancy''
  
 
==SPD API==
 
==SPD API==
Line 175: Line 184:
 
* '''Protocol''': SOAP
 
* '''Protocol''': SOAP
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
−
* '''Documentation'''
+
* [http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases/org/gcube/data/spd/species-products-discovery-service/2.3.2-4.0.0-130321/species-products-discovery-service-2.3.2-4.0.0-130321-javadoc.jar|'''Documentation''']
 
* '''Key Features'''
 
* '''Key Features'''
−
:* access API tailored to biodiversity data sources;
+
:* ''access API tailored to biodiversity data sources''
−
:* dynamically pluggable architecture of transformations (eg unified classification, synonyms presentation) from external sources of biodiversity data;
+
:* ''dynamically pluggable architecture of transformations (eg unified classification, synonyms presentation) from external sources of biodiversity data''
−
:* plugins for key biodiversity data sources, including OBIS, GBIF and Catalogue of Life;
+
:* ''plugins for key biodiversity data sources, including OBIS, GBIF and Catalogue of Life''
−
:* dynamic clustering of discovered items;
+
:* ''dynamic clustering of discovered items''
−
:* integration with workspace facilities;
+
:* ''integration with workspace facilities''
−
:* export of discovered items in multiple formats, including DarwinCore and DarwinCore Archive;
+
:* ''export of discovered items in multiple formats, including DarwinCore and DarwinCore Archive''
  
 
==SPD Plugin API==
 
==SPD Plugin API==
Line 188: Line 197:
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* '''Documentation'''
+
* [http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases/org/gcube/data/spd/spd-plugin-framework/3.0.0-4.0.0-126523/spd-plugin-framework-3.0.0-4.0.0-126523-javadoc.jar|'''Documentation''']
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''support for spd plugins implementation''
−
:TODO
+
:* ''partial interface implementations''
  
 
==SPD Client API==
 
==SPD Client API==
Line 198: Line 207:
 
* '''Protocol''': Local
 
* '''Protocol''': Local
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
−
* '''Documentation'''
+
* [[Species_Product_Discovery:_client_library|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''transparent endpoint discovery and caching''
−
:TODO
+
:* ''transparent fault tolerance over endpoint replicas''

Latest revision as of 17:38, 12 September 2016

Data Access and Storage APIs support both client and back-end access to a range of structured and unstructured data sources available in the Data e-Infrastructure. APIs include remote Web Service APIs, local client APIs, data model implementations, auxiliary APIs, and development frameworks for service plugins.

This document lists all the available APIs in these areas along with their key features, pointing to external documentation for details on their design and usage.

API Overview

Name Description Type Protocol Framework
Tree Manager Tree-based CRUD operations over pluggable sources of structured data WS SOAP gCore
Tree Manager Client Client proxies for Tree Manager endpoints Java Local CL
Tree Manager Plugin Development framework for Tree Manager plugins Java Local none
Trees Tree and tree pattern model implementations Java Local none
Streams Conversion, transformation, and publication of data streams Java Local none
gDoc gDoc model implementation Java Local none
gDoc Client gDoc-based client proxies for Tree Manager endpoints Java Local CL
View Manager Views over data sources accessible via Tree Manager endpoints WS SOAP gCore
View Manager Client Client proxies for View Manager endpoints Java Local CL
Storage Manager Access and storage of unstructured bytestreams, or files implementation Java Local none
Species Products Discovery (SPD) Read access to pluggable sources of biodiversity data WS SOAP gCore
SPD Plugin Development framework for SPD plugins Java Local CL
SPD Client Client proxies for SPD endpoints Java Local CL


The picture below offers an immediate view of the area and where each component is placed and which other interfaces:


DataManagementAPIs.png

Tree Manager API

  • Description: Tree-based CRUD operations over pluggable sources of structured data.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • lookup of whole trees as well as individual nodes
  • tree pattern matching on lookups and queries
  • in-place updates
  • streaming on all operations
  • native storage based on graph database technology

Tree Manager Client API

  • high-level proxy API
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
  • polling and notifications for asynchronous bindings
  • fluent proxy configuration API

Tree Manager Plugin API

  • support for a wide range of plugins: source-specific, source-generic, type-specific, type-generic, multi-type, etc
  • interface-based POJO framework, with no dependency on service implementation
  • transparent property publication and state persistence"
  • callback-based and event-based interaction with service"
  • full lifecycle support
  • partial interface implementations
  • derived implementation of node-based and streamed operations

Trees API

  • Description: Tree and tree pattern model implementations.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • untyped tree model with attributed nodes, labelled edges, and string-valued leaves
  • pattern language for characterisation of tree topology and leaf values
  • tree matching and pruning operations based on pattern language
  • tree difference and in-place update operations
  • fluent API for tree construction, tree navigation, and tree pattern construction
  • predefined XML bindings for trees and patterns
  • synthetic tree generation facilities for load testing

Streams API

  • Description: Conversion, transformation, and publication of data streams.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • high-level stream model based on Iterator-style interfaces
  • predefined implementations over standard Iterators and gRS2 resultsets
  • piping, folding, and unfolding stream operations
  • configurable error handling policies
  • stream lifecycle event notifications
  • publication as gRS2 resultset
  • fluent API for all stream operations

gDoc API

  • Description: gDoc model implementation.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • document model inclusive of metadata, annotations, parts, and alternatives
  • strongly-typed document construction
  • transparent bytestream resolution
  • annotation threading
  • new and proxy document modes
  • change tracking and staging for in-place updates without prior reads
  • XML binding facilities

gDoc Client API

  • creation of and access to gDoc document sources
  • bulk and streamed operations
  • caching, selection, and pruning for read operations
  • creation, discovery, and use of document views

View Manager API

  • Description: Views over data sources accessible via Tree Manager endpoints.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • view definition through arbitrary tree patterns
  • view characterisation through arbitrary properties
  • view publication in infrastructure
  • computation of dynamic view properties (cardinality, last update)

View Manager Client API

  • high-level proxy API
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
  • fluent proxy configuration API

Storage Manager API

  • Description: A client library for remote storage of files. The library allows clients to download, upload, remove, add, and list files. The file storage is based on MongoDB technology
  • Type: Java
  • Protocol: Custom
  • Documentation
  • Key Features
  • structured file storage: Clients can create folder hierarchies
  • secure file storage: File access is authenticated, every files can have private, group, or public access rights
  • scalable file storage: files are stored in chunks and chunks are distributed across clusters of servers based on the workload of individual servers;
  • fault-tolerant file storage: files are asynchronously replicated across cluster servers for data recovery and redundancy

SPD API

  • Description: Read access to pluggable sources of biodiversity data.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • access API tailored to biodiversity data sources
  • dynamically pluggable architecture of transformations (eg unified classification, synonyms presentation) from external sources of biodiversity data
  • plugins for key biodiversity data sources, including OBIS, GBIF and Catalogue of Life
  • dynamic clustering of discovered items
  • integration with workspace facilities
  • export of discovered items in multiple formats, including DarwinCore and DarwinCore Archive

SPD Plugin API

  • Description: Development framework for SPD plugins.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • support for spd plugins implementation
  • partial interface implementations

SPD Client API

  • Description: Client proxies for SPD endpoints.
  • Type: Java
  • Protocol: Local
  • Framework: CL
  • Documentation
  • Key Features
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
Retrieved from "https://wiki.gcube-system.org/index.php?title=Data_Access_and_Storage_APIs&oldid=26698"