Difference between revisions of "Data Access and Storage APIs"

From Gcube Wiki
Jump to: navigation, search
(→‎SPD Plugin API)
 
(36 intermediate revisions by 6 users not shown)
Line 1: Line 1:
−
Data Access and Storage APIs support front-end and back-end access to a range of structured and unstructured data sources, both within and outside the infrastructure.
+
Data Access and Storage APIs support both client and back-end access to a range of structured and unstructured data sources available in the Data e-Infrastructure. APIs include remote Web Service APIs, local client APIs, data model implementations, auxiliary APIs, and development frameworks for service plugins.  
  
−
This document lists the available APIs and their key features, pointing to external documentation for details on their design and usage.  
+
This document lists all the available APIs in these areas along with their key features, pointing to external documentation for details on their design and usage.  
  
 
==API Overview==
 
==API Overview==
Line 12: Line 12:
 
| align="center" style="background:#f0f0f0;"|'''Framework'''
 
| align="center" style="background:#f0f0f0;"|'''Framework'''
 
|-
 
|-
−
| [[#Tree Manager API|Tree Manager]]||Tree-based CRUD operations over pluggable remote sources of structured data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
+
| [[#Tree Manager API|Tree Manager]]||Tree-based CRUD operations over pluggable sources of structured data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
|-
 
|-
 
| [[#Tree Manager Client API|Tree Manager Client]]||Client proxies for [[#Tree Manager API|Tree Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
| [[#Tree Manager Client API|Tree Manager Client]]||Client proxies for [[#Tree Manager API|Tree Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
Line 19: Line 19:
 
|-
 
|-
 
| [[#Trees API|Trees]]||Tree and tree pattern model implementations||Java||Local||none
 
| [[#Trees API|Trees]]||Tree and tree pattern model implementations||Java||Local||none
 +
|-
 +
| [[#Trees API|Streams]]||Conversion, transformation, and publication of data streams||Java||Local||none
 
|-
 
|-
 
| [[#gDoc API|gDoc]]||<code>gDoc</code> model implementation||Java||Local||none
 
| [[#gDoc API|gDoc]]||<code>gDoc</code> model implementation||Java||Local||none
Line 24: Line 26:
 
| [[#gDoc Client API|gDoc Client]]||<code>gDoc</code>-based client proxies for [[#Tree Manager API|Tree Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
| [[#gDoc Client API|gDoc Client]]||<code>gDoc</code>-based client proxies for [[#Tree Manager API|Tree Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|-
 
|-
−
| [[#View Manager API|View Manager]]||View management for data sources accessible via [[#Tree Manager API|Tree Manager]] endpoints||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
+
| [[#View Manager API|View Manager]]||Views over data sources accessible via [[#Tree Manager API|Tree Manager]] endpoints||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
|-
 
|-
−
| [[#View Manager Client API|View Manager Client]]||Client proxies for [[#View Manager API|View Manager]] endpoints||Java||Java||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#View Manager Client API|View Manager Client]]||Client proxies for [[#View Manager API|View Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|-
 
|-
−
| [[#Storage Manager API|Storage Manager]]||TODO||Java||Java||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#Storage Manager API|Storage Manager]]||Access and storage of unstructured bytestreams, or files implementation||Java||Local||none
 
|-
 
|-
−
| [[#SPD API|Species Products Discovery (SPD)]]||Read access to pluggable, remote sources of biodiversity data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
+
| [[#SPD API|Species Products Discovery (SPD)]]||Read access to pluggable sources of biodiversity data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
|-
 
|-
 
| [[#SPD Plugin API|SPD Plugin]]||Development framework for [[#SPD API|SPD]] plugins||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
| [[#SPD Plugin API|SPD Plugin]]||Development framework for [[#SPD API|SPD]] plugins||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
Line 36: Line 38:
 
| [[#SPD Client API|SPD Client]]||Client proxies for [[#SPD API|SPD]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
| [[#SPD Client API|SPD Client]]||Client proxies for [[#SPD API|SPD]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|}
 
|}
 +
 +
 +
 +
The picture below offers an immediate view of the area and where each component is placed and which other interfaces:
 +
 +
 +
[[File:DataManagementAPIs.png]]
  
 
==Tree Manager API==
 
==Tree Manager API==
−
* '''Description''': Tree-based CRUD operations over pluggable remote sources of structured data.
+
* '''Description''': Tree-based CRUD operations over pluggable sources of structured data.
 
* '''Type''': Web Service
 
* '''Type''': Web Service
 
* '''Protocol''': SOAP
 
* '''Protocol''': SOAP
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
−
* [[The_Tree_Manager|'''Wiki Documentation''']]
+
* [[The_Tree_Manager|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
 
:* ''lookup of whole trees as well as individual nodes''
 
:* ''lookup of whole trees as well as individual nodes''
Line 56: Line 65:
 
* '''Protocol''': Local
 
* '''Protocol''': Local
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
−
* [[The_Tree_Manager_Library|'''Wiki Documentation''']]
+
* [[The_Tree_Manager_Library|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
 
:* ''high-level proxy API''
 
:* ''high-level proxy API''
Line 65: Line 74:
  
 
==Tree Manager Plugin API==
 
==Tree Manager Plugin API==
 +
 
* '''Description''': Development framework for [[#Tree Manager API|Tree Manager]] plugins.
 
* '''Description''': Development framework for [[#Tree Manager API|Tree Manager]] plugins.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* [[The_Tree_Manager_Framework|'''Wiki Documentation''']]
+
* [[The_Tree_Manager_Framework|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''support for a wide range of plugins: source-specific, source-generic, type-specific, type-generic, multi-type, etc''
−
:TODO
+
:* ''interface-based POJO framework, with no dependency on service implementation''
 +
:* ''transparent property publication and state persistence"
 +
:* ''callback-based and event-based interaction with service"
 +
:* ''full lifecycle support''
 +
:* ''partial interface implementations''
 +
:* ''derived implementation of node-based and streamed operations''
  
 
==Trees API==
 
==Trees API==
Line 77: Line 92:
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* [[The_Trees_Library|'''Wiki Documentation''']]
+
* [[The_Trees_Library|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''untyped tree model with attributed nodes, labelled edges, and string-valued leaves''
−
:TODO
+
:* ''pattern language for characterisation of tree topology and leaf values''
 +
:* ''tree matching and pruning operations based on pattern language''
 +
:* ''tree difference and in-place update operations''
 +
:* ''fluent API for tree construction, tree navigation, and tree pattern construction''
 +
:* ''predefined XML bindings for trees and patterns''
 +
:* ''synthetic tree generation facilities for load testing''
 +
 
 +
==Streams API==
 +
 
 +
* '''Description''': Conversion, transformation, and publication of data streams.
 +
* '''Type''': Java
 +
* '''Protocol''': Local
 +
* [[The_Streams_Library|'''Documentation''']]
 +
* '''Key Features'''
 +
:* ''high-level stream model based on <code>Iterator</code>-style interfaces''
 +
:* ''predefined implementations over standard <code>Iterator</code>s and <code>gRS2</code> resultsets''
 +
:* ''piping, folding, and unfolding stream operations''
 +
:* ''configurable error handling policies''
 +
:* ''stream lifecycle event notifications''
 +
:* ''publication as <code>gRS2</code> resultset''
 +
:* ''fluent API for all stream operations''
  
 
==gDoc API==
 
==gDoc API==
 +
 
* '''Description''': <code>gDoc</code> model implementation.
 
* '''Description''': <code>gDoc</code> model implementation.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* [[GCube_Document_Model|'''Wiki Documentation''']]
+
* [[GCube_Document_Model|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''document model inclusive of metadata, annotations, parts, and alternatives''
−
:TODO
+
:* ''strongly-typed document construction''
 +
:* ''transparent bytestream resolution''
 +
:* ''annotation threading''
 +
:* ''new and proxy document modes''
 +
:* ''change tracking and staging for in-place updates without prior reads''
 +
:* ''XML binding facilities''
  
 
==gDoc Client API==
 
==gDoc Client API==
 +
 
* '''Description''': <code>gDoc</code>-based client proxies for [[#Tree Manager API|Tree Manager]] endpoints.
 
* '''Description''': <code>gDoc</code>-based client proxies for [[#Tree Manager API|Tree Manager]] endpoints.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* [[GCube_Document_Library|'''Wiki Documentation''']]
+
* [[GCube Document Library (2.0)|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''creation of and access to <code>gDoc</code> document sources''
−
:TODO
+
:* ''bulk and streamed operations''
 +
:* ''caching, selection, and pruning for read operations''
 +
:* ''creation, discovery, and use of document views''
  
 
==View Manager API ==
 
==View Manager API ==
−
* '''Description''': View management for data sources accessible via [[#Tree Manager API|Tree Manager]] endpoints.
+
* '''Description''': Views over data sources accessible via [[#Tree Manager API|Tree Manager]] endpoints.
 
* '''Type''': Web Service
 
* '''Type''': Web Service
 
* '''Protocol''': SOAP
 
* '''Protocol''': SOAP
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
−
* [[View_Manager|'''Wiki Documentation''']]
+
* [[View_Manager|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''view definition through arbitrary tree patterns''
−
:TODO
+
:* ''view characterisation through arbitrary properties''
 +
:* ''view publication in infrastructure''
 +
:* ''computation of dynamic view properties (cardinality, last update)''
  
 
==View Manager Client API==
 
==View Manager Client API==
Line 115: Line 161:
 
* '''Protocol''': Local
 
* '''Protocol''': Local
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
−
* [[View_Manager|'''Wiki Documentation''']]
+
* [[View_Manager|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''high-level proxy API''
−
:TODO
+
:* ''transparent endpoint discovery and caching''
 +
:* ''transparent fault tolerance over endpoint replicas''
 +
:* ''fluent proxy configuration API''
  
 
==Storage Manager API==
 
==Storage Manager API==
−
* '''Description''': TODO
+
* '''Description''': A client library for remote storage of files. The library allows clients to download, upload, remove, add, and list files. The file storage is based on MongoDB technology
 
* '''Type''': Java
 
* '''Type''': Java
−
* '''Protocol''': Local
+
* '''Protocol''': Custom
−
* [[Storage_Management_NEW|'''Wiki Documentation''']]
+
* [[Storage_Management_NEW|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:*''structured file storage: Clients can create folder hierarchies''
−
:TODO
+
:*''secure file storage: File access is authenticated, every files can have private, group, or public access rights''
 +
:*''scalable file storage: files are stored in chunks and chunks are distributed across clusters of servers based on the workload of individual servers;''
 +
:*''fault-tolerant file storage: files are asynchronously replicated across cluster servers for data recovery and redundancy''
  
 
==SPD API==
 
==SPD API==
−
* '''Description''': Read access to pluggable, remote sources of biodiversity data.
+
* '''Description''': Read access to pluggable sources of biodiversity data.
 
* '''Type''': Web Service
 
* '''Type''': Web Service
 
* '''Protocol''': SOAP
 
* '''Protocol''': SOAP
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
−
* '''Wiki Documentation'''
+
* [http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases/org/gcube/data/spd/species-products-discovery-service/2.3.2-4.0.0-130321/species-products-discovery-service-2.3.2-4.0.0-130321-javadoc.jar|'''Documentation''']
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''access API tailored to biodiversity data sources''
−
:TODO
+
:* ''dynamically pluggable architecture of transformations (eg unified classification, synonyms presentation) from external sources of biodiversity data''
 +
:* ''plugins for key biodiversity data sources, including OBIS, GBIF and Catalogue of Life''
 +
:* ''dynamic clustering of discovered items''
 +
:* ''integration with workspace facilities''
 +
:* ''export of discovered items in multiple formats, including DarwinCore and DarwinCore Archive''
  
 
==SPD Plugin API==
 
==SPD Plugin API==
Line 143: Line 197:
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* '''Wiki Documentation'''
+
* [http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases/org/gcube/data/spd/spd-plugin-framework/3.0.0-4.0.0-126523/spd-plugin-framework-3.0.0-4.0.0-126523-javadoc.jar|'''Documentation''']
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''support for spd plugins implementation''
−
:TODO
+
:* ''partial interface implementations''
  
 
==SPD Client API==
 
==SPD Client API==
Line 153: Line 207:
 
* '''Protocol''': Local
 
* '''Protocol''': Local
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
−
* '''Wiki Documentation'''
+
* [[Species_Product_Discovery:_client_library|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''transparent endpoint discovery and caching''
−
:TODO
+
:* ''transparent fault tolerance over endpoint replicas''

Latest revision as of 17:38, 12 September 2016

Data Access and Storage APIs support both client and back-end access to a range of structured and unstructured data sources available in the Data e-Infrastructure. APIs include remote Web Service APIs, local client APIs, data model implementations, auxiliary APIs, and development frameworks for service plugins.

This document lists all the available APIs in these areas along with their key features, pointing to external documentation for details on their design and usage.

API Overview

Name Description Type Protocol Framework
Tree Manager Tree-based CRUD operations over pluggable sources of structured data WS SOAP gCore
Tree Manager Client Client proxies for Tree Manager endpoints Java Local CL
Tree Manager Plugin Development framework for Tree Manager plugins Java Local none
Trees Tree and tree pattern model implementations Java Local none
Streams Conversion, transformation, and publication of data streams Java Local none
gDoc gDoc model implementation Java Local none
gDoc Client gDoc-based client proxies for Tree Manager endpoints Java Local CL
View Manager Views over data sources accessible via Tree Manager endpoints WS SOAP gCore
View Manager Client Client proxies for View Manager endpoints Java Local CL
Storage Manager Access and storage of unstructured bytestreams, or files implementation Java Local none
Species Products Discovery (SPD) Read access to pluggable sources of biodiversity data WS SOAP gCore
SPD Plugin Development framework for SPD plugins Java Local CL
SPD Client Client proxies for SPD endpoints Java Local CL


The picture below offers an immediate view of the area and where each component is placed and which other interfaces:


DataManagementAPIs.png

Tree Manager API

  • Description: Tree-based CRUD operations over pluggable sources of structured data.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • lookup of whole trees as well as individual nodes
  • tree pattern matching on lookups and queries
  • in-place updates
  • streaming on all operations
  • native storage based on graph database technology

Tree Manager Client API

  • high-level proxy API
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
  • polling and notifications for asynchronous bindings
  • fluent proxy configuration API

Tree Manager Plugin API

  • support for a wide range of plugins: source-specific, source-generic, type-specific, type-generic, multi-type, etc
  • interface-based POJO framework, with no dependency on service implementation
  • transparent property publication and state persistence"
  • callback-based and event-based interaction with service"
  • full lifecycle support
  • partial interface implementations
  • derived implementation of node-based and streamed operations

Trees API

  • Description: Tree and tree pattern model implementations.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • untyped tree model with attributed nodes, labelled edges, and string-valued leaves
  • pattern language for characterisation of tree topology and leaf values
  • tree matching and pruning operations based on pattern language
  • tree difference and in-place update operations
  • fluent API for tree construction, tree navigation, and tree pattern construction
  • predefined XML bindings for trees and patterns
  • synthetic tree generation facilities for load testing

Streams API

  • Description: Conversion, transformation, and publication of data streams.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • high-level stream model based on Iterator-style interfaces
  • predefined implementations over standard Iterators and gRS2 resultsets
  • piping, folding, and unfolding stream operations
  • configurable error handling policies
  • stream lifecycle event notifications
  • publication as gRS2 resultset
  • fluent API for all stream operations

gDoc API

  • Description: gDoc model implementation.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • document model inclusive of metadata, annotations, parts, and alternatives
  • strongly-typed document construction
  • transparent bytestream resolution
  • annotation threading
  • new and proxy document modes
  • change tracking and staging for in-place updates without prior reads
  • XML binding facilities

gDoc Client API

  • creation of and access to gDoc document sources
  • bulk and streamed operations
  • caching, selection, and pruning for read operations
  • creation, discovery, and use of document views

View Manager API

  • Description: Views over data sources accessible via Tree Manager endpoints.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • view definition through arbitrary tree patterns
  • view characterisation through arbitrary properties
  • view publication in infrastructure
  • computation of dynamic view properties (cardinality, last update)

View Manager Client API

  • high-level proxy API
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
  • fluent proxy configuration API

Storage Manager API

  • Description: A client library for remote storage of files. The library allows clients to download, upload, remove, add, and list files. The file storage is based on MongoDB technology
  • Type: Java
  • Protocol: Custom
  • Documentation
  • Key Features
  • structured file storage: Clients can create folder hierarchies
  • secure file storage: File access is authenticated, every files can have private, group, or public access rights
  • scalable file storage: files are stored in chunks and chunks are distributed across clusters of servers based on the workload of individual servers;
  • fault-tolerant file storage: files are asynchronously replicated across cluster servers for data recovery and redundancy

SPD API

  • Description: Read access to pluggable sources of biodiversity data.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • access API tailored to biodiversity data sources
  • dynamically pluggable architecture of transformations (eg unified classification, synonyms presentation) from external sources of biodiversity data
  • plugins for key biodiversity data sources, including OBIS, GBIF and Catalogue of Life
  • dynamic clustering of discovered items
  • integration with workspace facilities
  • export of discovered items in multiple formats, including DarwinCore and DarwinCore Archive

SPD Plugin API

  • Description: Development framework for SPD plugins.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • support for spd plugins implementation
  • partial interface implementations

SPD Client API

  • Description: Client proxies for SPD endpoints.
  • Type: Java
  • Protocol: Local
  • Framework: CL
  • Documentation
  • Key Features
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
Retrieved from "https://wiki.gcube-system.org/index.php?title=Data_Access_and_Storage_APIs&oldid=26698"