Data Access and Storage APIs
From Gcube Wiki
Data Access and Storage APIs support both client and back-end access to a range of structured and unstructured data sources available in the Data e-Infrastructure. APIs include remote Web Service APIs, local client APIs, data model implementations, auxiliary APIs, and development frameworks for service plugins.
This document lists all the available APIs in these areas along with their key features, pointing to external documentation for details on their design and usage.
Contents
API Overview
| Name | Description | Type | Protocol | Framework |
| Tree Manager | Tree-based CRUD operations over pluggable sources of structured data | WS | SOAP | gCore |
| Tree Manager Client | Client proxies for Tree Manager endpoints | Java | Local | CL |
| Tree Manager Plugin | Development framework for Tree Manager plugins | Java | Local | none |
| Trees | Tree and tree pattern model implementations | Java | Local | none |
| Streams | Conversion, transformation, and publication of data streams | Java | Local | none |
| gDoc | gDoc model implementation |
Java | Local | none |
| gDoc Client | gDoc-based client proxies for Tree Manager endpoints |
Java | Local | CL |
| View Manager | Views over data sources accessible via Tree Manager endpoints | WS | SOAP | gCore |
| View Manager Client | Client proxies for View Manager endpoints | Java | Local | CL |
| Storage Manager | Access and storage of unstructured bytestreams, or files implementation | Java | Local | none |
| Species Products Discovery (SPD) | Read access to pluggable sources of biodiversity data | WS | SOAP | gCore |
| SPD Plugin | Development framework for SPD plugins | Java | Local | CL |
| SPD Client | Client proxies for SPD endpoints | Java | Local | CL |
The picture below offers an immediate view of the area and where each component is placed and which other interfaces:
Tree Manager API
- Description: Tree-based CRUD operations over pluggable sources of structured data.
- Type: Web Service
- Protocol: SOAP
- Framework: gCore
- Documentation
- Key Features
- lookup of whole trees as well as individual nodes
- tree pattern matching on lookups and queries
- in-place updates
- streaming on all operations
- native storage based on graph database technology
Tree Manager Client API
- Description: Client proxies for Tree Manager endpoints.
- Type: Java
- Protocol: Local
- Framework: CL
- Documentation
- Key Features
- high-level proxy API
- transparent endpoint discovery and caching
- transparent fault tolerance over endpoint replicas
- polling and notifications for asynchronous bindings
- fluent proxy configuration API
Tree Manager Plugin API
- Description: Development framework for Tree Manager plugins.
- Type: Java
- Protocol: Local
- Documentation
- Key Features
- support for a wide range of plugins: source-specific, source-generic, type-specific, type-generic, multi-type, etc
- interface-based POJO framework, with no dependency on service implementation
- transparent property publication and state persistence"
- callback-based and event-based interaction with service"
- full lifecycle support
- partial interface implementations
- derived implementation of node-based and streamed operations
Trees API
- Description: Tree and tree pattern model implementations.
- Type: Java
- Protocol: Local
- Documentation
- Key Features
- untyped tree model with attributed nodes, labelled edges, and string-valued leaves
- pattern language for characterisation of tree topology and leaf values
- tree matching and pruning operations based on pattern language
- tree difference and in-place update operations
- fluent API for tree construction, tree navigation, and tree pattern construction
- predefined XML bindings for trees and patterns
- synthetic tree generation facilities for load testing
Streams API
- Description: Conversion, transformation, and publication of data streams.
- Type: Java
- Protocol: Local
- Documentation
- Key Features
- high-level stream model based on
Iterator-style interfaces - predefined implementations over standard
Iterators andgRS2resultsets - piping, folding, and unfolding stream operations
- configurable error handling policies
- stream lifecycle event notifications
- publication as
gRS2resultset - fluent API for all stream operations
- high-level stream model based on
gDoc API
- Description:
gDocmodel implementation. - Type: Java
- Protocol: Local
- Documentation
- Key Features
- document model inclusive of metadata, annotations, parts, and alternatives
- strongly-typed document construction
- transparent bytestream resolution
- annotation threading
- new and proxy document modes
- change tracking and staging for in-place updates without prior reads
- XML binding facilities
gDoc Client API
- Description:
gDoc-based client proxies for Tree Manager endpoints. - Type: Java
- Protocol: Local
- Documentation
- Key Features
- creation of and access to
gDocdocument sources - bulk and streamed operations
- caching, selection, and pruning for read operations
- creation, discovery, and use of document views
- creation of and access to
View Manager API
- Description: Views over data sources accessible via Tree Manager endpoints.
- Type: Web Service
- Protocol: SOAP
- Framework: gCore
- Documentation
- Key Features
- view definition through arbitrary tree patterns
- view characterisation through arbitrary properties
- view publication in infrastructure
- computation of dynamic view properties (cardinality, last update)
View Manager Client API
- Description: Client proxies for View Manager endpoints.
- Type: Java
- Protocol: Local
- Framework: CL
- Documentation
- Key Features
- high-level proxy API
- transparent endpoint discovery and caching
- transparent fault tolerance over endpoint replicas
- fluent proxy configuration API
Storage Manager API
- Description: A client library for remote storage of files. The library allows clients to download, upload, remove, add, and list files. The file storage is based on MongoDB technology
- Type: Java
- Protocol: Custom
- Documentation
- Key Features
- structured file storage: Clients can create folder hierarchies
- secure file storage: File access is authenticated, every files can have private, group, or public access rights
- scalable file storage: files are stored in chunks and chunks are distributed across clusters of servers based on the workload of individual servers;
- fault-tolerant file storage: files are asynchronously replicated across cluster servers for data recovery and redundancy
SPD API
- Description: Read access to pluggable sources of biodiversity data.
- Type: Web Service
- Protocol: SOAP
- Framework: gCore
- Documentation
- Key Features
- access API tailored to biodiversity data sources
- dynamically pluggable architecture of transformations (eg unified classification, synonyms presentation) from external sources of biodiversity data
- plugins for key biodiversity data sources, including OBIS, GBIF and Catalogue of Life
- dynamic clustering of discovered items
- integration with workspace facilities
- export of discovered items in multiple formats, including DarwinCore and DarwinCore Archive
SPD Plugin API
- Description: Development framework for SPD plugins.
- Type: Java
- Protocol: Local
- Documentation
- Key Features
- support for spd plugins implementation
- partial interface implementations
SPD Client API
- Description: Client proxies for SPD endpoints.
- Type: Java
- Protocol: Local
- Framework: CL
- Documentation
- Key Features
- transparent endpoint discovery and caching
- transparent fault tolerance over endpoint replicas
