Annotation Management

The Annotation Back-End (ABE) service manages the entire life-cycle of annotation relationships between information objects, from their creation and collation to their retrieval, update, and deletion, independently from application-specific models of annotation content.

Clients may interact with the ABE service through a standard library of stubs generated automatically from the public service interfaces. They may also do so at a higher level of abstraction through the Annotation Back-end Library, an extension of the stub library which simplifies clients interaction with the service.

Annotation Relationships

Annotation relationships are specialisations of metadata relationships that give target objects the broad semantic of subjective and contextual assertions about source objects. More formally, annotation relationships are binary relationships with primary role is-described-by and secondary role is-annotated-by (IAB).

As specializations of metadata relationships, annotation relationships inherit all their properties:

• they are exclusive on their targets but repeatable on their sources: an annotation describes one and only one object even though the number of annotations for any given object may be unbounded;
• they preserve membership: an annotation belongs to a collection if and only if the annotated object does.

Annotation relationships induce a related specialization of the notion of collection: a collection is an annotation collection if it is a metadata collection of type IAB. More formally, A is an annotation collection for C if:

• A is an IAB-collection in the scope of C;
• all the members of A are annotations of members of C, and
• A is an annotation of C.

The definition of annotation relationships and annotation collections – as well as the relationships between these definitions and more generic notions in the gCube Information Model can be graphically illustrated as follows:

Annotation Model

The ABE service and library adopt a model of annotation content that is suitable for the exchange of potentially very heterogeneous annotations. In particular, the ABE exchange model complements arbitrary, application-specific notions of annotations with a small number of system-level and application-independent properties.

The model is defined in terms of XML serialisations of annotations and is in itself an extension of the exchange model adopted by Metadata Management services. In particular, it follows the same design pattern of grouping system-level properties into a header element and application-defined properties into a body element. The model is formally defined by the following schema:

<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema 
	xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
	xmlns:tns="http://gcube-system.org/namespaces/annotationmanagement/abe/model" 
	targetNamespace="http://gcube-system.org/namespaces/annotationmanagement/abe/model" 
	elementFormDefault="qualified" attributeFormDefault="unqualified">
	
	<xsd:import namespace="http://gcube-system.org/namespaces/metadatamanagement/mm/model"/>
	
	<xsd:element name="annotation" type="tns:annotationType"/>

	<xsd:complexType name="annotationType">
		<xsd:sequence>
			<xsd:element name="header" type="tns:headerType" />
			<xsd:element name="body" type="tns:bodyType" />
		</xsd:sequence>
	</xsd:complexType>
	
	<xsd:complexType name="headerType">
		<xsd:sequence>
			<xsd:element name="previous" type="xsd:string" minOccurs="0"/>
		</xsd:sequence>
	</xsd:complexType>
	
	<xsd:complexType name="bodyType">
		<xsd:sequence>
			<xsd:any namespace="##other" processContents="lax"/>
		</xsd:sequence>
	</xsd:complexType>
	
</xsd:schema>

Architecture

Within the gCube architecture, the ABE service is placed at the top of a stack of Information Organisation services, from Storage Management services to Content Management services and Metadata Management services. Its main role within the stack is one of mediation between the users of Information Presentation services and Metadata Management services one step below. In this role, the value it offers to its clients is one of specialisation and transparent aggregation of functionality available more generically for the management of metadata objects.

Design

The design of the service is distributed across three port-types: the Manager, the Broker, and the </code>Factory</code>.

Managers

The Manager port-type manages annotations of objects which belong to a target collection. It is the preferred point of contact for clients that:

• wish to work with such annotations regardless of the annotation collections to which individual annotations belong;
• wish to work with annotations which belong to specific annotation collections.

Clients may create annotation collections for the target collection, and then add, update, retrieve or delete annotation from them. These operations map directly onto equivalent operations of the Metadata Management service and are offered to present a single point of contact for annotations. Clients may also retrieve annotations across multiple annotation collections, though of course no write operation is offered at this level of abstraction.

All operations process and produce bulk data to avoid the latency of finer-grained interactions. Similarly, all operations are conceptually overloaded to work either by-value – inputs and outputs are entirely included in message payloads – or else by-reference – input and outputs are throttled in ‘pages’ using the resultset abstraction. Finally, all operations are tolerant to fine-grained failures, through they record them and then report them as return values.

The Manager is stateful, in that it maintains (in memory and on the local file system) summary information about the annotation collections of target collections. The information is grouped under local ‘proxies’ of the target collections, and the proxies are bound to the port-type interface on a per-request basis, in line with the implied resource pattern of WSRF. In particular, the pairing of the Manager interface and collection proxies identifies WS-Resources informally referred to as Managers.

Managers are created in response to client requests to the Factory [[#The Factory|port-type]. At the time and within the scope of their creation, a selection of the state of Managers is published in the Information System. In WSRF terminology, these are the Resource Properties that identify the target collection and their annotation collections and by which Managers can be discovered by clients.

Following publication, Managers are kept up-to-date with respect to the creation and deletion of annotation collections which do not occur through interaction with the Managers themselves. This form of maintenance is achieved by polling the Information System at regular intervals. During their lifetime, Managers satisfy client requests by interacting with WS-Resources and Running Instances of the Metadata Manager service. The interactions are based on best-effort and caching strategies.

Brokers

The Broker port-type manages annotations of objects which belong to either one of a number of target collections. Clients may retrieve annotations for these objects regardless of the collections to which the objects or their annotations belong. For example, the Broker is the recommended point of contact for clients that wish to browse the annotations of the results of a distributed search. Clients may also narrow their interaction to single target collections by obtaining a reference to dedicated Managers.

Like for Managers, the operations of the ,Broker operate in bulk, either by-value or by-reference, and report fine-grained failures within return values. Similarly, the Broker is stateful and its state is an aggregation of the state of the Manager. In particular the local ‘collection proxies’ of Managers are grouped in collection sets, and then collections sets are bound to the Broker into WS-Resources informally called Brokers.

Like Managers, Brokers are created by the Factory [[#The Factory|port-type] persisted and publish the identifiers of the target collections in the bound set as a Resource Property. During their lifetime, Brokers also interact with WS-Resources and Running Instances of the Metadata Manager service. The interactions, however, occur indirectly, in that they are delegated to Managers dedicated to the collections in the collection set.

The Factory

The Factory is the point of contact to the ABE for clients that wish to create Managers and Brokers for one or more target collections, starting from their public identifiers. In this role, it is stateless.

The ABE Library

The library offers a number of facilities for interacting with the port-types and WS-Resources of the ABE service.

First of all, it encapsulates stub-based interaction behind a local object-oriented interface. The interface is distributed across a set of classes which model remote port-types (AnnotationManager, AnnotationBroker) and related state abstractions (Collection, AnnotationCollection), or else manage configuration aspects of the interaction with the ABE, such as scope and security(AnnotationContext).

Throughout, the library makes use of language features which are not found in the stubs automatically generated from the remote interfaces, including high-level models of inputs and outputs, method overloading, and parametric type-checking. Behind these abstractions, the library engages in optimised and best-effort interactions with factory and WS-Resources of the ABE service. In particular, it hides from clients the discovery or creation of WS-Resources, but not the existence of remote interactions and the possibility of their failure.

In addition, the library offers convenient object bindings for the XML representations of annotations that are required by the remote interface in input or produced by it in output. Clients that wish to do so can model annotations as objects with the advantages of expressiveness and early type checking which normally characterise data binding solutions. Bindings occurs within an extensible framework defined by a root class (BaseAnnotation) that can be extended by classes which model particular types of annotations. In particular, the root class defines the protocol that subclasses implement to serialise their ‘body’ to and from the exchange model. The framework is built atop analogous frameworks offered by Metadata and ResultSet libraries, so that annotations are transparently suitable for use as generic metadata objects and as records of a resultset.

When it comes to actually manipulate annotations, the library provides annotation buffering so as to mediate between the fine-grained nature of operations in interactive sessions and the coarse-grained nature of interaction with the remote port-types. This may enable responsive user interfaces by concentrating bulk network interactions at distinguished 'commit' points which are entirely under client control. Specifically, clients that retrieve annotations from managers or brokers transparently fill buffers specific to individual annotation collections. They can then modify the buffered annotations locally for maximum responsiveness and commit the changes remotely before filling the buffer with the next retrieval operation.

Finally, the library offers built-in support for linking annotations of the same object into 'threads' similar to those typically formed by postings in mailing lists and discussion forums, or those associated with chains of versions.

Sample Usage

Interaction with the ABE service is illustrated with the examples below. The examples are based on the ABE library, partly because it is the recommended way to interact with the service and partly because the use of plain stubs can be inferred from the public interfaces of the service

The examples are modelled as static methods of a hypothetical test class, so as to explicitate their inputs. We assume that the following variables are defined in the closure of the test methods:

//some logger
GCUBELog logger = ...

//some scope
GCUBEScope scope = ....;

Preliminaries

First, we configure the library's behaviour through its AnnotationContext. In this sense, the AnnotationContext acts as a GCUBEScopeManager for the remote interactions with the ABE service which the library undertakes on behalf of its clients. As the examples are assumed to execute within the same thread, the simplest of scope configurations will suffice:

//configures the library to call the ABE service with a specific scope in this thread
AnnotationContext.setScope(scope) 

Note, however, that the AnnotationContext:

• exposes the full GCUBEScopeManager for multi-threaded executions;
• it can be configured to use any ScopeManager implementation which pre-exists in the client environment (e.g. a GCUBEServiceContext in clients that are services in their own right:

//configures the library to use an existing scope manager 
//AnnotationContext.setScopeManager(someExistingManager);

Managers and Collections

The first test method, uses the AnnotationContext to create and return an AnnotationManager for some annotated collection, i.e. a collection whose elements may be associated with annotations in one or more annnotation collections:

/**
* Creates an annotation manager for an annotated collection.
* @param collectionID the collection identifier.
*/

public static AnnotationManager getAnnotationManager(String collectionID) throws Exception {

  //Creates an annotation manager
  // note: this triggers a remote interaction and may return a GCUBEException 
  // we don't specifically deal with here
  return AnnotationContext.createManager(collectionID);

}

The following methods are utilities used in the test methods below to display summaries of the properties of the annotation collections associated with the annotated collection:

/**
 * Displays the properties of the annotation collections an annotation manager.
 * @param manager the manager.
*/
public static void listCollections(AnnotationManager manager) {
   //displays the properties of its annotation collections
   if (manager.getAnnotationCollections().size()==0) logger.info("No annotation collections for target collection"+manager.getCollectionID());
   for (AnnotationCollection<?> coll : manager.getAnnotationCollections().values()) showCollection(coll);
}

/**
* Displays the properties of an annotation collection.
* @param collection the collection.
*/
public static void showCollection(AnnotationCollection<GenericAnnotation> collection) {
  logger.info("***********************");
  logger.info("ID:"+collection.getID());
  for (Map.Entry<CollectionDescription.CreationParameters,String> prop : collection.getCreationParameters().entrySet())
	logger.info(prop.getKey().name()+":"+prop.getValue());
	logger.info("***********************");
}

Note that the creation properties form a subset of those defined by Metadata Management Library, which the ABE library extends (indeed, it is its only dependency). Note also that AnnotationCollections are parametric in the type of annotations they handled. Here the predefined type GenericAnnotation is used to indicate that no specific knowledge of annotations is required for the task.

Reading Annotations

The following test methods show how to the AnnotationCollections handled by the AnnotationManager may be used to localise the remote annotations which belong to those collections:

/**
* Localises all the annotations in a given annotation collection for one of more objects in the annotated collection.
* @param collection the annotation collection.
* @param oids the object identifiers.
*/
public static void getAnnotationsByCollection(AnnotationCollection collection, List<String> oids) {
  Map<String,List<String>> unparsedAnnotations = collection.localiseAnnotations(oids); //remote operation
  listUnparsedAnnotations(unparsedAnnotations);
}

/**
* Displays one or more unparsed annotations. 
 * @param annotations the annotations, grouped by their identifiers
 */
public static void listUnparsedAnnotations(Map<String,List<String>> annotations)  {
   logger.info("Unparsed Annotations:");
   for (String oid : annotations.keySet()) {
	 logger.info("Object "+oid);
	   for (String annoString : annotations.get(oid)) 
                   logger.info("Annotations\n"+annoString);
    }
}

Note that AnnotationCollections return the serialisations of all the annotations which could not be parsed into objects. Note also that localiseAnnotations() could also be invoked on AnnotationManagers to retrieve all the annotations of the input objects across all the annotation collections for the annotated collection:

<pre>
/**
* Localises all the annotations for one of more objects in the annotated collection, regardless of the annotation collection to which the annotations belong.
* @param collection the manager of the annotated collection.
* @param oids the object identifiers.
*/
public static void getAnnotations(AnnotationManager manager, List<String> oids) {
 listUnparsedAnnotations(manager.localiseAnnotations(oids)); //remote operation
}

Finally note that, in both cases, multiple invocations of localiseAnnotations() are allowed. However, annotations for a given object will only be localised once and any attempt to localise them again will generate a warning in the log (this is because annotations may have been updated between the two invocations.)

Annotation Types

The GenericAnnotations used so far reflect properties shared by all types of annotations and, more generically, all types of metadata (e.g. their identifier and the date of their last modification, as above). This generality is necessary to work uniformly across collections of different annotation types, but it limits what clients can do with annotations once they have been localised. Upon parsing into objects, any type-specific information is discarded and thus cannot be serialised again. In particular, GenericAnnotations cannot be created and updated, nor can their collections be created, as clients most often will wish to do.

However, the library can be instructed to parse and expose the properties of specific types of annotations, and even to create collections of such types. This requires clients to:

(*) define an object model of the required annotation type; (*) bind the model to the schema of the annotation type so that the library can identify it and use it when parsing annotations of a collection that is associated with that schema (as one of its creation properties).

To create an object model, clients subclass the abstract Annotation class, as the following example shows:

 ** Sample annotation type.*/
 public class TestAnnotation extends Annotation<TestAnnotation> {


	//Using XML Pull technology here for parsing and serialising

	/** XML Parser. */
	private KXmlParser parser = new KXmlParser();
	/** XML Serializer. */
	private KXmlSerializer serializer = new KXmlSerializer();
	
	/**Sample*/
	String foo;
        /** Elements name of sample annotation-specific property */
	private static final String FOO_ELEMENT_NAME = "foo";
	
	/** Changes the property of the annotation
	 * @return the property.
	 */
	public String getFoo() {return this.foo;}

	/**
	 * Returns the property of the annotation
	 * @param foo the property
	 */
	public void setFoo(String foo) {this.foo = foo;}

	/** Annotation serialisation namespace. */
	static final String NS="http://example.org";
	
	/**{@inheritDoc}*/
	public void fromXML(String xml) throws Exception { //parses annotation-specific properties

		super.fromXML(xml); //trigger parsing of generic annotation properties
		try {
			parser.setInput(new StringReader(xml));
			parser.setFeature(KXmlParser.FEATURE_PROCESS_NAMESPACES, true);
			loop: while (true) {
				int tokenType = parser.next();
				switch (tokenType){			
					case KXmlParser.START_TAG : 
						// remember position and name of tag
						String tag = parser.getName();
						if (tag.equals(FOO_ELEMENT_NAME) && parser.getNamespace().equals(NS))
							this.setFoo(parser.nextText());
						break;
					case KXmlParser.END_DOCUMENT :
						break loop;	
				}
			}
		} catch (Exception e){
			logger.error("Could not parse annotation:"+e.toString()+":"+e.getStackTrace()[0].toString());
			throw new Exception("Could not parse annotation:", e);
		}
	}

	/**{@inheritDoc}*/
	public String toXML(String body) throws Exception {//serialises annotatio-specific properties
		StringWriter output = new StringWriter();
		serializer.setOutput(output);
		parser.setFeature(KXmlParser.FEATURE_PROCESS_NAMESPACES, true);
		serializer.setPrefix("test", NS);
		serializer.startTag(NS,FOO_ELEMENT_NAME).text(this.getFoo()==null?"":this.getFoo()).endTag(NS,FOO_ELEMENT_NAME);
		return super.toXML(output.toString()); //passes serialisation up for embedding in annotation envelope
                //ignore body parameters, this is a 'leaf annotation' does not define an envelope to fill by subclasses
                //if it did, it would have had to embed the body of subclasses in its own envelope in turn. 
	}}

//note:this awkward parameterisation allows type-checking of inherited methods which return or take values of the //same type (emulates the type of the 'this' object, which is not natively supported by Java)

THIS SECTION OF GCUBE DOCUMENTATION IS CURRENTLY UNDER UPDATE.