Archive Import Service

The Archive Import Service (AIS) is in charge of defining collections and importing their content into the gCube infrastructure, by interacting with the collection management service, the content management service and other services at the content management layer, such as the Metadata Manager Service. While the functionality it offers is, from a logical point of view, well defined and rather confined, the AIS must be able to deal with a large number of different ways to define collections and offer extensibility features so to accommodate types of collections and ways to describe them not known/required at the time of its initial design. These needs impact on the architecture of the service and on its interface. From an architectural point of view, the AIS offers the possibility to add new functionality using pluggable software modules. At the interface level, high flexibility is ensured by relying on a scripting language, AISL, used to define import tasks, rather than on a interface with fixed parameters. The language is designed to support the most common tasks needed during the definition of an import task, is based on a flexible data model used to describe the resources to import, and is itslef extensible with new features. Beside the interface based on AISL, the AIS also offers other interfaces, called adapters, that ease the specification of common tasks and are also pluggable (i.e. new adapters can be added to the AIS). As importing collections might be an expensive task, resource-wise, the AIS offers features that can be used used to optimize import tasks. In particular, it supports incremental import of collections. The description that follows introduces first the rationale behind the functionality of the AIS and its overall architecture, then describes its scripting language AIS, its extensibility features, and the concepts related to incremental import. Finally, it presents the interface of the service.

Importing Resources

Overall, the execution of an import task can be tought as divided in two main phases 1) Representation phase and 2) Import phase. During the first phase, the resources to be imported and their relationships are modeled using a graph-based model. This representation is used during the import phase by appropriate "importers", software modules that encapsulate the logics needed to import specific kind of resources, interfacing with the appropriate services.

Representation phase During this phase, the resources to be imported are modeled using a graph based data model. The model contains three main constructs: collection, object and relationship, thus resembling the collection and information object models on which the content management services are based. The constructs in the modeling language can be used to assemble a graph of resources, specifying their content and relationships. Each construct has a type, and can be annotated with a number of properties. The type of the resource is a marker that is used by importers to select the resources they are dedicated. The properties are just name-value pairs. The values can be of any java type. Notice that only some types of resources and their properties are fixed in advance. In particular, the existing types are used to allow the import of content and metadata. To support the extension of functionality of the AIS, it is possible to define new types, annotated with different properties.

Import phase During this phase, an import-engine dynamically loads and invokes a number of importers, modules that are responsible to import specific kinds of resources. Each importer inspects the representation graph, identifies the resources for which it is responsible, and imports them. Beside performing the import, the importer may also perform some further annotation of the resources in the graph. These annotations are used in later execution of tasks that involve the same resources, and may also be exploited by other importers. For example, in the case of importing metadata related to some content, the content objects to which the metadata objects refer should have already been stored in the content-management service, and should have been annotated with the identifiers used by that service. Similar considerations hold for content collections. Importers enabled to handle content-objects and metadata objects are already provided by the AIS. Additional importers, dedicated to store specific kind of objects, can be added.

At the end of the import phase, the resource graph created during the representation phase and annotated during the import phase is stored persistently.

Interface to the AIS

The client interface to the AIS is based on a scripting language, called AISL (Archive Import Service Language). The language can be used directly to submit tasks to the service, but also to create parametric task-managers that are in charge of managing well defined tasks that only require a fixed number of parameters. The language is designed to create a representation graph, according to the model described above, in a user-friendly way.

AISL AISL is an interpreted scripting language with XML-based syntax. As most programming languages, AISL supports various flow-control structures, allows to manipulate variables and evaluate various kinds of expressions. However, the goal of an AISL script is not that of performing arbitrary computations, but to create a graph of resources. Representation objects (collections, objects and relationships) are first class entities in the language, that provides constructs to build them and assign them properties. Representation objects may resemble objects in oo-languages, in that their properties can be accessed as fields and assigned values, and a reference to representation-objects themselved can be stored in variables. A fundamental difference is that, once created, representation objects are never destroyed, even when the control-flow exits the scope in which they were created.

Rather than describing in detail the syntax of the language, for which we refer to the documentation of the AIS, we provide here an example of script to define and import a collection.

<?xml version="1.0" ?> <program> <createCollection type="content" name="contentCollection"> <property name="collectionName" expr="'TestContentCollection'"/> <property name="isVirtual" expr="false"/> <property name="isUser" expr="true"/> [...] </createCollection>

<createCollection type="metadata" name="metadataCollection"> <property name="collectionName" expr="'TestMetadataCollection'"/> <property name="collectionDescription" expr="'A test collection'"/> [...] <property name="relatedContentCollection" expr="$contentCollection"/> </createCollection>

   <define name=”file” expr=”first(file(http:some.web.site/docs/metadata.xml))”/>

<foreach name="a" in="xpath(dom($file),'//record')"> <createObject type="content" name="contentObject" collections="contentCollection"> <property name="documentName" expr="'testDocument'"/> <property name="content"

            expr="file(string(first(xpath:($a,'//uri/text()'))))"/> 

<property name="hasMaterializedContent" expr="false"/> <property name="isVirtualImport" expr="false"/> </createObject>

<createObject type="metadata" name="metadataObject" collections="metadataCollection"> <property name="content" expr="string($a)"/> [...] </createObject> <createRelationship type="metadata" from="metadataObject" to="contentObject" name="rel"/> </foreach> </program>

Here, the instructions createCollection, createObject and createRelationship define respectively collections and objects and assign them properties whose value are either constants and obtained by evaluating expressions. When a representation construct is created this way, it is possible to specify a name for a variable that is initialized to its value and can be used (while in scope) to refer to the construct, for instance to pass it to other constructs (as in the case of creating a relationship, that requires a reference to two objects) or to alter its properties. The define instruction defines a variable an initializes it to the result of the evaluation of the file function, which returns a remote file (see below for more details on this). The foreach instruction executes a bounded cycle over a list of values resulting from the evaluation of an expression, which in this case is a list of DOM nodes obtained by evaluating an XPath expression over a document. Another Xpath expression is evaluated inside the cycle, this time over a DOM fragment whose value is stored in a variable.

The script downloads an XML file at a given URI, parses it, and for each <record> element present in it: a)Creates a content-object whose content is the file at the address stored as a sub-element <uri> of the <record element> b)Creates a metadata object whose content is the <record> element. c)Associates them with a relationship d)Stores the objects in appropriate collections.

AISL is not typed, i.e. variables can be assigned values of any kind, and is the responsibility of the programmer to ensure proper type concordance. Basically, AISL does not even define its own type system, but exploits the type system of the underlying Java language. The language offers constructs for building objects of some basic types, like strings, sets, files, and integers. However, expressions in AISL can return any Java object. Specific functions can be thus used to build objects of given types in a controlled way. For instance, the built-in "dom()" function accepts as input a file object and produces a DOM document object. Similarly, the "xpath" function takes a DOM object and an expression, and returns the result of the evaluation of an xpath over the DOM node object as a set of DOM Node objects. In general, thus, types cannot be directly manipulated from the language, except for a few cases, but the variables of the language can be assigned with any java type, and objects of given type can be built using functions. This means that addon-functions can produce as result objects of user-defined types. These objects can be stored in representation objects as properties, and later used by specific addon-importers. However, AISL itslef provides a few special data-types, not present in the standard java libraries. In particular, the type AISLFile is designed to work in combination with some AISL built-in functions and to optimize the management of files during import, in particolar with regard to memory and disk storage resources consumption. An AISLFile object encapsulates information about a given file, such as file length, and allow access to its content. However, the file content is not necessarily stored locally. If the file is obtained through the file() function, then the download of its actual content is deferred, and only performed when needed. When describing a large number of resources, as in the case of large collections, it is not feasible (or anyway not efficient) to store locally to the archive import service all contents that need to be imported. This is especially true for content that has to be imported without having to be processed in any way before the import itself. Even for those files that might need some processing (for instance for extracting information), it might be desirable for the AISL script-writer to be able to import the file, use for the time he needs it (e.g. pass it to the xslt() function) and then free the memory resources used to maintain it, without having to deal directly with the details. The AISL file offers a transparent mechanism to handle access to the content of remote files accessible through a variety of protocols, by having a placeholder (an AISFile object) that can be treated as a file. Internally, an AISLFile implements a caching strategy for documents whose content has to be accessed at resource-description time. For files whose contents have to be handled only at import time, it offers a way to encapsulate easily inside a representation object all the information needed to pass to other service like the storage management or the content management service.

Functions To encapsulate complex operations, AISL provides functions, that work as in most programming languages. A number of built-in functions is already defined inside the language. These functions cover common needs arising during the definition of resource graphs, and are detailed below. The language can be expanded by adding new functions. Furthermore, AISL provides extensible-functions, meaning that their functionality can be extended to handle special kinds of arguments. For instance, the file() function allows to retrieve a list of files given a set of arguments that define their location. The built-in function is already able to deal with a number of protocols, including http, ftp and gridftp. However, the function can be extended to handle additional protocols, as explained in more thorough details later on. The motivation behind extensible functions is to keep the syntax of AISL as lean and transparent for the user as possible.

Built-in functions: first(), get(index), add(index, object), remove(index), size()

These functions all take as argument an object of type List and perfom some operation on its elements, like getting the first object in the list, adding or removing an object, returning the size of the set.

dom()

this function takes as input an AISFile object and returns a DOM document obtained by parsing the file (or null if the parsing fail, e.g. if the file is not a valid XML file). 

Built-in extensible functions file() Accepts as input a string used to describe the location of some file, and returns a set of AISFile objects. There are no constraints set on the format of the string, which might be a URL or some other kind of identifier. Internally, the file function is able to deal with a number of communication protocols, including http, ftp, and gridftp.

string() This function provides a way to perform a custom string serialization of objects. It accepts as input an object of any type and returns an object of type String. If a suitable extension is provided for it for a given Java type, then it will be used to process tat type of object. Otherwise, the fuction will just return the result of invoking the java toString() function on its argument. As built-in functionality, the string() function recognizes DOM objects and provides for them a serialization as XML document.

Extensibility features

The entire architecture of the AIS is design to offer an high degree of extensibility at various levels. All these mechanisms are based on a plug-in style approach: the service doesn't have to be recompiled, redeployed or even stopped whenever an extension is added to it.

At the level of the interface with the service, "adapters" can be incorporated in the service. In the simplest case, such adapters will just offer a simplified interface to certain functionality that has to be invoked often without many variations. For instance, in the case when the files to import are stored at certain directory accessible through ftp, or in the case when a single file available at some location describes the location of files and related metadata, it would be desirable to invoke the AIS with the bare number of parameters needed to perform the task, rather than having to write an AISL script from scratch.

The AISL language itself provides extension mechanisms. New functions can be defined for the language, and existing extensible functions can be extended to treat a larger number of argument types. The representation model used to describe resources is fully flexible: new types can be defined for collections, objects and relationships, and all these representation constructs can be attached with arbitrary properties of arbitrary types. Regarding the importing phase, new objects types can be handled by definining pluggable importers. These importers will be invoked together with all other importers available to the AIS service, and handle specific types of representation objects. It is to be observed that the definition of new representation constructs and that of new importers are not disjoint activities. In order for new object types to have meaning, there must be an appropriate importer which is able to handle them (otherwise they will just be ignored). In order for an importer to work with a specific kind of object type, the importer must be aware of its interface, i.e. its properties. While properties attached to representation objects can be always accessed by name, in order to make easier the development of importers it is possible to define new representation constructs so that their properties can be accessed via getters and setters. During the description phase, the AISL interpreter will recognize if a new type is connected to a subclass of the related representation construct and build an object of the most specific class, so that later on the importers can work directly on objects of the types they recognize.

Plugin mechanism In order to support these extensibility mechanism, the AIS exploits a simple plugin pattern. The modules (Java classes) that have to be made available must extend appropriate interfaces/classes defined inside the AIS framework, and be defined in specific Java packages. To be compiled properly, these classes must of course be linked against the rest of the code of the AIS. After compilation, the resulting .class files can be made available to an AIS instance by putting them into a special folder under the control of the AIS instance. The classes will be dynamically loaded and used (partially using the Java reflection facilities).

In general, The main interfaces/classes that can be extended for producing plugins are the following:

To extend the AISL language: General Extension: AbstractRule CompositeRule Function Processor

To extend the representation model: RepresentationObject RepresentationObjectCollection Relationship

To define a new Importer: Importer

For a full description of these interfaces/classes, especially regarding the internal contracts/policies that the subclasses classes must follow, we refer the reader to the documentation of the AIS.

Support for incremental import

The archive import service must fully support incremental import. With the term incremental import, we denote the fact that if the same collection is imported twice, the import task should only perform the actions needed to update the collection with the changes occurred within the two imports, and not re-import the entire collection. Incremental import requires two features. First, it must be possible to specify that a collection is the same an another collection already imported. For the existing importers, this is achieved by specifying inside the description of a collection the unique identifier of the collection in the related service. For instance, for content collections, this would be the collectionID attached to each collection by the Collection Management Service. After the description of the new collection has been created, the service will compare this description with that resulting from the previous import, and decide which objects must be imported, which must be substituted, and which must be deleted from the collection.

When comparing two collection, the import service must know how to decide wheter two objects present inside a collection are actually the same object. In order to support this behaviour, the AIS must support two concepts, that of external-object-identity and that of content-equality.

External object identity is an external object-identifier. Two objects are considered to be the same if and only if they have the same external identifier. Notice that the external identifier is distinct from the internal-object identity that is used by services to distinguish between objects. Of course, the AIS must ensure a correspondence between internal and external identifiers. Thus, if an object with a given external identifier has been stored with a given internal identifier, then another object is imported with the same external identifier then the AIS will not create a new object, but (eventually) update the contents/properties of the existing object.

External Identity is sufficient to decide whether an object to be imported already exists in a given collection. If an object already exists, and it has changed, then it will be updated. Deciding whether an object has changed also requires additional knowledge. For many properties attached to an object, the comparison is straightforward. However, for the actual content of an object this is less immediate. The content of an object (that is a file) might reside at the same location, but differ from that previously imported. Furthermore, comparing the old content with the new content is an expensive task: it requires at least to fetch the entire content of the new object and thus is as expensive as just reimporting the content itslef. For this reason the AIS also supports the concept of content-identity. A content-identifier can be specified. If two identical objects (i.e. having the same external-object-identity) have the same content-identifier, than the content is not reimported. If, on the other hand, the identifier differs, then the new content is re-imported. A content identifier can be a signature of the actual content, or any other string.

If a content identifier is not specified, the AIS the content-identifier is obtained exploiting the following information location of the content, size of the content (when available, depends on the particular protocol used), date of last modification of the content (when available, depends on the protocol used), an hash signature of the content (when available from the server, depends on the protocol used. For instance, it is possible to get an MD5 signature of a file from an HTTP server).

Service Operations

The operations of the archive Import Service Are described in the following:

Signature Description ImportTaskResponse submitImportTask(String AISLScript) This operation allows to submit an import task defined via the AISL scripting Lanaguage. ImportTaskResponse submitAdapterTask(String adapter, String[] adapterParameters) This operation allows to submit an import task to a specific adapter, which must have been previously registered as plugin. The parameters needed by the adapter to perform the task are specified via an array of string parameters. The number and semantics of these parameters are not defined in advance, and may differ for each single adapter. RegisterPluginResponse registerPlugin(String PluginURL) This operation allows to remotely load a new plugin component into the AIS. The plugin can be loaded by specifiying an URL at which the compiled java class or jar file can be downloaded.