Difference between revisions of "Archive Import Service"

Revision as of 21:05, 14 October 2008

Introduction

The role of the AISL Service is to import resources which are external to the gCube infrastructure into the infrastructure itself. "Importing", here and in the following, refers to the description of such resources inside the Information Organization stack of services, and not necessarily to the fact that the content of such resources is actually stored within facilities that belong to the infrastructure. Similarly, kind of resources that can be imported are not necessarily objects that exist physically. The association between an image and a file containing metadata referring to it might not exist physically, but can still be considered a resource that can be imported. The same holds for a collection of images. The word "external resource" is used to denote any resource outside the gCube infrastructure (independing of the fact that it has been imported already or not). The world "internal resource" is used to denote the entity that represents an external resource inside the gCube infrastructure. This is normally an object or a relationship of the info-object model, and is identified, inside the gCube infrastructure, by an Object Identifier.

The service should support the import by providing a way to specify which resources should be imported and how, and offer facilities to automate this task whenever possible.

Import procedure

The task of importing a set of external resources is articulated in two major steps. First, a description of the resources to import is built. This description is based on a custom data model, which is described later on in this document. The resulting description is essentially a graph labelled on nodes and arcs with key-value pairs. This description is called a graph of resources (GoR). Second, the GoR built during the first phase is processed by a chain of software modules called "importers". Each importer is dedicated to import resources interacting with a specific subpart of the Information Organization set of Services. For instance, the metadata importer is responsible for the import of metadata, and it handles their import by interacting with the Metadata Management Service. The precise way in which importers handle the import task, and in particular how they define a specific description of the resources they need to consume inside a GoR is left to the single importers. Details about the importers which are already included in the Archive Import Service are given below in this document. The creation of a graph of resources is done trhough the execution of a script written in the AISL language, which is described later on in this document.

Data Model

The data model handled by AISL features three main types of constructs:

• Collection
• Resource
• Relationship

A set of objects of these three main types, built by AISL script, form a so-called Graph of Resources (GoR). A graph of resources is a graph composed by nodes (resources) and edges (relationships). Furthermore, The collection construct allows to group nodes (resources) into sets. All constructs of the model can be annotated with properties, which are name/value pairs.

The three main types of constructs cannot be instantiated directly. Instead, objects of specific subtypes of these constructs must be instantiated. These subtypes are defined by specific plugins called importers that manage the import of different kinds of resources inside the gCube infrastructure. The precise semantics of the properties attached to types and the precise use of the constructs is not fixed in advance, but is determined by the specific importer that defines and manage a specific subtype. In particular, there is no direct correspondance between constructs in the GoR and how the resources are represented inside the gCube Information Organization facilities.

Beside being annoted with properties, each construct of the model must be assigned an external identifier. An external identifier is a string that uniquely identifies a certain external resource, and the model object that refers to it. This identification must hold across multiple, different invocations of the AIS.

For instance, consider the two files:

1. http://mydomain.org/myimage.jpg
2. http://myotherdomain.org/mymetadata.jpg

representing respectively an image and a file of metadata describing it. Here, we have three external resources to import: the files 1) and 2) and the association between the two. This set of resources can be represented by instantiating two model objects of type "resource", having specific type respectively equal to "content" and "metadata", and and a model object of type "relationship" and subtype "metadata". Furthermore, the GoR should contain two model objects of type collection and subtype respectively "content" and "metadata" which will contain the objects referring to resources 1) and 2).

When all these resources are created, they must be assigned an external identifier, which specifies their identity in the real worls. So, for instance, it would be possible to choose the string "http://mydomain.org/myimage.jpg" to identify the resource 1). The first time an import task is run that specifies this gor, the AIS will create an internal resource referring to the external resource 1). This internal resource will receive an (internal) object identifier. If another import task is executed, and another resource with external identifier "http://mydomain.org/myimage.jpg" is created, the AIS will treat this as the same resource, and so it will not create another internal resource but modify, if needed, the one already created.

The AISL Language

AISL is a scripting language intended to create graphs of resources for subsequent import. Its main features are a tight integration with the AIS, in the sense that the creation of model objects are first citizens in the language, and the ability to treat in a way as much as possibile transparent to the user some tasks which are frequent during import, like accessing files at remote locations. Beside avoiding the complexity of full fledged programming languages like Java, its limited expressivity - tailored to import tasks only - prevents security issues related to the execution of code in remote systems. The language does not allow the definition of new functions/types of objects inside a program, but can be easily extended with new functionality by defining new functions as plugin modules. This feature is detailed later on in this document.

Type System

The language is currently non typed. Variables can be assigned with any kind of object from the Java type system. However, it is planned to enforce at least a partial static type checking in the future, to allow early detection of errors in the scripts. For this reason, the grammar already requires variables to be declared before their use. Even though the language process does not currently check for type compliance, it is strongly suggested that implementors try to use the appropriate types, in order to reduce the effort of reconverting scripts later on. The types supported by the language(i.e. for which the language allows an explicit variable declaration) are:

• Primitive types:
  • integer
  • float
  • boolean
  • string
  • list
  • file

• Model object types:
  • collection
  • resource
  • relationship

Notice that AISL is not an object oriented language. Even if some of the types correspond to java types, there are no methods nor fields. Access to properties of objects is instead possible thorugh appropriate functions. So, for example, the size of a list value can be obtained by invoking the function listsize() on it. However, model object types have "properties" that resemble fields in object oreinted programming languages and that can be accessed through the familiar dot (".") notation.

Values of the primitive types are treated as in the Java language. As in Java, strings can be created by supplying a value between double quotes, and can be concatenated using the "+" operator. Values of the "list" type are similar to lists in the java language, but array-like selection of their elements is supported, and the language provides a built in constructor for lists (se below at the section type constructors). As in java, lists indexes are zero-based. Lists are currently heterogeneous (i.e. they can contain values of different types). In future releases, it is planned to provide facilities to enforce list-type checking. The "file" type represents local or remote external resources in a way that is independent of the specific protocol used to access these resources.

The types collection, resource and relationship correspond to the model object constructs introduced in the previous sections. Variables of these types are used to hold instances of objects of the resource graph data model.

Syntax

This section describes briefly the syntax of AISL and semantics of AISL constructs, focusing especially on aspects in which the language differs from the Java programming language syntax, to which it is close. A formal description of the syntax can be found in the appendix following this document.

An AISL script is a sequence of instructions, which are either variable declarations, conditional statements, loop statements and some kind of expressions like variable assignments and function invocations. Values for the various types in the AISL can be built through appropriate type constructors. Values can then be manipulated and composed using expressions of various kind, which are in most cases similar to the corresponsing expressions in the Java programming language.

Type constructors

Primitive type constructors

integer

integer values are sequences of digits, starting with a non-zero digit, or a single '0' digit. In other words, they match the production DECIMAL_LITERAL: "0"|(["1"-"9"] (["0"-"9"])*)

float

floating point values are expressed by a decimal decimal value eventually prefixed with an integer value: FLOATING_POINT_LITERAL: (["0"-"9"])+ "." (["0"-"9"])*

boolean

the words true and false are reserved words in AISL, and they are interpreded as the corresponding boolean values

string

string literals are sequences of characters between double quotes. Special characters like newline and tab are escaped and treted as in the Java programming language.

list

lists are built by enclosing a list of expressions separated by commas into curly brackets. For example:

list myList = {3+4, 56, "a"};

file

file objects are built by invoking the constructor functions getfile(string locator) and getFile(string locator, list<string> accessinformation) A locator is a string encoding a location and a protocol to be used to access the file. For instance the instruction:

file f= getFile("ftp://ftp.example.org/pub/share/myfile.xml);

builds a file object that accesses its content through the ftp protocol at the given location. The format of the locator string is not defined in advance, as it depnds on the specific protocol used. Currently supported protocols are ftp, http, file. They all accept an URL as locator. Different formats may be provided by different subtypes of the AISLFile class. This is described in more detail later on in this document. The two-arguments constructor allows to pass in login information that might be needed to access remote resources.

Model Object Constructors

These constructors allow to create elements of the resource graph which is later used for import by the service. Once created, the properties of an object can be modified but the object itself cannot be deleted. In other words, it sufficient to invoke one of these constructors for the object to be in the final graph of resources. In general, all constructors impose to provide:

• the type of construct to be created (i.e. collection, resource or relationship)
• the specific subtype of the object. This subtype should be defined in the context of a specific importer, by subclassing

appropriately the class definining the basic construct. This allows to perform checks during the parsing of the script, e.g. on the properties of constructs. For example, the type collection::content is a subtype of the type collection that defines the properties collectionName, isVirtual and isUser.

• a unique "external identifier". This string value uniquely identifies a certain construct, so that it can be recognized during subsequent import phases.
• in the case of resources, it is possible to supply to the constructor a list of collections to which the resource must belong
• in the case of relationships, the resources that the relationship links must be specified.

Furthermore, the body of the constructor allow to initialize one or more of the properties eventually defined by the construct. The names, types and precise semantics of these properties are described in the section about importers.

Examples of constructors are as follow:

collection metadatacollection = collection::metadata["medspiration_test_metadata"]{ collectionName = "medspiration_test_metadata", collectionDescription = "test for the AIS with medspiration data", relatedContentCollection=contentcollection, isUser=true, isIndexable=false, metadataName="dc", metadataLanguage="en", metadataSchemaURI="http://www.opendlib.com/resources/schemas/metadata_dc.xsd" };

This constructor defines a metadata collection. Here the type is "collection", the subtype is "metadata", the external identifier is given by a static string ("medspiration_test_metadata"). The body of the constructor initializes a number of properties specific of the "metadata" subtype. Notice that the object created by the constructor is then assigned to a variable of the appropriate type (collection).

resource::content[url] in ccoll{ isVirtualImport=false, contentSourceLocator=url, documentName= name, hasMaterializedContent = false };

This constructor defines a resource of type content. The external identifier here is a variable (url) that must evaluate to a string. Furthermore, the object is specified to belong to a specific collection again using a variable (ccoll) holding an instance of a (content) collection.

relationship rel= relationship::metadata(metadata, content)["metadatarel"+url]{};

this constructor specifies an relationship of subtype metadata. The couple of variables (metadata, content) specify the resources to and from which the relationship holds. The external identifier is computed as an expression evaluating to a string.

Expressions

Arithmetic Expressions numeric types (integer and float) can be combined using the same operators available in the Java programming language, i.e. the unary operators + and - and the binary operators +, -, /, * and %. These operators have the same precedence and semantics as in Java. If the operands of a binary operator have different type, the type of the result is always "float".

Relational Expressions

The relational operators ==, !=, <, <=, >, >= have the same precedence and semantics as in java. They all evaluate to a boolean value and they can all be applied to numeric values. Furthermore, the operators == and != can be applied to all other types.

Boolean Expressions

Boolean expressions are built from boolean values by applying the unary operator ! (not) and the binary operators | (or), & (and), ^ (exclusive or), whit the same precedence and semantics as in Java. Notice that differently from java AISL does not support the conditional boolean operators ||, && and ^^.

Selectors

The elements of list-typed values can be obtained with the same syntax that in Java is used to access the elements of arrays. E.g.

list myList = {3+4, 56, "a"}; integer myInt = myList[1];

Lists can be nested, and selectors can be combined: list myList = {3+4, {45, 10}, "a"}; integer myInt = myList[1][0];

the properties of model object typed values can be accessed by name with a dot notation. e.g.

resource myContentResource = resource::content[url] in ccoll{ isVirtualImport=false, contentSourceLocator=url, documentName=name, hasMaterializedContent = false };

myContentResource.documentName="test";

Variable Declarations

Variable Declarations contain a specification of an AISL type, a variable identifier and an optional initializer. E.g.

list myList = {3+4, 56, "a"};

Functions

Function invocation in AISL is analogue to function invocation in Java, except that all function have global visibility and there are no objects or classes thorugh which invoke a function. An example is:

... string mystring= "test"; boolean matches= match(mystring, "t.*t"); print(matches); ...

This code snippet contains two function invocations, namely of the functions match and print (it prints "true"). AISL comes with a set of predefined functions, described below. New functions can be added to the language. This is described later on in this document.

Predefined AISL Functions

Functions on file

These predefined functions provide access to the properties of objects of type file. string filename(file f) returns the name of the file integer filesize(file f) returns the size of the file boolean isdirectory(file f) returns true if this file is a directory boolean isfile(file f) returns true if this file is a regular file list<file> children(file f) returns a list containing a file object for each of the children of f. The list returned is empty if this file is a regular file (i.e. not a directory) or is the protocol used to access the file does not model a hierarchical fileystem. list<file> descendants(file f) returns a list containing a file object for each of the descendants of f, obtained by recursively exploring all subdirectores. Notice that the list contains all files in the subtree rooted at f, not only its leaves (i.e. it also contains all directories taht are descendants of f. The list returned is empty if this file is a regular file (i.e. not a directory) or is the protocol used to access the file does not model a hierarchical fileystem.

Functions on string

boolean match(string str, string patter) returns true if the given string matches the given regular expression pattern, false otherwise. boolean print(string str) prints str.

Functions on list

integer listsize(list l) returns the size of the list l

Functions on dom objects

list<dom> xpath(dom file, string xpathexpression) dom xslt(dom file, string xslt transformation) string toString(dom file) converts a given dom object into a string (i.e. to its XML serialization).

Control Flow Statements

AISL contains three control flow statements: if, switch and foreach. The major syntactic difference between these statements and the corresponding ones in the java language is that instructions inside the constructs must be enclosed in curly brackets (even when they contain a single instruction). Notice that these statements are not terminated by a ";" character. For the rest, if and switch statements are completely similar to their Java counterparts, while the foreach statement has a special syntax.

Conditional Statements

If Statement

This statement has the same syntax and semantics as in Java, and takes the two forms:

if( conditional expression ){ ... }

and

if( conditional expression ){ ... } else{ ... }

Switch Statement

This statement has the same syntax and semantics as in java:

switch( expression ){ case expression1: ... break;

...

case expressionN: ... break;

default: ... break; }

Loop Statements

The AISL language tries to avoid as much as possible unbounded loops. For this reason it does not have a while statement and has a foreach statement that only allows bounded loops. In particular, foreach allows to iterate over a range of integer values, with a fixed increment (or decrement).

foreach loopvariable in [ expression to expression by expression]{ ... }

The three expressions appearing in the statement correspond to the minimum and maximum value of the range and to the increment. If no increment is given, its value is assumed to be one. The variable loopvariable is defined inside the foreach loop block only, and its value can be read but not assigned. Example.

foreach i in [0 to listsize(mylist)-1]{

    print(mylist[i]);

}

this code snippet will print the value of all objects in the list "mylist".

Importers

The Archive Import Service perform the import of external resources by representing them in a Graph of Resources and passing this graph to a chain of software modules called "importers". Each importer is responsible for treating specific kind of resource (e.g. metadata), and essentially is the bridge between the archive import service and the services of the Information Organization stack responsible for managing certain kind of internal resources (collections, metadata, documents etc). The precise way in which the importer performs the import is thus dependent on the specific subsystem the importer will interact with. Similarly, different importers will need to obtain different information about the resources to import. For instance, to import a document it is necessary to have its content or an url at which the content can be accessed. To create a metadata collection, it is necessary to specify some properties like the location of the schema of the objects contained in the collection. These values are passed to an importer by annotating objects in a graph of resource with appropriate properties. In order to constrain the kind of properties that the model objects it manipulates must have, an importer must define a set of subtypes of the model object types. For instance, the metadata importer (described below) defines a subtype for each basic type of the Resource Model types:

• collection::metadata,
• resource::metadata and
• relationship::metadata

Each of these subtypes has specific properties that are understood, used and manipulated by that importer. The way subtyping is accomplished is described in more detail later in the section "writing new importers". The types defined by an importer and its properties must be publicly available, as AISL script developers must known which are the properties available for them and what is their semantics. Furthermore, subsequent importers in the chain may also need to access some properties. For example, an importer for metadata needs to access also model objects representing content to get their object id (internal identifier). Notice that in general an AISL script will not necessarily assign all properties defined by a subtype. Some of these properties may be conditionally needed, while come will only be written by an importer at import time. For example, when a new content object is imported, the importer must record into the GoR object the OID of the newly created object. For this reason, the specification of the subtypes defined by importers must also provide information about what properties are mandatory (i.e. they must be assigned during the creation of a GoR) and which properties are private (i.e. they should NOT be assigned during the creation of the GoR).

Built-in importers

The AIS comes already with the capability to import documents and metadata. This is provided by two importers called the content importer and the metadata importer. The types defined by these importers are described below:

Content Importer

This importer defines two subtypes. In particular, it defines a new collection type and a new resource type: collection::content resource::content

The properties of these subtypes, their type and semantics are as follows:

[collection::content] collectionName : string  : mandatory - The name of the collection. isUser : boolean : mandatory - Denotes if a collection is or not a user collection collectionId : string  : private - The id assigned to the collection to the collection management service

[resource::content] isVirtualImport  : boolean : mandatory documentName  : string  : mandatory hasMaterializedContent : boolean : mandatory contentSourceLocator  : string  : content  : file  : documentId  : string  : private - The id assigned to the collection by the storage management service

Note: the fields contentSourcelocator and content are alternative. They depend on the value of the field hasMaterializedContent

Metadata Importer

This importer defines three subtypes, one for each basic construct in the Resource Model. They are:

collection::metadata resource::metadata relationship::metadata

[collection::metadata] relatedContentCollection: collection : mandatory - The content collection containing the objects to which this metadata collection refers collectionName : string  : mandatory - The name of the collection collectionDescription : string  : mandatory - A description of the collection isUser : boolean  : mandatory - Indicates wheter this is a user collection isIndexable : boolean  : mandatory - Indicates wheter this collection is indexable metadataName : string  : mandatory - Name of the metadata schema in this collection metadataLanguage : string  : mandatory - Language of the metadata in this collection metadataSchemaURI : string  : mandatory - URI of the schema of the metadata in this collection collectionId : string  : private - The id assigned to the metadata collection during the import

[resource::metadata] content  : string  : mandatory - the content of this metadata object objectID  : string  : private - the id assigned to the metadata object during the import

[relationship::metadata] This subtype does not defined any property. It denotes an edge from a metadata resource object to a content resource object

Extension Mechanisms

The Archive Import Service can be extended in a number of ways, mostly via a plugin-based mechanisms. In particular, it is possible to extend the AISL language by adding new Functions and by adding the support for different protocols managed by the file built-in type, and it is possible to add to the AISL new importers that deal with new new kinds of data managed by gCube Information Orgainzation Services.

Extending the AISL

Creating new Functions

The language can be extended by adding functions. Adding a new function amounts to two steps:

1. Creating a new java class implementing the AISLFunction class. This is more easily done by subclassing the AbstractAISLFunction class. See below for further details.
2. Registering the function in the "Functions" class. This step will be removed in later released, which will implement automatic plugin-like registration

The AISLFunction interface provides a way to specify a number of signatures (number and type of arguments and return type) for an AISL function. It is design to allow for overloaded functions. The number and types of the parameters are used to perform a number of static checks on the invocation of the function. The method evaluate provides the main code to evaluated during an invocation of the function in an AISL script. In the case of overloaded functions, theis method should redirect to appropriate methods based on the number and types of the arguments.

public interface AISLFunction { public String getName();

public void setFunctionDefinitions(FunctionDefinition ... defs); public FunctionDefinition[] getFunctionDefinitions();

public Object evaluate(Object[] args) throws Exception;

public interface FunctionDefinition{ Class<?>[] getArgTypes(); Class<?> getReturnType(); }

}

A partial implementation of the AISLFunction interface is provided by the AbstractAISLFunction class. A developer can simply extend this class and then provide an appropriate constructor and implement the appropriate evaluate method. An example is given below. The function match returns a boolean value according to the match of a string value with a given regular expression pattern. Its signature is thus:

boolean match(string str, string pattern)

the class Match.class below is an implementation of this function. In the constructor, the function declares its name and its parameters. The method evaluate(Object[] args), which must be implemented to comply with the interface AISLFunction, performs some casting of the parameters and then redirects the evaluation to another evaluate function (Note, in this case, as the function is not overloaded, there is no actual need for a separate evaluate method, here it has been added for clarity).

public class Match extends AbstractAISLFunction{

public Match(){ setName("match"); setFunctionDefinitions( new FunctionDefinitionImpl(Boolean.class, String.class, String.class) ); }

public Object evaluate(Object[] args) throws Exception{ return evaluate((String)args[0], (String)args[1]);

}

private Boolean evaluate(String str, String pattern){ return str.matches(pattern); }

}

Extending Support for Access To Remote Resources

To be added

Adding new Importers

To be added

Appendix - AISL Grammar formal specification

The following EBNF rules define the syntax of the AISL scripting language

1. Program ::= ( Instruction )*

1. Instruction ::= ( VariableDeclaration ";" | Statement )

1. Statement ::= StatementExpression ";" | SwitchStatement | IfStatement | ForeachStatement

1. SwitchStatement ::= "switch" "(" Expression ")" "{" ( SwitchBlock )* "}"

1. SwitchBlock ::= ( "case" Expression ":" ( Instruction )* "break;" | "default" ":" ( Instruction )* "break;" )

1. IfStatement ::= "if" "(" Expression ")" IfBlock ( "else" ElseBlock )?

1. ElseBlock ::= "{" ( Instruction )* "}"

1. IfBlock ::= "{" ( Instruction )* "}"

1. ForeachStatement ::= "foreach" <IDENTIFIER> "in" ( Expression | ForRange ) ForBlock

1. ForRange ::= "[" Expression "to" Expression ( "," Expression )? "]"

1. ForBlock ::= "{" ( Instruction )* "}"

1. VariableDeclaration ::= Type VariableDeclarator ( "," VariableDeclarator )*

1. VariableDeclarator ::= <IDENTIFIER> ( "=" Expression )?

1. Type ::= BuiltinType

1. BuiltinType ::= ( "boolean" | "int" | "float" | "list" ( "<" Type ">" )? | "file" | "string" | "collection" | "resource" | "relationship" )

1. StatementExpression ::= PrimaryExpression ( "=" Expression )?

1. PrimaryExpression ::= ( Literal | Function | Variable | Constructor ) ( Selection )*

1. Literal ::= ( <INTEGER_LITERAL> | <FLOATING_POINT_LITERAL> | <STRING_LITERAL> | BooleanLiteral )

1. BooleanLiteral ::= ( "true" | "false" )

1. Variable ::= Name

1. Name ::= <IDENTIFIER>

1. Function ::= Name Arguments

1. Arguments ::= "(" ( Expression )? ( "," Expression )* ")"

1. Constructor ::= ModelObjectConstructor | ListConstructor

1. ModelObjectConstructor ::= CollectionConstructor | ResourceConstructor | RelationshipConstructor

1. CollectionConstructor ::= "collection" "::" <IDENTIFIER> "[" Expression "]" "{" ( PropertyAssignment ( "," PropertyAssignment )* )? "}"

1. ResourceConstructor ::= "resource" "::" <IDENTIFIER> "[" Expression "]" "in" CollectionsList "{" ( PropertyAssignment ( "," PropertyAssignment )* )? "}"

1. CollectionsList ::= Expression ( "," Expression )*

1. RelationshipConstructor ::= "relationship" "::" <IDENTIFIER> "(" Expression "," Expression ")" "[" Expression "]" "{" ( PropertyAssignment ( "," PropertyAssignment )* )? "}"

1. PropertyAssignment ::= <IDENTIFIER> "=" Expression

1. ListConstructor ::= "{" ( Expression )? ( "," Expression )* "}"

1. Selection ::= PropertySelection | ElementSelection

1. PropertySelection ::= "." <IDENTIFIER>

1. ElementSelection ::= "[" Expression "]"

1. Expression ::= OrExpression

1. OrExpression ::= ExclusiveOrExpression ( "|" ExclusiveOrExpression )*

1. ExclusiveOrExpression ::= AndExpression ( "^" AndExpression )*

1. AndExpression ::= EqualityExpression ( "&" EqualityExpression )*

1. EqualityExpression ::= RelationalExpression ( ( "==" | "!=" ) RelationalExpression )*

1. RelationalExpression ::= AdditiveExpression ( ( "<" | ">" | "<=" | ">=" ) AdditiveExpression )*

1. AdditiveExpression ::= MultiplicativeExpression ( ( "+" | "-" ) MultiplicativeExpression )*

1. MultiplicativeExpression ::= UnaryExpression ( ( "*" | "/" | "%" ) UnaryExpression )*

1. UnaryExpression ::= ( ( "+" | "-" | "!" ) PrimaryExpression | PrimaryExpression )