Latest revision as of 14:37, 19 June 2014

Contextual Query Language Compliance

The gCube Index Framework consists of the Index Service which provides both FullText Index and Forward Index capabilities. All of them are able to answer CQL queries. The CQL relations that each of them supports, depends on the underlying technologies. The mechanisms for answering CQL queries, using the internal design and technologies, are described later for each case. The supported relations are:

• Index Service : =, ==, within, >, >=, <=, adj, fuzzy, proximity, within

Index Service

The Index Service is responsible for providing quick full text data retrieval and forward index capabilities in the gCube environment.

Index Service exposes a REST API, thus it can be used by different general purpose libraries that support REST. For example, the following HTTP GET call is used in order to query the index:

http://{host}/index-service-1.0.0-SNAPSHOT/{resourceID}/query?queryString=((e24f6285-46a2-4395-a402-99330b326fad = tuna) and (((gDocCollectionID == 8dc17a91-378a-4396-98db-469280911b2f)))) project ae58ca58-55b7-47d1-a877-da783a758302

Index Service is consisted by a few components that are available in our Maven repositories with the following coordinates:

<!-- index service web app -->
<groupId>org.gcube.index</groupId>
<artifactId>index-service</artifactId>
<version>...</version>
 
 
<!-- index service commons library -->
<groupId>org.gcube.index</groupId>
<artifactId>index-service-commons</artifactId>
<version>...</version>
 
<!-- index service client library -->
<groupId>org.gcube.index</groupId>
<artifactId>index-service-client-library</artifactId>
<version>...</version>
 
<!-- helper common library -->
<groupId>org.gcube.index</groupId>
<artifactId>indexcommon</artifactId>
<version>...</version>

Implementation Overview

Services

The new index is implemented through one service. It is implemented according to the Factory pattern:

• The Index Service represents an index node. It is used for managements, lookup and updating the node. It is a compaction of the 3 services that were used in the old Full Text Index.

The following illustration shows the information flow and responsibilities for the different services used to implement the Index Service:

Generic Editor

It is actually a wrapper over ElasticSearch and each IndexNode has a 1-1 relationship with an ElasticSearch Node. For this reason creation of multiple resources of IndexNode service is discouraged, instead the best case is to have one resource (one node) at each container that consists the cluster.

Clusters can be created in almost the same way that a group of lookups and updaters and a manager were created in the old Full Text Index (using the same indexID). Having multiple clusters within a scope is feasible but discouraged because it usually better to have a large cluster than multiple small clusters.

The cluster distinction is done through a clusterID which is either the same as the indexID or the scope. The deployer of the service can choose between these two by setting the value of defaultSameCluster variable in the deploy.properties file true of false respectively.

Example

defaultSameCluster=true

or

defaultSameCluster=false

ElasticSearch, which is the underlying technology of the new Index Service, can configure the number of replicas and shards for each index. This is done by setting the variables noReplicas and noShards in the deploy.properties file

Example:

noReplicas=1
noShards=2

Highlighting is a new supported feature by Full Text Index (also supported in the old Full Text Index). If highlighting is enabled the index returns a snippet of the matching query that is performed on the presentable fields. This snippet is usually a concatenation of a number of matching fragments in those fields that match queries. The maximum size of each fragment as well as the maximum number of the fragments that will be used to construct a snippet can be configured by setting the variables maxFragmentSize and maxFragmentCnt in the deploy.properties file respectively:

Example:

maxFragmentCnt=5
maxFragmentSize=80

The folder where the data of the index are stored can be configured by setting the variable dataDir in the deploy.properties file (if the variable is not set the default location is the folder that the container runs).

Example :

dataDir=./data

In order to configure whether to use Resource Registry or not (for translation of field ids to field names) we can change the value of the variable useRRAdaptor in the deploy.properties

Example :

useRRAdaptor=true

Since the Index Service creates resources for each Index instance that is running (instead of running multiple Running Instances of the service) the folder where the instances will be persisted locally have to be set in the variable resourcesFoldername in the deploy.properties.

Example :

resourcesFoldername=./resources/index

Finally, the hostname of the node as well as the port and the scope that the node is running on have to set in the in the variables hostname and scope in the deploy.properties.

Example :

hostname=dl015.madgik.di.uoa.gr
port=8080
scope=/gcube/devNext

CQL capabilities implementation

Full Text Index uses Lucene as its underlying technology. A CQL Index-Relation-Term triple has a straightforward transformation in lucene. This transformation is explained through the following examples:

In a complete CQL query, the triples are connected with boolean operators. Lucene supports AND, OR, NOT(AND-NOT) connections between single criteria. Thus, in order to transform a complete CQL query to a lucene query, we first transform CQL triples and then we connect them with AND, OR, NOT equivalently.

RowSet

The content to be fed into an Index, must be served as a ResultSet containing XML documents conforming to the ROWSET schema. This is a very simple schema, declaring that a document (ROW element) should contain of any number of FIELD elements with a name attribute and the text to be indexed for that field. The following is a simple but valid ROWSET containing two documents:

<ROWSET idxType="IndexTypeName" colID="colA" lang="en">
    <ROW>
        <FIELD name="ObjectID">doc1</FIELD>
        <FIELD name="title">How to create an Index</FIELD>
        <FIELD name="contents">Just read the WIKI</FIELD>
    </ROW>
    <ROW>
        <FIELD name="ObjectID">doc2</FIELD>
        <FIELD name="title">How to create a Nation</FIELD>
        <FIELD name="contents">Talk to the UN</FIELD>
        <FIELD name="references">un.org</FIELD>
    </ROW>
</ROWSET>

The attributes idxType and colID are required and specify the Index Type that the Index must have been created with, and the collection ID of the documents under the <ROWSET> element. The lang attribute is optional, and specifies the language of the documents under the <ROWSET> element. Note that for each document a required field is the "ObjectID" field that specifies its unique identifier.

IndexType

How the different fields in the ROWSET should be handled by the Index, and how the different fields in an Index should be handled during a query, is specified through an IndexType; an XML document conforming to the IndexType schema. An IndexType contains a field list which contains all the fields which should be indexed and/or stored in order to be presented in the query results, along with a specification of how each of the fields should be handled. The following is a possible IndexType for the type of ROWSET shown above:

    <index-type>
        <field-list>
            <field name="title">
                <index>yes</index>
                <store>yes</store>
                <return>yes</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <highlightable>yes</highlightable>
                <boost>1.0</boost>
            </field>
            <field name="contents">
                <index>yes</index>
                <store>yes</store>
                <return>yes</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>
            </field>
            <field name="references">
                <index>yes</index>
                <store>no</store>
                <return>no</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <highlightable>no</highlightable> <!-- will not be included in the highlight snippet -->
                <boost>1.0</boost>
            </field>
            <field name="gDocCollectionID">
                <index>yes</index>
		<store>yes</store>
		<return>yes</return>
		<tokenize>yes</tokenize>
		<sort>no</sort>
		<boost>1.0</boost>
	    </field>
	    <field name="gDocCollectionLang">
		<index>yes</index>
		<store>yes</store>
		<return>yes</return>
		<tokenize>yes</tokenize>
		<sort>no</sort>
		<boost>1.0</boost>
	    </field>
        </field-list>
    </index-type>

Note that the fields "gDocCollectionID", "gDocCollectionLang" are always required, because, by default, all documents will have a collection ID and a language ("unknown" if no collection is specified). Fields present in the ROWSET but not in the IndexType will be skipped. The elements under each "field" element are used to define how that field should be handled, and they should contain either "yes" or "no". The meaning of each of them is explained bellow:

• index

specifies whether the specific field should be indexed or not (ie. whether the index should look for hits within this field)

• store

specifies whether the field should be stored in its original format to be returned in the results from a query.

• return

specifies whether a stored field should be returned in the results from a query. A field must have been stored to be returned. (This element is not available in the currently deployed indices)

• highlightable

specifies whether a returned field should be included or not in the highlight snippet. If not specified then every returned field will be included in the snippet.

• tokenize

Not used

• sort

Not used

• boost

Not used

We currently have five standard index types, loosely based on the available metadata schemas. However any data can be indexed using each, as long as the RowSet follows the IndexType:

• index-type-default-1.0 (DublinCore)
• index-type-TEI-2.0
• index-type-eiDB-1.0
• index-type-iso-1.0
• index-type-FT-1.0

Query language

The Full Text Index receives CQL queries and transforms them into Lucene queries. Queries using wildcards will not return usable query statistics.

Deployment Instructions

In order to deploy and run Index Service on a node we will need the following:

• index-service-{version}.war
• smartgears-distribution-{version}.tar.gz (to publish the running instance of the service on the IS and be discoverable)
  • see here for installation
• an application server (such as Tomcat, JBoss, Jetty)

There are a few things that need to configured in order for the service to be functional. All the service configuration is done in the file deploy.properties that comes within the service war. Typically, this file should be loaded in the classpath so that it can be read. The default location of this file (in the exploded war) is webapps/service/WEB-INF/classes.

The hostname of the node as well as the port and the scope that the node is running on have to set in the in the variables hostname and scope in the deploy.properties.

Example :

hostname=dl015.madgik.di.uoa.gr
port=8080
scope=/gcube/devNext

Finally, Resource Registry should be configured to not run in client mode. This is done in the deploy.properties by setting:

clientMode=false

NOTE: it is important to note that resourcesFoldername as well as dataDir properties have relative paths in their default values. In some cases these values maybe evaluated by taking into account the folder that the container was started, so in order to avoid problems related to this behavior it is better for these properties to take absolute paths as values.

Usage Example

Create an Index Service Node, feed and query using the corresponding client library

The following example demonstrate the usage of the IndexClient and IndexServiceClient. Both are created according to the Builder pattern.

final String scope = "/gcube/devNext"; 
 
// create a client for the given scope (we can provide endpoint as extra filter)
IndexFactoryClient indexFactoryClient = new IndexFactoryClient.Builder().scope(scope).build();
 
factoryClient.createResource("myClusterID", scope);
 
// create a client for the same scope (we can provide endpoint, resourceID, collectionID, clusterID, indexID as extra filters)
IndexClient indexClient = new IndexFactoryClient.Builder().scope(scope).build();
 
try{
  	indexClient.feedLocator(locator);
  	indexClient.query(query);
} catch (IndexException) {
  	// handle the exception
}

Index Common library

The Index Common library is another component which is meant to be used internally by the other index components. It provides some common functionality required by the various index types, such as interfaces, XML parsing utilities and definitions of some common attributes of all indices. The jar containing the library should be deployed on every node where at least one index component is deployed.