Index Management Framework

From Gcube Wiki
Revision as of 17:05, 25 April 2014 by Alex.antoniadi (Talk | contribs) (→‎Deployment Instructions)

Jump to: navigation, search

Contextual Query Language Compliance

The gCube Index Framework consists of the Index Service which provides both FullText Index and Forward Index capabilities. All of them are able to answer CQL queries. The CQL relations that each of them supports, depends on the underlying technologies. The mechanisms for answering CQL queries, using the internal design and technologies, are described later for each case. The supported relations are:

  • Index Service : =, ==, within, >, >=, <=, adj, fuzzy, proximity, within

Index Service

The Index Service is responsible for providing quick full text data retrieval and forward index capabilities in the gCube environment.

Index Service exposes a REST API, thus it can be used by different general purpose libraries that support REST. For example, the following HTTP GET call is used in order to query the index: 

http://{host}/index-service-1.0.0-SNAPSHOT/{resourceID}/query?queryString=((e24f6285-46a2-4395-a402-99330b326fad = tuna) and (((gDocCollectionID == 8dc17a91-378a-4396-98db-469280911b2f)))) project ae58ca58-55b7-47d1-a877-da783a758302

Index Service is consisted by a few components that are available in our Maven repositories with the following coordinates: 

<!-- index service web app -->
<groupId>org.gcube.index</groupId>
<artifactId>index-service</artifactId>
<version>...</version>
 
 
<!-- index service commons library -->
<groupId>org.gcube.index</groupId>
<artifactId>index-service-commons</artifactId>
<version>...</version>
 
<!-- index service client library -->
<groupId>org.gcube.index</groupId>
<artifactId>index-service-client-library</artifactId>
<version>...</version>
 
<!-- helper common library -->
<groupId>org.gcube.index</groupId>
<artifactId>indexcommon</artifactId>
<version>...</version>

Implementation Overview

Services

The new index is implemented through one service. It is implemented according to the Factory pattern:

  • The Index Service represents an index node. It is used for managements, lookup and updating the node. It is a compaction of the 3 services that were used in the old Full Text Index.

The following illustration shows the information flow and responsibilities for the different services used to implement the Index Service:

Generic Editor

It is actually a wrapper over ElasticSearch and each IndexNode has a 1-1 relationship with an ElasticSearch Node. For this reason creation of multiple resources of IndexNode service is discouraged, instead the best case is to have one resource (one node) at each container that consists the cluster.

Clusters can be created in almost the same way that a group of lookups and updaters and a manager were created in the old Full Text Index (using the same indexID). Having multiple clusters within a scope is feasible but discouraged because it usually better to have a large cluster than multiple small clusters.

The cluster distinction is done through a clusterID which is either the same as the indexID or the scope. The deployer of the service can choose between these two by setting the value of defaultSameCluster variable in the deploy.properties file true of false respectively.

Example 

defaultSameCluster=true

or 

defaultSameCluster=false

ElasticSearch, which is the underlying technology of the new Index Service, can configure the number of replicas and shards for each index. This is done by setting the variables noReplicas and noShards in the deploy.properties file

Example: 

noReplicas=1
noShards=2


Highlighting is a new supported feature by Full Text Index (also supported in the old Full Text Index). If highlighting is enabled the index returns a snippet of the matching query that is performed on the presentable fields. This snippet is usually a concatenation of a number of matching fragments in those fields that match queries. The maximum size of each fragment as well as the maximum number of the fragments that will be used to construct a snippet can be configured by setting the variables maxFragmentSize and maxFragmentCnt in the deploy.properties file respectively:

Example: 

maxFragmentCnt=5
maxFragmentSize=80


The folder where the data of the index are stored can be configured by setting the variable dataDir in the deploy-jndi-config.xml file (if the variable is not set the default location is the folder that the container runs).

Example : 

dataDir=./data

In order to configure whether to use Resource Registry or not (for translation of field ids to field names) we can change the value of the variable useRRAdaptor in the deploy-jndi-config.xml

Example : 

useRRAdaptor=true


Since the Index Service creates resources for each Index instance that is running (instead of running multiple Running Instances of the service) the folder where the instances will be persisted locally have to be set in the variable resourcesFoldername in the deploy-jndi-config.xml.

Example : 

resourcesFoldername=/tmp/resources/index

Finally, the hostname of the node as well as the scope that the node is running on have to set in the in the variable resourcesFoldername in the deploy-jndi-config.xml'.

Example : 

hostname=dl015.madgik.di.uoa.gr
scope=/gcube/devNext

CQL capabilities implementation

Full Text Index uses Lucene as its underlying technology. A CQL Index-Relation-Term triple has a straightforward transformation in lucene. This transformation is explained through the following examples:

CQL triple explanation lucene equivalent
title adj "sun is up" documents with this phrase in their title title:"sun is up"
title fuzzy "invorvement" documents with words "similar" to invorvement in their title title:invorvement~
allIndexes = "italy" (documents have 2 fields; title and abstract) documents with the word italy in some of their fields title:italy OR abstract:italy
title proximity "5 sun up" documents with the words sun, up inside an interval of 5 words in their title title:"sun up"~5
date within "2005 2008" documents with a date between 2005 and 2008 date:[2005 TO 2008]

In a complete CQL query, the triples are connected with boolean operators. Lucene supports AND, OR, NOT(AND-NOT) connections between single criteria. Thus, in order to transform a complete CQL query to a lucene query, we first transform CQL triples and then we connect them with AND, OR, NOT equivalently.

RowSet

The content to be fed into an Index, must be served as a ResultSet containing XML documents conforming to the ROWSET schema. This is a very simple schema, declaring that a document (ROW element) should contain of any number of FIELD elements with a name attribute and the text to be indexed for that field. The following is a simple but valid ROWSET containing two documents: 

<ROWSET idxType="IndexTypeName" colID="colA" lang="en">
    <ROW>
        <FIELD name="ObjectID">doc1</FIELD>
        <FIELD name="title">How to create an Index</FIELD>
        <FIELD name="contents">Just read the WIKI</FIELD>
    </ROW>
    <ROW>
        <FIELD name="ObjectID">doc2</FIELD>
        <FIELD name="title">How to create a Nation</FIELD>
        <FIELD name="contents">Talk to the UN</FIELD>
        <FIELD name="references">un.org</FIELD>
    </ROW>
</ROWSET>

The attributes idxType and colID are required and specify the Index Type that the Index must have been created with, and the collection ID of the documents under the <ROWSET> element. The lang attribute is optional, and specifies the language of the documents under the <ROWSET> element. Note that for each document a required field is the "ObjectID" field that specifies its unique identifier.

IndexType

How the different fields in the ROWSET should be handled by the Index, and how the different fields in an Index should be handled during a query, is specified through an IndexType; an XML document conforming to the IndexType schema. An IndexType contains a field list which contains all the fields which should be indexed and/or stored in order to be presented in the query results, along with a specification of how each of the fields should be handled. The following is a possible IndexType for the type of ROWSET shown above: 

    <index-type>
        <field-list>
            <field name="title">
                <index>yes</index>
                <store>yes</store>
                <return>yes</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <highlightable>yes</highlightable>
                <boost>1.0</boost>
            </field>
            <field name="contents">
                <index>yes</index>
                <store>yes</store>
                <return>yes</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>
            </field>
            <field name="references">
                <index>yes</index>
                <store>no</store>
                <return>no</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <highlightable>no</highlightable> <!-- will not be included in the highlight snippet -->
                <boost>1.0</boost>
            </field>
            <field name="gDocCollectionID">
                <index>yes</index>
		<store>yes</store>
		<return>yes</return>
		<tokenize>yes</tokenize>
		<sort>no</sort>
		<boost>1.0</boost>
	    </field>
	    <field name="gDocCollectionLang">
		<index>yes</index>
		<store>yes</store>
		<return>yes</return>
		<tokenize>yes</tokenize>
		<sort>no</sort>
		<boost>1.0</boost>
	    </field>
        </field-list>
    </index-type>

Note that the fields "gDocCollectionID", "gDocCollectionLang" are always required, because, by default, all documents will have a collection ID and a language ("unknown" if no collection is specified). Fields present in the ROWSET but not in the IndexType will be skipped. The elements under each "field" element are used to define how that field should be handled, and they should contain either "yes" or "no". The meaning of each of them is explained bellow:

  • index
specifies whether the specific field should be indexed or not (ie. whether the index should look for hits within this field)
  • store
specifies whether the field should be stored in its original format to be returned in the results from a query.
  • return
specifies whether a stored field should be returned in the results from a query. A field must have been stored to be returned. (This element is not available in the currently deployed indices)
  • highlightable
specifies whether a returned field should be included or not in the highlight snippet. If not specified then every returned field will be included in the snippet.
  • tokenize
Not used
  • sort
Not used
  • boost
Not used

We currently have five standard index types, loosely based on the available metadata schemas. However any data can be indexed using each, as long as the RowSet follows the IndexType:

  • index-type-default-1.0 (DublinCore)
  • index-type-TEI-2.0
  • index-type-eiDB-1.0
  • index-type-iso-1.0
  • index-type-FT-1.0

Query language

The Full Text Index receives CQL queries and transforms them into Lucene queries. Queries using wildcards will not return usable query statistics.


Deployment Instructions

In order to deploy and run Index Service on a node we will need the following:

  • index-service-{version}.war
  • smartgears-distribution-{version}.tar.gz (to publish the running instance of the service on the IS and be discoverable)
    • see here for installation
  • an application server (such as Tomcat, JBoss, Jetty)

Before starting the application service we should provide the configuration needed by Resource Registry. This configuration should be placed in the folder $CATALINA/conf/infrastructure.properties and the variables that need to be set are: infrastructure, scopes and clientMode (clientMode should be set to false)

Example : 

infrastructure=gcube
scopes=devNext
clientMode=false


Finally, the hostname of the node as well as the scope that the node is running on have to set in the in the variables hostname and scope in the deploy.properties'.

Example : 

hostname=dl015.madgik.di.uoa.gr
scope=/gcube/devNext

Usage Example

Create an Index Service Node, feed and query using the corresponding client library

The following example demonstrate the usage of the IndexClient and IndexServiceClient. Both are created according to the Builder pattern. 

final String scope = "/gcube/devNext"; 
 
// create a client for the given scope (we can provide endpoint as extra filter)
IndexFactoryClient indexFactoryClient = new IndexFactoryClient.Builder().scope(scope).build();
 
factoryClient.createResource("myClusterID", scope);
 
// create a client for the same scope (we can provide endpoint, resourceID, collectionID, clusterID, indexID as extra filters)
IndexClient indexClient = new IndexFactoryClient.Builder().scope(scope).build();
 
try{
  	indexClient.feedLocator(locator);
  	indexClient.query(query);
} catch (IndexException) {
  	// handle the exception
}



Index Common library

The Index Common library is another component which is meant to be used internally by the other index components. It provides some common functionality required by the various index types, such as interfaces, XML parsing utilities and definitions of some common attributes of all indices. The jar containing the library should be deployed on every node where at least one index component is deployed.

Retrieved from "https://wiki.gcube-system.org/index.php?title=Index_Management_Framework&oldid=21363"