Difference between revisions of "Full Text Index"

From Gcube Wiki
Jump to: navigation, search
(→‎Implementation Overview)
(→‎Implementation Overview)
Line 11: Line 11:
 
It is important to note that none of the three services have to reside on the same server; they are only connected through WebService calls and the DILIGENT CMS. The following illustration shows the information flow and responsibilities for the different services used to implement the Full Text Index:
 
It is important to note that none of the three services have to reside on the same server; they are only connected through WebService calls and the DILIGENT CMS. The following illustration shows the information flow and responsibilities for the different services used to implement the Full Text Index:
  
−
(illustration will be improved shortly...)
+
(illustration will be improved shortly... )
 
<pre>
 
<pre>
 
________________________________
 
________________________________
 
| |
 
| |
−
| |
+
|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
−
| |
+
|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
−
| |
+
|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
 
|    So Pretty Index Design...  |
 
|    So Pretty Index Design...  |
−
| |
+
|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
−
| |
+
|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
−
| |
+
|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
 
|________________________________|
 
|________________________________|
 
</pre>
 
</pre>
Line 40: Line 40:
 
</ROWSET>
 
</ROWSET>
 
</pre>
 
</pre>
 +
 +
===IndexType===
 +
How the different fields in the ROWSET should be handled by the Index, and how the different fields in an Index should be handled during a query, is specified through an IndexType; an XML document conforming to the IndexType schema. An IndexType contains a field list which contains all the fields which should be indexed and/or stored in order to be presented in the query results, along with a specification of how each of the fields should be handled. The following is a possible IndexType for the type of ROWSET shown above:
 +
 +
<pre>
 +
    <index-type>
 +
        <field-list>
 +
            <field name="title" lang="en">
 +
                <index>yes</index>
 +
                <store>yes</store>
 +
                <return>yes</return>
 +
                <tokenize>yes</tokenize>
 +
                <sort>no</sort>
 +
                <boost>1.0</boost>
 +
            </field>
 +
            <field name="contents" lang="en>
 +
                <index>yes</index>
 +
                <store>no</store>
 +
                <return>no</return>
 +
                <tokenize>yes</tokenize>
 +
                <sort>no</sort>
 +
                <boost>1.0</boost>
 +
            </field>
 +
            <field name="references" lang="en>
 +
                <index>yes</index>
 +
                <store>no</store>
 +
                <return>no</return>
 +
                <tokenize>yes</tokenize>
 +
                <sort>no</sort>
 +
                <boost>1.0</boost>
 +
            </field>
 +
        </field-list>
 +
    </index-type>
 +
</pre>
 +
 +
Fields present in the ROWSET but not in the IndexType will be skipped. The elements under each "field" element are used to define that field should be handled, and they should contain either "yes" or "no". The meaning of each of them is explained bellow:
 +
 +
*index
 +
**specifies whether the specific field should be indexed or not (ie. whether the index should look for hits within this field)
 +
*store
 +
**specifies whether the field should be stored in its original format to be returned in the results from a query.
 +
*return
 +
**specifies whether a stored field should be returned in the results from a query. A field must have been stored to be returned. (This element is not available in the currently deployd indices)
 +
*tokenize
 +
**specifies whether the field should be tokenized. Should usually contain "yes".
 +
*sort
 +
**Not used
 +
*boost
 +
**Not used
 +
 +
For more complex content types, one can also specify sub-fields as in the following example:
 +
 +
<pre>
 +
 +
<index-type>
 +
    <field-list>
 +
        <field name="contents">
 +
            <index>yes</index>
 +
            <store>no</store>
 +
            <tokenize>yes</tokenize>
 +
            <sort>no</sort>
 +
            <boost>1.0</boost>
 +
 +
            <!-- (subfields of contents) -->
 +
            <field name="title">
 +
                <index>yes</index>
 +
                <store>yes</store>
 +
                <tokenize>yes</tokenize>
 +
                <sort>no</sort>
 +
                <boost>1.0</boost>
 +
 +
                <!-- (subfields of title which itself is a subfield of contents) -->
 +
                <field name="bookTitle">
 +
                    <index>yes</index>
 +
                    <store>yes</store>
 +
                    <tokenize>yes</tokenize>
 +
                    <sort>no</sort>
 +
                    <boost>1.0</boost>
 +
                </field>
 +
                <field name="chapterTitle">
 +
                    <index>yes</index>
 +
                    <store>yes</store>
 +
                    <tokenize>yes</tokenize>
 +
                    <sort>no</sort>
 +
                    <boost>1.0</boost>
 +
                </field>
 +
            </field>
 +
 +
            <field name="foreword">
 +
                <index>yes</index>
 +
                <store>yes</store>
 +
                <tokenize>yes</tokenize>
 +
                <sort>no</sort>
 +
                <boost>1.0</boost>
 +
                </field>
 +
            <field name="startChapter">
 +
                <index>yes</index>
 +
                <store>yes</store>
 +
                <tokenize>yes</tokenize>
 +
                <sort>no</sort>
 +
                <boost>1.0</boost>
 +
            </field>
 +
            <field name="endChapter">
 +
                <index>yes</index>
 +
                <store>yes</store>
 +
                <tokenize>yes</tokenize>
 +
                <sort>no</sort>
 +
                <boost>1.0</boost>
 +
            </field>
 +
        </field>
 +
 +
        <!-- (not a subfield) -->
 +
        <field name="references">
 +
            <index>yes</index>
 +
            <store>no</store>
 +
            <tokenize>yes</tokenize>
 +
            <sort>no</sort>
 +
            <boost>1.0</boost>
 +
        </field>
 +
 +
    </field-list>
 +
</index-type>
 +
 +
</pre>
 +
Querying the field "contents" in an index using this IndexType would return hitsin all its sub-fields, which is all fields except references. Querying the field "title" would return hits in both "bookTitle" and "chapterTitle" in addition to hits in the  "title" field. Querying the field "startChapter" would only return hits in from  "startChapter" since this field does not contain any sub-fields. Please be aware that using sub-fields adds extra fields in the index, and therefore uses more disks pace.
 +
 +
We currently have five standard index types, loosely based on the available metadata schemas. However any data can be indexed using each, as long as the RowSet follows the IndexType:
 +
*index-type-default-1.0 (DublinCore)
 +
*index-type-TEI-2.0
 +
*index-type-eiDB-1.0
 +
*index-type-iso-1.0
 +
*index-type-FT-1.0
 +
 +
The IndexType of a FullTextIndexManagement Service resource can be changed as long as no FullTextIndexBatchUpdater resources have connected to it. The reason for this limitation is that the processing of fields should be the same for all documents in an index; all documents in an index should be handled according to the same IndexType.
 +
 +
The IndexType of a FullTextIndexLookup Service resource is originally retrieved from the FullTextIndexManagement Service resource it is connected to. However, the "returned" property can be changed at any time in order to change which fields are returned. Keep in mind that only fields which have a "stored" attribute set to "yes" can have their "returned" field altered to return content.

Revision as of 16:12, 31 May 2007

Introduction

The Full Text Index is responsible for providing quick full text data retrieval capabilities in the DILIGENT environment.

Implementation Overview

Services

The full text index is implemented through three services. They are all implemented according to the Factory pattern:

  • The FullTextIndexManagement Service represents an index manager. There is a one to one relationship between an Index and a Management instance, and their life-cycles are closely related; an Index is created by creating an instance (resource) of FullTextIndexManagement Service, and an index is removed by terminating the corresponding FullTextIndexManagement resource. The FullTextIndexManagement Service should be seen as an interface for managing the life-cycle and properties of an Index, but it is not responsible for feeding or querying its index. In addition, a FullTextIndexManagement Service resource does not store the content of its Index locally, but contains references to content stored in Content Management Service.
  • The FullTextIndexBatchUpdater Service is responsible for feeding an Index. One FullTextIndexBatchUpdater Service resource can only update a single Index, but one Index can be updated by multiple FullTextIndexBatchUpdater Service resources. Feeding is accomplished by instantiating a FullTextIndexBatchUpdater Service resources with the EPR of the FullTextIndexManagement resource connected to the Index to update, and connecting the updater resource to a ResultSet containing the content to be fed to the Index.
  • The FullTextIndexLookup Service is responsible for creating a local copy of an index, and exposing interfaces for querying and creating statistics for the index. One FullTextIndexLookup Service resource can only replicate and lookup a single instance, but one Index can be replicated by any number of FullTextIndexLookup Service resources. Updates to the Index will be propagated to all FullTextIndexLookup Service resources replicating that Index.

It is important to note that none of the three services have to reside on the same server; they are only connected through WebService calls and the DILIGENT CMS. The following illustration shows the information flow and responsibilities for the different services used to implement the Full Text Index:

(illustration will be improved shortly... ) 

			 ________________________________
			|				 |
			|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
			|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
			|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
			|    So Pretty Index Design...   |
			|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
			|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
			|•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘•∘|
			|________________________________|

RowSet

The content to be fed into an Index, must be served as a ResultSet (ResultSet Framework) containing XML documents conforming to the ROWSET schema. This is a very simple schema, declaring that a document (ROW element) should contain of any number of FIELD elements with a name attribute and the text to be indexed for that field. The following is a simple but valid ROWSET containing two documents: 

<ROWSET>
    <ROW id="doc1">
        <FIELD name="title">How to create an Index</FIELD>
        <FIELD name="contents">Just read the WIKI</FIELD>
    </ROW>
    <ROW id="doc2">
        <FIELD name="title">How to create a Nation</FIELD>
        <FIELD name="contents">Talk to the UN</FIELD>
        <FIELD name="references">un.org</FIELD>
    </ROW>
</ROWSET>

IndexType

How the different fields in the ROWSET should be handled by the Index, and how the different fields in an Index should be handled during a query, is specified through an IndexType; an XML document conforming to the IndexType schema. An IndexType contains a field list which contains all the fields which should be indexed and/or stored in order to be presented in the query results, along with a specification of how each of the fields should be handled. The following is a possible IndexType for the type of ROWSET shown above: 

    <index-type>
        <field-list>
            <field name="title" lang="en">
                <index>yes</index>
                <store>yes</store>
                <return>yes</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>
            </field>
            <field name="contents" lang="en>
                <index>yes</index>
                <store>no</store>
                <return>no</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>
            </field>
            <field name="references" lang="en>
                <index>yes</index>
                <store>no</store>
                <return>no</return>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>
            </field>
        </field-list>
    </index-type>

Fields present in the ROWSET but not in the IndexType will be skipped. The elements under each "field" element are used to define that field should be handled, and they should contain either "yes" or "no". The meaning of each of them is explained bellow:

  • index
    • specifies whether the specific field should be indexed or not (ie. whether the index should look for hits within this field)
  • store
    • specifies whether the field should be stored in its original format to be returned in the results from a query.
  • return
    • specifies whether a stored field should be returned in the results from a query. A field must have been stored to be returned. (This element is not available in the currently deployd indices)
  • tokenize
    • specifies whether the field should be tokenized. Should usually contain "yes".
  • sort
    • Not used
  • boost
    • Not used

For more complex content types, one can also specify sub-fields as in the following example: 


<index-type>
    <field-list>
        <field name="contents">
            <index>yes</index>
            <store>no</store>
            <tokenize>yes</tokenize>
            <sort>no</sort>
            <boost>1.0</boost>

            <!-- (subfields of contents) -->
            <field name="title">
                <index>yes</index>
                <store>yes</store>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>

                <!-- (subfields of title which itself is a subfield of contents) -->
                <field name="bookTitle">
                    <index>yes</index>
                    <store>yes</store>
                    <tokenize>yes</tokenize>
                    <sort>no</sort>
                    <boost>1.0</boost>
                </field>
                <field name="chapterTitle">
                    <index>yes</index>
                    <store>yes</store>
                    <tokenize>yes</tokenize>
                    <sort>no</sort>
                    <boost>1.0</boost>
                </field>
            </field>

            <field name="foreword">
                <index>yes</index>
                <store>yes</store>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>
                </field>
            <field name="startChapter">
                <index>yes</index>
                <store>yes</store>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>
            </field>
            <field name="endChapter">
                <index>yes</index>
                <store>yes</store>
                <tokenize>yes</tokenize>
                <sort>no</sort>
                <boost>1.0</boost>
            </field>
        </field>

        <!-- (not a subfield) -->
        <field name="references">
            <index>yes</index>
            <store>no</store>
            <tokenize>yes</tokenize>
            <sort>no</sort>
            <boost>1.0</boost>
        </field>

    </field-list>
</index-type>

Querying the field "contents" in an index using this IndexType would return hitsin all its sub-fields, which is all fields except references. Querying the field "title" would return hits in both "bookTitle" and "chapterTitle" in addition to hits in the "title" field. Querying the field "startChapter" would only return hits in from "startChapter" since this field does not contain any sub-fields. Please be aware that using sub-fields adds extra fields in the index, and therefore uses more disks pace.

We currently have five standard index types, loosely based on the available metadata schemas. However any data can be indexed using each, as long as the RowSet follows the IndexType:

  • index-type-default-1.0 (DublinCore)
  • index-type-TEI-2.0
  • index-type-eiDB-1.0
  • index-type-iso-1.0
  • index-type-FT-1.0

The IndexType of a FullTextIndexManagement Service resource can be changed as long as no FullTextIndexBatchUpdater resources have connected to it. The reason for this limitation is that the processing of fields should be the same for all documents in an index; all documents in an index should be handled according to the same IndexType.

The IndexType of a FullTextIndexLookup Service resource is originally retrieved from the FullTextIndexManagement Service resource it is connected to. However, the "returned" property can be changed at any time in order to change which fields are returned. Keep in mind that only fields which have a "stored" attribute set to "yes" can have their "returned" field altered to return content.

Retrieved from "https://wiki.gcube-system.org/index.php?title=Full_Text_Index&oldid=1545"