Difference between revisions of "ResultSet Framework"

Latest revision as of 15:03, 14 July 2008

THIS SECTION OF GCUBE DOCUMENTATION IS CURRENTLY UNDER UPDATE.

ResultSet Framework

Introduction

The ResultSet Framework provides the enabling mechanism to pass by reference data between components and / or services that can be hosted in the same or remote nodes. It utilizes the WSRF Framework for keeping state and offers value adding opeartions for encapsulating remote and local calls, paging of content, data movement, remote processing and more described in available code documentation.

Implementation Overview

The framework consists of 4 sub components :

1. the ResultSet library
2. the ResultSet Service
3. the ResultSet Client
4. the ResultSet Garbage Collector

• The ResultSet is the enabling element that creates a linked list of pages holding the records that are part of the specific result set. It can iterate over the created parts and perform various operations on the structure of the list and the contents themselves. Its operations are mosty part specific bu they can also affect the entire result chain. Once a page of content has been created and the next in line has been initialized the part cannot be altered. Once it has been declared that the authoring of a result set has finished, the entire chain of it is immutible.
• The ResultSet Service is merely a WS front-end to the ResultSet library. It creates and manages WS-Resources holding rferences to instances of the ResultSet components. These instances are responsible to hold state regarding the iteration over the chain of pages and for accessing / modifying / querying the data held.
• The ResultSet Client is a library giving access to both high and low level operations that can be performed on a specific ResultSet component instance and its underlying data. It enables handling of ResultSets that are both wrapped through a WS front-end and that are locally manipulated though direct java invocations (within JVM).
• The ResultSet Garbage Collector is a deamon-like program that can be initialized to clear up resources used by the ResultSet Framework (removing filesystem resources and destroying created WS-Resources)

Dependencies

• ResultSetLibrary
  • jdk 1.5
  • WS-Core
• ResultSetService
  • jdk 1.5
  • WS-Core
  • ResultSetLibrary
• ResultSetClient
  • jdk 1.5
  • WS-Core
  • ResultSetLibrary
  • ResultSetService
• ResultSetGarbageCollector
  • jdk 1.5
  • WS-Core
  • ResultSetLibrary
  • ResultSetService stubs

Usage Example

The ResultSets that can be created are content independent but depending on the content inserted different operations can be performed. So different readers and writers have been implemented to offer different levels of versatility depending on the content. Here he will mainly focus on the XML authoring and retrieving which is most commonly used and thouroughly tested up to this point. As more types of readers / writers become popular additional documentation will be made available

The code examples are not exaustive in error checking and should only be considered as usage templates

ResultSet usage Example code

Creating an RS

  /*A method that creates a new RSXMLWriter and populates it with some results*/
  public static RSXMLWriter createRSWriter() throws Exception{
     /*Create a new writer with the default parametrizable values*/ 
     RSXMLWriter writer=RSXMLWriter.getRSXMLWriter(); 
     /*Create a new writer which should have as a condition for paging inserted results either having 
     30 records per page or having a total size surpasing 1024 bytes */ 
     //writer=RSXMLWriter.getRSXMLWriter(30,1024); 
     /*Create a new writer and set a lifetime property of 1000 millisecs. Also set that the paging 
     condition will be having 10 records per page or a total size surpasing teh default value*/ 
     //writer=RSXMLWriter.getRSXMLWriter(new PropertyElementBase[]{new PropertyElementLifeSpanGC(1000)}); 
     //writer.setRecsPerPart(10); 
     /*Add a new result with the given id, collection, rank and payload*/ 
     writer.addResults(new ResultElementGeneric("id1","collection1","rank1","payload")); 
     /*Add a new result with the given id, collection, rank and payload*/ 
     writer.addResults(new ResultElementGeneric("id2","collection1","rank2","payload")); 
     /*Add an array of results with the given id, collection, rank and payload*/ 
     writer.addResults(new ResultElementBase[]{new ResultElementGeneric("id3","collection1","rank3","payload"),
        new ResultElementGeneric("id4","collection1","rank4","payload")}); 
     /*Give back the writer*/ 
     /*One should ALWAYS close the RS he is creating in order to make the full payload available to readers*/ 
     writer.close(); 
     return writer; 
  }

Retrieving a locator

From a writer

  /*This method retrieves an instance of an RSLocator capable of identifying a ResultSet authored by the provided RSXMLWriter. Depending on the type of 
  resource requested, this identifier can either be an identifier pointing to the local node (filesystem) or an idenitfier capable of pinpointing the RS 
  through a web service using the WS-Resource pattern. One can create many locators (of the same or different) type for the same authored RS*/
  public static RSLocator getLocator(RSXMLWriter writer) throws Exception{
     RSLocator locator=null; 
     /*Retrieves a locator identifying the RS in the local node*/ 
     //locator=writer.getRSLocator(new RSResourceLocalType()); 
     /*Retrieves a locator identifying the RS through WS-Resource pattern accesible from any node. The 
     The service pointed to must be in the localhost. This method can be used when manipulating the RSXMLWriter 
     outside container context. Otherwise the following method can be used*/ 
     //locator=writer.getRSLocator(new RSResourceWSRFType("http://localhost:8080/wsrf/services/diligentproject/searchservice/ResultSetService")); 
     /*Retrieves a locator identifying the RS through WS-Resource pattern accesible from any node. The 
     default constructor utilizes the container context (which must be available) to compose the localy hosted 
     ResultSetService that must be deployed*/ 
     locator=writer.getRSLocator(new RSResourceWSRFType()); 
     return locator; 
  }

From a reader

  /*This method retrieves an instance of an RSLocator capable of identifying the ResultSet read by the provided RSXMLReader. Everythime the method is called    
  the same resource is identified. Both types  of locators can be serialized to string and utilized to initialize a new RSLocator*/
  public static RSLocator getLocator(RSXMLReader reader) throws Exception{
     RSLocator locator=null; 
     /*Retrieves a locator identifying the RS. The type is the same as the one used to initialize the RSXMLReader*/ 
     locator=reader.getRSLocator(); 
     return new RSLocator(locator.getLocator()); 
  }

Reading an RS

Page by Page

  /*This method instantiates a new RSXMLReader to point to the RS identified by the provided RSLocator and iterates over each page retrieving the contents*/
  public static void readRS(RSLocator locator) throws Exception {
     PropertyElementBase []props=reader.getProperties(PropertyElementEstimationCount.class,PropertyElementEstimationCount.propertyType);
     System.out.println(((PropertyElementEstimationCount)props[0]).toXML());
     RSXMLReader reader=RSXMLReader.getRSXMLReader(locator); 
     while(!reader.isLast()){
        reader.getNumberOfResults(); 
        /*Retrieve the records in the current page as instances of the provided class that extends the ResultElementBase*/ 
        ResultElementBase []res=reader.getResults(ResultElementGeneric.class); 
        reader.getNextPart(); 
     } 
     reader.getFirstPart(); 
     do{
        reader.getFullPayload(); 
     }while(reader.getNextPart()); 
  }

Iterating

  /*This method instantiates a new RSXMLReader to point to the RS identified by the provided RSLocator and iterates over each record retrieving the    
  contents*/
  public static void iterateRS(RSLocator locator) throws Exception {
     RSXMLIterator iter=RSXMLReader.getRSXMLReader(locator).getRSIterator(); 
     while(iter.hasNext()){
        ResultElementGeneric elem=(ResultElementGeneric)iter.next(ResultElementGeneric.class); 
        /*Retrieve the DocID of the retrieved record*/ 
        if(elem!=null) elem.getRecordAttributes(ResultElementGeneric.RECORD_ID_NAME); 
     } 
  }

Extending Base elements

Two types of elements are currently extendable in the context of the Framework.

• PropertyElementBase - Which is used to add / retrieve property elements describing the authored RSs
• ResultElementBase - Which is used to add / retrieve records from the RS

These elements must be available to both the producer and the consumer as library elements in order to be used. of course this is not mandatory and a consumer can ustilize a differnt element to retrieve the records that a consumer inserted using some other element. It is in the repsonsibility of the producer / consumer to synchronize the elements they use. These elements serve mainly as containers for the actual payload that should be inserted / retrieved possibly adding some common handling functionality on top of that payload.

Extending PropertyElementBase

  /*This is ane example of extending the PropertyElementBase to derive Property Element that is used to add WSRF EndpointReferenceType serializations to 
  the RS head page*/
  public class PropertyElementWSEPR extends PropertyElementBase{
     /* The Type of the Property this Property element produces*/ 
     public static String propertyType="WS-EPR"; 
     private String epr=null; 
     /* Default contructor required by the PropertyElementBase in order to instantiate the class using reflection*/ 
     public PropertyElementWSEPR(){} 
     /* Initializes a new PropertyElementWSEPR with the given EndpointReferenceType serialization */ 
     public PropertyElementWSEPR(String epr) throws Exception{
        this.epr=epr; 
        setType(PropertyElementWSEPR.propertyType); 
     } 
     /* The usefull property payload from the PropertyElementWSEPR point of view as valid xml */ 
     public String toXML() throws Exception{ 
        return this.epr; 
     } 
     /* The usefull property payload from the PropertyElementWSEPR point of view as returned by 
     PropertyElementWSEPR.fromXML */ 
     public void fromXML(String xml) throws Exception{
        this.epr=xml; 
     } 
  }

Extending ResultElementBase

  /* A Result element that extends the ResultElementBase and can be used to insert and retrieve xml records from an RS*/
  public class ResultElementFoo extends ResultElementBase{
     /* The name of the attribute holding a desired record attribute */ 
     public static final String RECORD_FOO_NAME="Foo"; 
     private String payload; 
     /* Default contructor nessecary for the framework to instantiate the ResultElementFoo though reflection */ 
     public ResultElementFoo(){} 
     /* Creates a new link ResultElementFoo with the provided Foo attribute value and payload */ 
     public ResultElementFoo(String foo,String payload) throws Exception{
        Vector<RecordAttribute> at=new Vector<RecordAttribute>(); 
        at.add(new RecordAttribute(ResultElementFoo.RECORD_FOO_NAME,foo)); 
        setRecordAttributes(at.toArray(new RecordAttribute[0])); 
        this.payload=payload; 
     } 
     private void setPayload(String payload){
        this.payload=payload; 
     } 
     public String getPayload(){
        return this.payload; 
     } 
     /* The Attributes that can be defined for a record are handled internally by the ResultElementBase and can be 
     set / accessed using the Base elements respective methods*/ 
     public String getFooValue(){
        return getRecordAttributes(ResultElementFoo.RECORD_FOO_NAME)[0].getAttrValue(); 
     } 
     /* The xml representation of the record payload */ 
     public String toXML() throws Exception{
        return payload; 
     } 
     /* The payload of the result element as returned by the ResultElementFoo.toXML */ 
     public void fromXML(String xml) throws Exception{
        setPayload(xml); 
     } 
  }

Complex Operations

RS creation within Workflows

In the context of a workflow it might be desirable to send to the asking component a locator capable of identifying the RS that the producer will be populating. This is best done in a non blocking manner in a background thread.

  /*This method creates an RS and starts a background thread populating the RS. It then returns the locator to the RS that is beeing populated*/
  public static RSLocator populateRS() throws Exception{
     RSXMLWriter writer=RSXMLWriter.getRSXMLWriter(); 
     BackgroundPopulating rst=new BackgroundPopulating(writer); 
     rst.start(); 
     return writer.getRSLocator(new RSResourceLocalType()); 
  }

  /*This extends Thread in order to enable populating of the RS the RSXMLWriters it is initialized with points to*/
  public class BackgroundPopulating extends Thread{
     RSXMLWriter writer=null; 
     /*The RSXMLWriter to use for populating the RS*/
     public BackgroundPopulating(RSXMLWriter writer){
        this.writer=writer; 
     } 
     /*Populate the RS*/
     public void run(){
        try{
           writer.addResults(new ResultElementGeneric("id1","collection1","rank1","payload")); 
           writer.addResults(new ResultElementGeneric("id2","collection1","rank2","payload")); 
           writer.addResults(new ResultElementBase[]{new ResultElementGeneric("id3","collection1","rank3","payload"),
               new ResultElementGeneric("id4","collection1","rank4","payload")}); 
           writer.close(); 
        }catch(Exception e){
           e.printStackTrace(); 
        } 
     } 
  }

Partial Localization

  /*This method localizes a ResultSet keeping the top count records of the identified throught the RSLocator RS. The localization is in terms of the node    
  that hosts the RS content and not the Resource type. This can be further defined by the RSResourceType*/
  public static RSLocator localizePart(RSLocator locator,int count) throws Exception {
     RSXMLReader reader=RSXMLReader.getRSXMLReader(locator); 
     reader.isLocal(); //false 
     //RSXMLReader readerTop=reader.keepTop(count); 
     //readerTop.isLocal(); //false 
     //RSXMLReader readerLocalJVM=readerTop.makeLocal(new RSResourceLocalType()); 
     //readerLocalJVM.isLocal(); //true 
     //RSXMLReader readerLocalWS=readerTop.makeLocal(new RSResourceWSRFType()); 
     //readerLocalWS.isLocal(); //true 
     RSXMLReader readerLocalTop=reader.makeLocal(new RSResourceLocalType(),count); 
     readerLocalTop.isLocal(); //true 
     return readerLocalTop.getRSLocator(); 
  }

RS xPath filtering

  /*This method scans through every record of the RS and evaluates the provided xPath expression against each. If the evaluation returns some result, the    
  record is inserted in a new RS. A new RSXMLReader is created holding the matching record. The xPath expression must start with a reference to the current    
  node (eg .//[..])*/
  public static RSLocator filterRS(RSLocator locator,String xPath) throws Exception {
     RSXMLReader reader=RSXMLReader.getRSXMLReader(locator); 
     return reader.filter(xPath).getRSLocator(); 
  }

RS as Flow Control Mechanism

The ResultSet Framework offers the additional functionality of synchronizing result production with result consumption. This functionality is simulated in the following attached code

ResultSet as Flow Contol Mechanism Example code

Pool Of RS Writers

Another feature that is available for usage is a pool of pre-created writers with already created locators. In cases of container heavy load and of WS based locators, this pool can save a significant amount of time in the creation of a writer. In the context of a service for example on can staticly define a factory of writers that include a configurable pool of precreated writers and use this factory to retrieve the writers to use. The code below shows this in sudo-code.

import org.diligentproject.searchservice.searchlibrary.rsclient.elements.pool.PoolConfig;
import org.diligentproject.searchservice.searchlibrary.rsclient.elements.pool.PoolObjectConfig;
import org.diligentproject.searchservice.searchlibrary.rsclient.elements.pool.RSPoolObject;
import org.diligentproject.searchservice.searchlibrary.rswriter.RSWriterFactory;
import org.diligentproject.searchservice.searchlibrary.rswriter.RSXMLWriter;

public class ExampleService implements gCubeRIStateICallback {
/**
* Factory of rs writers
*/
  private static RSWriterFactory factory=null; //the factory that will be used and include the pool
  static{
     PoolConfig config=new PoolConfig(); //the pool configuration
     PoolObjectConfig oConf=new PoolObjectConfig(); //configuration item
     try{
        oConf.FlowControl=false; //the specific writers should not allow flow control
        oConf.MaxSize=20; //the maximum size of the specific writer pool
        oConf.MinSize=8; //the minimum size (threshlod to repopulate)
        oConf.ObjectType=RSPoolObject.PoolObjectType.WriterXML; //the type of writer
        oConf.ResourceType=RSPoolObject.PoolObjectResourceType.WSRFType; //the type of locator to pre-contsruct
        oConf.ServiceEndPoint=null; //not staticly created but in service context
        config.add(oConf); //add this object in the pool
     }catch(Exception e){
        log.error("Could not initialize factory pool. Continuing",e);
     }
     factory=new RSWriterFactory(config);
  }
  public String doWork(DoWork params) throws SomeBaseFault{
     Vector<String> results=MyLibrary.search(params.getQuery);
     //background add records
     PropertyElementBase [] props=new PropertyElementBase []{
        new PropertyElementEstimationCount(results.size(),results.size(),results.size())};
     // without the factory this call would have been RSXMLWriter.getRSXMLWriter(props)
     return writer=factory.getRSXMLWriter(props).getRSLocator(new RSResourceWSRFType();
  }
}

--Gpapanikos 13:14, 14 November 2007 (EET)