How to Interact with the Statistical Manager by client

Prerequisites

IDE: Eclipse Java EE IDE for Web Developers. Version: 3.7+

Introduction

Here we show how to invoke an algorithm residing on the Statistical Manager (SM), from outside, by means of a thin Java client. Let's start creating a project using the eclipse IDE that is mavenized according to our indications. After having mavenized the project in eclipse you have to add the following dependency for the SM client library.

Maven coordinates

The maven artifact coordinates are:

<dependency>
   <groupId>org.gcube.data.analysis</groupId>
   <artifactId>statistical-manager-cl</artifactId>
   <version>[1.0.0-SNAPSHOT,2.0.0-SNAPSHOT)</version>
</dependency>
<dependency>
   <groupId>org.gcube.core</groupId>
   <artifactId>common-fw-clients</artifactId>
   <version>[1.0.0-SNAPSHOT, 2.0.0-SNAPSHOT)</version>
</dependency>
<dependency>
  <groupId>org.gcube.contentmanagement</groupId>
  <artifactId>storage-manager-wrapper</artifactId>
  <version>2.0.2-3.0.0</version>
</dependency>
<dependency>
  <groupId>org.gcube.contentmanagement</groupId>
  <artifactId>storage-manager-core</artifactId>
  <version>2.0.3-3.0.0</version>
</dependency>

Features

Get the list of algorithms

Given a VRE Scope in the e-Infrastructure (selected among these for Production and set to /gcube/devsec/devVRE for Development environment), the list of available algorithms is retrievable in the following way:

	ScopeProvider.instance.set("/gcube/devsec/devVRE");
		StatisticalManagerFactory factory = StatisticalManagerDSL
		  .createStateful().build();
		SMListGroupedAlgorithms groups = factory.getAlgorithms();
		String algorithmId = null;
		String algDescr = null;
		for (SMGroupedAlgorithms group : groups.thelist()) {
 
		 for (SMAlgorithm algorithm : group.thelist()) {
			System.out.println(algorithm.name());
			if (algorithm.name().equals("SAMPLEONTABLE")) { // this is just an example to filter on one of the algorithms
				algorithmId = algorithm.name();
				algDescr = algorithm.description();
			}
		 }
		}

Get the Inputs of an Algorithm

Referring to the previous code, once selected an algorithm, it is possible to retrieve its parameters in the following way:

		SMParameters smParams = factory.getAlgorithmParameters(algorithmId);
		SMInputEntry[] list = new SMInputEntry[smParams.list().size()];
		int i = 0;
 
		for (SMParameter smParam : smParams.list()) {
			list[i++] = new SMInputEntry(smParam.name(), "value");
		}

Execute an Algorithm

Before invoking an algoritm, you must setup it by reusing the maps of Strings contained in the SMInputEntry Object. Furthermore, you can specify a user name and other optional metadata. The SM uses maps of Strings to specify all the input parameters.

		SMComputationRequest request = new SMComputationRequest();
		SMComputationConfig config = new SMComputationConfig();
		config.parameters(new SMEntries(list));
		config.algorithm(algorithmId);
		request.user("user");
		request.title(algorithmId);
		request.description(algDescr);
		request.config(config);
		String computationId = factory.executeComputation(request);

Retrieve the Status and the Output

Follow this code to check the status of a computation and eventually retrieve the output:

SMComputation computation = factory.getComputation(computationId);
		SMOperationStatus status = SMOperationStatus.values()[computation
		    .operationStatus()];
 
		float percentage = 0;
		if (status == SMOperationStatus.RUNNING) {
		    SMOperationInfo infos = factory.getComputationInfo(computationId,"user");
		    percentage = Float.parseFloat(infos.percentage());
		} else if (status == SMOperationStatus.COMPLETED) {
			SMAbstractResource abstractResource = computation
					.abstractResource();
			SMResource smResource = abstractResource.resource();
			int resourceTypeIndex = smResource.resourceType();
			SMResourceType smResType = SMResourceType.values()[resourceTypeIndex];
			switch (smResType) {
					case FILE:
						SMFile fileRes = (SMFile) smResource;
						break;
					case OBJECT:
						SMObject objRes = (SMObject) smResource;
 
						if (objRes.name().contentEquals(
								PrimitiveTypes.MAP.toString())) {
						} else if (objRes.name().contentEquals(
								PrimitiveTypes.IMAGES.toString())) {
						} else
 
							break;
					case TABULAR:
						SMTable tableRes = (SMTable) smResource;
						break;
					}
 
			smResource.name();
			smResource.resourceId();
			smResource.description();
 
			}

Import your own data

You can import a dataset on the SM in order to (i) store it in the e-Infrastructure and (ii) to use it for calculations. This operation is necessary to ensure high efficiency.

File tempFile = new File("path");
StatisticalManagerDataSpace dataSpace = StatisticalManagerDSL.dataSpace().build();;
String user = "user";
String template = "GENERIC";
 
TableTemplates tableTemplate=null;
for (TableTemplates t: TableTemplates.values())
	if (template.contentEquals(t.toString())) {
		tableTemplate = t;
		break;
	}
//first row is header of column
boolean hasHeader=false;
String delimiter=",";
String id = dataSpace.createTableFromCSV(tempFile, hasHeader, delimiter , "", "table name", tableTemplate, "table descritpion", user);

Complete Example

A complete example that retrieves a map of strings from an algorithm

import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLStreamHandler;
import java.util.LinkedHashMap;
import java.util.Map;
 
import org.gcube.common.scope.api.ScopeProvider;
import org.gcube.contentmanager.storageclient.model.protocol.smp.SMPURLConnection;
import org.gcube.data.analysis.statisticalmanager.proxies.StatisticalManagerDSL;
import org.gcube.data.analysis.statisticalmanager.proxies.StatisticalManagerFactory;
import org.gcube.data.analysis.statisticalmanager.stubs.types.SMAlgorithm;
import org.gcube.data.analysis.statisticalmanager.stubs.types.SMComputationConfig;
import org.gcube.data.analysis.statisticalmanager.stubs.types.SMComputationRequest;
import org.gcube.data.analysis.statisticalmanager.stubs.types.SMGroupedAlgorithms;
import org.gcube.data.analysis.statisticalmanager.stubs.types.SMListGroupedAlgorithms;
import org.gcube.data.analysis.statisticalmanager.stubs.types.SMOperationStatus;
import org.gcube.data.analysis.statisticalmanager.stubs.types.SMResourceType;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMAbstractResource;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMComputation;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMEntries;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMFile;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMInputEntry;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMObject;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMOperationInfo;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMResource;
import org.gcube.data.analysis.statisticalmanager.stubs.types.schema.SMTable;
import org.gcube.dataanalysis.ecoengine.datatypes.enumtypes.PrimitiveTypes;
 
import com.thoughtworks.xstream.XStream;
 
public class TestStatistical {
 
	public static void listAlgorithms(String[] args) throws Exception {
		ScopeProvider.instance.set("/gcube/devsec/devVRE");
		StatisticalManagerFactory factory = StatisticalManagerDSL.createStateful().build();
		SMListGroupedAlgorithms groups = factory.getAlgorithms();
 
		for (SMGroupedAlgorithms group : groups.thelist()) {
			for (SMAlgorithm algorithm : group.thelist()) {
				System.out.println("Name:" + algorithm.name());
 
			}
		}
	}
 
	public static InputStream getStorageClientInputStream(String url) throws Exception {
		URL u = new URL(null, url, new URLStreamHandler() {
			@Override
			protected URLConnection openConnection(URL u) throws IOException {
				return new SMPURLConnection(u);
			}
		});
		return u.openConnection().getInputStream();
	}
 
	public static void displayOutput(SMResource smResource, SMResourceType smResType) throws Exception {
		switch (smResType) {
		case FILE:
			SMFile fileRes = (SMFile) smResource;
			System.out.println("Output is a file");
			break;
		case OBJECT:
			SMObject objRes = (SMObject) smResource;
			if (objRes.name().contentEquals(PrimitiveTypes.MAP.toString())) {
				System.out.println("Output is a map");
				Map<String, String> outputMap = getMap(objRes);
			} else if (objRes.name().contentEquals(PrimitiveTypes.IMAGES.toString())) {
				System.out.println("Output are images");
			} else
				System.out.println("Output is other");
			break;
		case TABULAR:
			SMTable tableRes = (SMTable) smResource;
			System.out.println("Output is a table");
			break;
		}
	}
 
	public static Map<String, String> getMap(SMObject objRes) throws Exception {
		InputStream is = getStorageClientInputStream(objRes.url());
		// object serializer
		XStream xstream = new XStream();
		xstream.alias("org.gcube_system.namespaces.data.analysis.statisticalmanager.types.SMObject", SMObject.class);
		xstream.alias("org.gcube_system.namespaces.data.analysis.statisticalmanager.types.SMFile", SMFile.class);
		xstream.alias("org.gcube_system.namespaces.data.analysis.statisticalmanager.types.SMResource", SMResource.class);
		xstream.alias("org.gcube_system.namespaces.data.analysis.statisticalmanager.types.SMTable", SMTable.class);
 
		@SuppressWarnings("unchecked")
		Map<String, SMResource> smMap = (Map<String, SMResource>) (xstream.fromXML(is));
		is.close();
		LinkedHashMap<String, String> outputmap = new LinkedHashMap<String, String>();
		for (String key : smMap.keySet()) {
			SMResource smres = smMap.get(key);
			int resourceTypeIndex = smres.resourceType();
			SMResourceType smsubResType = SMResourceType.values()[resourceTypeIndex];
			System.out.println("ResourceType:" + smsubResType);
			if (smsubResType == SMResourceType.OBJECT) {
				SMObject obje = (SMObject) smres;
				String outstring = obje.url();
 
				System.out.println("Object:" + new String(outstring.getBytes(), "UTF-8"));
 
				outputmap.put(key, outstring);
			}
		}
 
		System.out.println("Map retrieved with size: " + smMap.size());
		return outputmap;
	}
 
	public static float checkComputationStatus(String scope, String computationId, String user) throws Exception {
		ScopeProvider.instance.set(scope);
		StatisticalManagerFactory factory = StatisticalManagerDSL.createStateful().build();
 
		SMComputation computation = factory.getComputation(computationId);
		SMOperationStatus status = SMOperationStatus.values()[computation.operationStatus()];
 
		float percentage = 0;
		if (status == SMOperationStatus.RUNNING) {
			SMOperationInfo infos = factory.getComputationInfo(computationId, user);
			percentage = Float.parseFloat(infos.percentage());
			System.out.println("Percentage:" + percentage);
		} else if ((status == SMOperationStatus.COMPLETED) || (status == SMOperationStatus.FAILED)) {
			percentage = 100;
			SMAbstractResource abstractResource = computation.abstractResource();
			SMResource smResource = abstractResource.resource();
			int resourceTypeIndex = smResource.resourceType();
			SMResourceType smResType = SMResourceType.values()[resourceTypeIndex];
			displayOutput(smResource, smResType);
			System.out.println("SM resource Name: " + smResource.name());
			System.out.println("SM resource ID: " + smResource.resourceId());
			System.out.println("SM resource Description: " + smResource.description());
		}
 
		return percentage;
	}
 
	public static void startAlgorithm(String algorithmName,String username, String scope, SMInputEntry[] list) throws Exception {
		ScopeProvider.instance.set(scope);
		StatisticalManagerFactory factory = StatisticalManagerDSL.createStateful().build();
 
		SMComputationRequest request = new SMComputationRequest();
		SMComputationConfig config = new SMComputationConfig();
		System.out.println("Running Sample on table");
//		SMParameters smParams = factory.getAlgorithmParameters(algorithmName); To manage algorithms parameters and their default values
 
		config.parameters(new SMEntries(list));
 
		config.algorithm(algorithmName);
		request.user(username);
		request.title(algorithmName);
		request.description(algorithmName);
		request.config(config);
 
		String computationId = factory.executeComputation(request);
		float percentage = 0;
		while (percentage < 100) {
			percentage = checkComputationStatus(scope, computationId, username);
			Thread.sleep(1000);
		}
	}
 
	public static void main(String [] args) throws Exception{
 
		String scope = "/gcube/devsec/devVRE";
		String username = "gianpaolo.coro";
		String algorithm = "SAMPLEONTABLE";
 
		SMInputEntry[] list = new SMInputEntry[3];
		int i = 0;
 
		list[i++] = new SMInputEntry("ResourceName", "CatalogOfLife2010");
		list[i++] = new SMInputEntry("DatabaseName", "col2oct2010");
		list[i++] = new SMInputEntry("TableName", "common_names");
 
		startAlgorithm(algorithm, username, scope, list);
 
	}
}

WPS Client

A WPS Client is also available to invoke algorithms from R. Click this link to get the client and examples.

• An example of long running model (Ichtyop) invoked via POST and asynchronously is available at this link.
• An example of model (XMeans) invoked via POST and asynchronously with input data table embedding and uploading is available at this link

Related Links

Statistical Manager Tutorial

How-to Implement Algorithms for the Statistical_Manager

Notes on authorization

The username required to use our clients is the one of the Web portal, e.g. john.smith, whereas the authorization token identifies the VRE to interact with. The token must be generated from the VRE home page through the "Service Authorization Token" tool that is present in the VRE home page.

If you are uncertain about your user name, go on the portal and follow these steps:

1 - click on your name (upper right corner)

2 - click on "Edit Profile Manually"

3 - select "Details" from the right panel

The "Screen Name" reports your user name