How to Interact with the DataMiner by client

Introduction

Here we show how to invoke an algorithm residing on the DataMiner (DM), from outside an e-Infrastructure or from a client.

Notes on Authorization

The username required to use our clients is the one of the Web portal, e.g. john.smith, whereas the authorization token identifies the VRE to interact with. The token must be generated from the VRE home page through the "Service Authorization Token" tool that is present in the VRE home page. The user name is displayed by the same panel.

HTTP interface

Each DM execution has an HTTP link associated that can be obtained by pressing the "Show" button on the DM interface during the execution of an experiment. By changing the parameters after DataInputs parameter it is possible to manage new computations. Note that a gCube token is required(change <VRE token> with your token).

Examples of WPS requests to the DataMiner D4Science cluster:

 
 https://dataminer.d4science.org/wps/WebProcessingService?Request=GetCapabilities&Service=WPS&gcube-token=<VRE token>

 https://dataminer.d4science.org/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<VRE token>&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN

 https://dataminer.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=<VRE token>&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN&DataInputs=OccurrencePointsClusterLabel=OccClustersTest;epsilon=10;min_points=1;OccurrencePointsTable=http://goo.gl/VDzpch;FeaturesColumnNames=depthmean|sstmnmax|salinitymean;

The Web Processing Service(WPS) v1.0.0 Specification:

File:WebProcessingServiceWPSv1.pdf

Java Client

The 52North WPS Client can be used to interact with DataMiner. Note that interactions should use either basic HTTP authentication or add the gcube-token parameter to the requests.

Plain Java Client

Here is one example of Java call, that invokes an algorithm asynchronously through a template file

package it.test;
 
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.io.Reader;
import java.io.StringWriter;
import java.io.Writer;
import java.net.HttpURLConnection;
import java.net.ProtocolException;
import java.net.URL;
import java.net.URLConnection;
 
public class InvokeDataMinerViaPost {
 
	static String dataMinerUrl = "https://dataminer-prototypes.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0";
	static String token = "your-token-here";
 
	private static void pipe(Reader reader, Writer writer) throws IOException {
		char[] buf = new char[1024];
		int read = 0;
		while ((read = reader.read(buf)) >= 0) {
			writer.write(buf, 0, read);
		}
		writer.flush();
	}
 
	public static void postData(Reader data, URL endpoint, Writer output) throws Exception {
		HttpURLConnection urlc = null;
		try {
			urlc = (HttpURLConnection) endpoint.openConnection();
			try {
				urlc.setRequestMethod("POST");
			} catch (ProtocolException e) {
				throw new Exception("Shouldn't happen: HttpURLConnection doesn't support POST??", e);
			}
			urlc.setDoOutput(true);
			urlc.setDoInput(true);
			urlc.setUseCaches(false);
			urlc.setAllowUserInteraction(false);
			urlc.setRequestProperty("Content-type", "text/xml; charset=" + "UTF-8");
 
			OutputStream out = urlc.getOutputStream();
 
			try {
				Writer writer = new OutputStreamWriter(out, "UTF-8");
				pipe(data, writer);
				writer.close();
			} catch (IOException e) {
				throw new Exception("IOException while posting data", e);
			} finally {
				if (out != null)
					out.close();
			}
 
			InputStream in = urlc.getInputStream();
			try {
				Reader reader = new InputStreamReader(in);
				pipe(reader, output);
				reader.close();
			} catch (IOException e) {
				throw new Exception("IOException while reading response", e);
			} finally {
				if (in != null)
					in.close();
			}
 
		} catch (IOException e) {
			throw new Exception("Connection error (is server running at " + endpoint + " ?): " + e);
		} finally {
			if (urlc != null)
				urlc.disconnect();
		}
	}
 
	public static String getStatus(String endpoint) {
		String result = null;
 
			// Send a GET request to the servlet
			try {
				// Send data
				String urlStr = endpoint;
 
				URL url = new URL(urlStr);
				URLConnection conn = url.openConnection();
				conn.setConnectTimeout(120000);
				conn.setReadTimeout(120000);
 
				// Get the response
				BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
				StringBuffer sb = new StringBuffer();
				String line;
				while ((line = rd.readLine()) != null) {
					sb.append(line);
				}
				rd.close();
				result = sb.toString();
			} catch (Exception e) {
				e.printStackTrace();
			}
 
		return result;
	}
 
 
 
	public static void main(String args[]) throws Exception{
		String template = "templateDBScan.txt";
//		String template = "templateNOV_QRA.txt";
 
		StringWriter sw = new StringWriter();
		FileReader fr = new FileReader(new File(template));
 
		postData(fr , new URL(dataMinerUrl+"&gcube-token="+token), sw);
 
		fr.close();
 
		String answer = sw.toString();
 
		String statusLocation = answer.substring(answer.indexOf("statusLocation=\"")+"statusLocation=\"".length(), answer.indexOf("\">")); 
 
		System.out.println(sw.toString());
		System.out.println(statusLocation);
 
		String status = getStatus(statusLocation+"&gcube-token="+token);
 
		while (!(status.contains("wps:ProcessSucceeded") || status.contains("wps:ProcessFailed"))){
			System.out.println(status);
			status = getStatus(statusLocation+"&gcube-token="+token);
			Thread.sleep(5000);
		}
 
 
		System.out.println(status);
 
		if (status.contains("wps:ProcessFailed"))
			System.out.println("Process Failed!");
		else{
			String UrlToOutput = status.substring(status.lastIndexOf("<d4science:Data>")+"<d4science:Data>".length(), status.lastIndexOf("</d4science:Data>"));
			System.out.println("Url to output:"+UrlToOutput);
		}
	}
}

Another template file example is available here

WPS Client

A WPS Client is also available to invoke algorithms from R (R 3.3.1). Click this link to get the client and examples.

• An example of model (BiOnym) invoked via POST and asynchronously with input data table embedding and uploading is available at this link
• An example of long running model (Ichtyop) invoked via POST and asynchronously is available at this link.
• An example of model (XMeans) invoked via POST and asynchronously with input data table embedding and uploading is available at this link

The template to invoke a process, copies the process description that can be obtained by building a link like this:

 https://<DataMiner Cluster>/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<your_token>&Identifier=<Process ID>

For example, the BiOnym process description is:

 https://dataminer-cloud1.d4science.org/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<your_token>&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.generators.BIONYM

Note, that the following information is required in order to reconstruct a description like the one above in the VRE you are using:

1. Your token: you can find this in the home page of the VRE;

2. The DataMiner cluster (can change depending on the VRE): after executing your target process through the Web interface, press "Show". The DataMiner cluster address is the one reported after "https://";

3. The Process ID: in the link contained in the "Show" window, the Process ID is the value of the "Identifier=" parameter.

QGIS

QGIS supports a number of clients for WPS. We advice using the GeoLabs WPS plugin (add the plugin repository at http://geolabs.fr/plugins.xml) and enter the https://dataminer.d4science.org/wps/WebProcessingService endpoint.

QGIS WPS interface

Maps comparison with QGIS

Related Links

• DataMiner Manager
• Statistical Algorithms Importer (SAI)
• Data Mining page