Difference between revisions of "How to Interact with the DataMiner by client"

From Gcube Wiki
Jump to: navigation, search
(HTTP interface)
 
(33 intermediate revisions by 2 users not shown)
Line 2: Line 2:
 
[[Category:Developer's Guide]]
 
[[Category:Developer's Guide]]
 
<!-- END CATEGORIES -->
 
<!-- END CATEGORIES -->
=Prerequisites=
 
One of the following software is required:
 
 
* Firefox or Google Chrome Web browser
 
* IDE: Eclipse Java EE IDE for Web Developers. Version: 3.7+
 
* R 3.3.1
 
* QGIS
 
  
 
=Introduction=
 
=Introduction=
Line 17: Line 10:
  
 
=HTTP interface=
 
=HTTP interface=
Each DM execution has an HTTP link associated that can be obtained by pressing the "Show" button on the [https://wiki.gcube-system.org/gcube/DataMiner_Manager#Execute_an_Experiment DM interface during the execution of an experiment]. By changing the parameters after DataInputs parameter it is possible to manage new computations. Note that a gCube token is required.
+
Each DM execution has an HTTP link associated that can be obtained by pressing the "Show" button on the [https://wiki.gcube-system.org/gcube/DataMiner_Manager#Execute_an_Experiment DM interface during the execution of an experiment]. By changing the parameters after DataInputs parameter it is possible to manage new computations. Note that a gCube token is required(change <span>&lt;VREToken&gt;</span> with your token).
  
 
Examples of WPS requests to the DataMiner D4Science cluster:
 
Examples of WPS requests to the DataMiner D4Science cluster:
  
  http://dataminer.d4science.org/wps/WebProcessingService?Request=GetCapabilities&Service=WPS&gcube-token=<VRE token>
+
<pre style='overflow-x:auto;white-space:pre-wrap;white-space:-moz-pre-wrap;white-space:-pre-wrap;white-space:-o-pre-wrap;word-wrap:break-word;'>
 +
  https://dataminer.d4science.org/wps/WebProcessingService?Request=GetCapabilities&Service=WPS&gcube-token=<VREToken>
 +
</pre>
  
  http://dataminer.d4science.org/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<VRE token>&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN
+
<pre style='overflow-x:auto;white-space:pre-wrap;white-space:-moz-pre-wrap;white-space:-pre-wrap;white-space:-o-pre-wrap;word-wrap:break-word;'>
 +
  https://dataminer.d4science.org/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<VREToken>&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN
 +
</pre>
  
  http://dataminer.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=<VRE token>&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN&DataInputs=OccurrencePointsClusterLabel=OccClustersTest;epsilon=10;min_points=1;OccurrencePointsTable=http://goo.gl/VDzpch;FeaturesColumnNames=depthmean|sstmnmax|salinitymean;
+
<pre style='overflow-x:auto;white-space:pre-wrap;white-space:-moz-pre-wrap;white-space:-pre-wrap;white-space:-o-pre-wrap;word-wrap:break-word;'>
 +
  https://dataminer.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=<VREToken>&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN&DataInputs=OccurrencePointsClusterLabel=OccClustersTest;epsilon=10;min_points=1;OccurrencePointsTable=http://goo.gl/VDzpch;FeaturesColumnNames=depthmean|sstmnmax|salinitymean;
 +
</pre>
 +
 
 +
 
 +
So DataMiner uses the WPS specification and adds a gcube-token to authorize the calls.
 +
The Web Processing Service(WPS) v1.0.0 Specification:
 +
:[[File:WebProcessingServiceWPSv1.pdf|WebProcessingServiceWPSv1.pdf]]
  
 
=Java Client=
 
=Java Client=
 
The [https://wiki.52north.org/Geoprocessing/ClientAPI 52North WPS Client] can be used to interact with DataMiner. Note that interactions should use either basic HTTP authentication or add the gcube-token parameter to the requests.
 
The [https://wiki.52north.org/Geoprocessing/ClientAPI 52North WPS Client] can be used to interact with DataMiner. Note that interactions should use either basic HTTP authentication or add the gcube-token parameter to the requests.
 +
 +
=Plain Java Client=
 +
 +
Here is one example of Java call, that invokes an algorithm asynchronously through a [http://data.d4science.org/S0x1VzJ5YXZaOHNQc3NicTZ1SkpGa01VcEkrcVZmbWdHbWJQNStIS0N6Yz0 template file]
 +
 +
<source lang="java">
 +
package it.test;
 +
 +
import java.io.BufferedReader;
 +
import java.io.File;
 +
import java.io.FileReader;
 +
import java.io.IOException;
 +
import java.io.InputStream;
 +
import java.io.InputStreamReader;
 +
import java.io.OutputStream;
 +
import java.io.OutputStreamWriter;
 +
import java.io.Reader;
 +
import java.io.StringWriter;
 +
import java.io.Writer;
 +
import java.net.HttpURLConnection;
 +
import java.net.ProtocolException;
 +
import java.net.URL;
 +
import java.net.URLConnection;
 +
 +
public class InvokeDataMinerViaPost {
 +
 +
static String dataMinerUrl = "https://dataminer-prototypes.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0";
 +
static String token = "your-token-here";
 +
 +
private static void pipe(Reader reader, Writer writer) throws IOException {
 +
char[] buf = new char[1024];
 +
int read = 0;
 +
while ((read = reader.read(buf)) >= 0) {
 +
writer.write(buf, 0, read);
 +
}
 +
writer.flush();
 +
}
 +
 +
public static void postData(Reader data, URL endpoint, Writer output) throws Exception {
 +
HttpURLConnection urlc = null;
 +
try {
 +
urlc = (HttpURLConnection) endpoint.openConnection();
 +
try {
 +
urlc.setRequestMethod("POST");
 +
} catch (ProtocolException e) {
 +
throw new Exception("Shouldn't happen: HttpURLConnection doesn't support POST??", e);
 +
}
 +
urlc.setDoOutput(true);
 +
urlc.setDoInput(true);
 +
urlc.setUseCaches(false);
 +
urlc.setAllowUserInteraction(false);
 +
urlc.setRequestProperty("Content-type", "text/xml; charset=" + "UTF-8");
 +
 +
OutputStream out = urlc.getOutputStream();
 +
 +
try {
 +
Writer writer = new OutputStreamWriter(out, "UTF-8");
 +
pipe(data, writer);
 +
writer.close();
 +
} catch (IOException e) {
 +
throw new Exception("IOException while posting data", e);
 +
} finally {
 +
if (out != null)
 +
out.close();
 +
}
 +
 +
InputStream in = urlc.getInputStream();
 +
try {
 +
Reader reader = new InputStreamReader(in);
 +
pipe(reader, output);
 +
reader.close();
 +
} catch (IOException e) {
 +
throw new Exception("IOException while reading response", e);
 +
} finally {
 +
if (in != null)
 +
in.close();
 +
}
 +
 +
} catch (IOException e) {
 +
throw new Exception("Connection error (is server running at " + endpoint + " ?): " + e);
 +
} finally {
 +
if (urlc != null)
 +
urlc.disconnect();
 +
}
 +
}
 +
 +
public static String getStatus(String endpoint) {
 +
String result = null;
 +
 +
// Send a GET request to the servlet
 +
try {
 +
// Send data
 +
String urlStr = endpoint;
 +
 +
URL url = new URL(urlStr);
 +
URLConnection conn = url.openConnection();
 +
conn.setConnectTimeout(120000);
 +
conn.setReadTimeout(120000);
 +
 +
// Get the response
 +
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
 +
StringBuffer sb = new StringBuffer();
 +
String line;
 +
while ((line = rd.readLine()) != null) {
 +
sb.append(line);
 +
}
 +
rd.close();
 +
result = sb.toString();
 +
} catch (Exception e) {
 +
e.printStackTrace();
 +
}
 +
 +
return result;
 +
}
 +
 +
 +
 +
public static void main(String args[]) throws Exception{
 +
String template = "templateDBScan.txt";
 +
// String template = "templateNOV_QRA.txt";
 +
 +
StringWriter sw = new StringWriter();
 +
FileReader fr = new FileReader(new File(template));
 +
 +
postData(fr , new URL(dataMinerUrl+"&gcube-token="+token), sw);
 +
 +
fr.close();
 +
 +
String answer = sw.toString();
 +
 +
String statusLocation = answer.substring(answer.indexOf("statusLocation=\"")+"statusLocation=\"".length(), answer.indexOf("\">"));
 +
 +
System.out.println(sw.toString());
 +
System.out.println(statusLocation);
 +
 +
String status = getStatus(statusLocation+"&gcube-token="+token);
 +
 +
while (!(status.contains("wps:ProcessSucceeded") || status.contains("wps:ProcessFailed"))){
 +
System.out.println(status);
 +
status = getStatus(statusLocation+"&gcube-token="+token);
 +
Thread.sleep(5000);
 +
}
 +
 +
 +
System.out.println(status);
 +
 +
if (status.contains("wps:ProcessFailed"))
 +
System.out.println("Process Failed!");
 +
else{
 +
String UrlToOutput = status.substring(status.lastIndexOf("<d4science:Data>")+"<d4science:Data>".length(), status.lastIndexOf("</d4science:Data>"));
 +
System.out.println("Url to output:"+UrlToOutput);
 +
}
 +
}
 +
}
 +
 +
</source>
 +
 +
 +
Another template file example is available [http://data.d4science.org/dlhZbHNqazVGbjBQc3NicTZ1SkpGZ21ZbXk4TW1HSDZHbWJQNStIS0N6Yz0 here]
  
 
=WPS Client=
 
=WPS Client=
A WPS Client is also available to invoke algorithms from R. Click [http://goo.gl/62SbqB this link] to get the client and examples.
+
A WPS Client is also available to invoke algorithms from R (R 3.3.1). Click [https://svn.research-infrastructures.eu/public/d4science/gcube/trunk/data-analysis/RConfiguration/RD4SFunctions/WPSRConnector.zip this link] to get the client and examples.
  
* An example of long running model (Ichtyop) invoked via POST and asynchronously is available at [http://data.d4science.org/TzZMN0hqTysvcWFCV2tNdUhsL2VIT0Nld3Fid1pGblRHbWJQNStIS0N6Yz0 this link].
+
* An example of model (BiOnym) invoked via POST and asynchronously with input data table embedding and uploading is available at [https://svn.research-infrastructures.eu/public/d4science/gcube/trunk/data-analysis/RConfiguration/RD4SFunctions/BiOnym%20from%20R_v3.zip this link]
* An example of model (XMeans) invoked via POST and asynchronously with input data table embedding and uploading is available at [http://goo.gl/OJazer this link]
+
* An example of long running model (Ichtyop) invoked via POST and asynchronously is available at [https://svn.research-infrastructures.eu/public/d4science/gcube/trunk/data-analysis/RConfiguration/RD4SFunctions/IchtyopClient.zip this link].
 +
* An example of model (XMeans) invoked via POST and asynchronously with input data table embedding and uploading is available at [https://svn.research-infrastructures.eu/public/d4science/gcube/trunk/data-analysis/RConfiguration/RD4SFunctions/XMeansClientExample.zip this link]
 +
 
 +
 
 +
The template to invoke a process, copies the process description that can be obtained by building a link like this:
 +
 
 +
<pre>
 +
https://<DataMiner Cluster>/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<your_token>&Identifier=<Process ID>
 +
</pre>
 +
 
 +
For example, the BiOnym process description is:
 +
 
 +
<pre>
 +
https://dataminer-cloud1.d4science.org/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<your_token>&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.generators.BIONYM
 +
</pre>
 +
 
 +
Note, that the following information is required in order to reconstruct a description like the one above in the VRE you are using:
 +
 
 +
1. '''Your token''': you can find this in the home page of the VRE;
 +
 
 +
2. '''The DataMiner cluster''' (can change depending on the VRE): after executing your target process through the Web interface, press "Show". The DataMiner cluster address is the one reported after "https://";
 +
 
 +
3. '''The Process ID''': in the link contained in the "Show" window, the Process ID is the value of the "Identifier=" parameter.
  
 
=QGIS=
 
=QGIS=
QGIS supports a number of clients for WPS. We advice using the [http://planet.qgis.org/plugins/wps/ WPS plugin] and enter the http://dataminer.d4science.org/wps/WebProcessingService endpoint.
+
QGIS supports a number of clients for WPS. We advice using the [http://geolabs.fr/BlogPost;id=80 GeoLabs WPS plugin] (add the plugin repository at http://geolabs.fr/plugins.xml) and enter the https://dataminer.d4science.org/wps/WebProcessingService endpoint.
  
[[Image:qgis.png|frame|center|QGIS WPS interface]]
+
[[Image:Qgis.png|frame|center|QGIS WPS interface]]
  
= Related Links =
+
[[Image:Mapscompqgis.png|frame|center|Maps comparison with QGIS]]
[[DataMiner_Manager | DataMiner Tutorial]]
+
  
[[Data_Mining_Facilities | Data Mining page]]
+
= Related Links =
 +
* [[DataMiner_Manager | DataMiner Manager]]
 +
* [[Statistical_Algorithms_Importer|Statistical Algorithms Importer (SAI)]]
 +
* [[Data_Mining_Facilities | Data Mining page]]

Latest revision as of 17:01, 15 February 2021


Introduction

Here we show how to invoke an algorithm residing on the DataMiner (DM), from outside an e-Infrastructure or from a client.

Notes on Authorization

The username required to use our clients is the one of the Web portal, e.g. john.smith, whereas the authorization token identifies the VRE to interact with. The token must be generated from the VRE home page through the "Service Authorization Token" tool that is present in the VRE home page. The user name is displayed by the same panel.

HTTP interface

Each DM execution has an HTTP link associated that can be obtained by pressing the "Show" button on the DM interface during the execution of an experiment. By changing the parameters after DataInputs parameter it is possible to manage new computations. Note that a gCube token is required(change <VREToken> with your token).

Examples of WPS requests to the DataMiner D4Science cluster:

 https://dataminer.d4science.org/wps/WebProcessingService?Request=GetCapabilities&Service=WPS&gcube-token=<VREToken>
 https://dataminer.d4science.org/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<VREToken>&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN
 https://dataminer.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=<VREToken>&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.clusterers.DBSCAN&DataInputs=OccurrencePointsClusterLabel=OccClustersTest;epsilon=10;min_points=1;OccurrencePointsTable=http://goo.gl/VDzpch;FeaturesColumnNames=depthmean|sstmnmax|salinitymean;


So DataMiner uses the WPS specification and adds a gcube-token to authorize the calls. The Web Processing Service(WPS) v1.0.0 Specification:

File:WebProcessingServiceWPSv1.pdf

Java Client

The 52North WPS Client can be used to interact with DataMiner. Note that interactions should use either basic HTTP authentication or add the gcube-token parameter to the requests.

Plain Java Client

Here is one example of Java call, that invokes an algorithm asynchronously through a template file

package it.test;
 
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.io.Reader;
import java.io.StringWriter;
import java.io.Writer;
import java.net.HttpURLConnection;
import java.net.ProtocolException;
import java.net.URL;
import java.net.URLConnection;
 
public class InvokeDataMinerViaPost {
 
	static String dataMinerUrl = "https://dataminer-prototypes.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0";
	static String token = "your-token-here";
 
	private static void pipe(Reader reader, Writer writer) throws IOException {
		char[] buf = new char[1024];
		int read = 0;
		while ((read = reader.read(buf)) >= 0) {
			writer.write(buf, 0, read);
		}
		writer.flush();
	}
 
	public static void postData(Reader data, URL endpoint, Writer output) throws Exception {
		HttpURLConnection urlc = null;
		try {
			urlc = (HttpURLConnection) endpoint.openConnection();
			try {
				urlc.setRequestMethod("POST");
			} catch (ProtocolException e) {
				throw new Exception("Shouldn't happen: HttpURLConnection doesn't support POST??", e);
			}
			urlc.setDoOutput(true);
			urlc.setDoInput(true);
			urlc.setUseCaches(false);
			urlc.setAllowUserInteraction(false);
			urlc.setRequestProperty("Content-type", "text/xml; charset=" + "UTF-8");
 
			OutputStream out = urlc.getOutputStream();
 
			try {
				Writer writer = new OutputStreamWriter(out, "UTF-8");
				pipe(data, writer);
				writer.close();
			} catch (IOException e) {
				throw new Exception("IOException while posting data", e);
			} finally {
				if (out != null)
					out.close();
			}
 
			InputStream in = urlc.getInputStream();
			try {
				Reader reader = new InputStreamReader(in);
				pipe(reader, output);
				reader.close();
			} catch (IOException e) {
				throw new Exception("IOException while reading response", e);
			} finally {
				if (in != null)
					in.close();
			}
 
		} catch (IOException e) {
			throw new Exception("Connection error (is server running at " + endpoint + " ?): " + e);
		} finally {
			if (urlc != null)
				urlc.disconnect();
		}
	}
 
	public static String getStatus(String endpoint) {
		String result = null;
 
			// Send a GET request to the servlet
			try {
				// Send data
				String urlStr = endpoint;
 
				URL url = new URL(urlStr);
				URLConnection conn = url.openConnection();
				conn.setConnectTimeout(120000);
				conn.setReadTimeout(120000);
 
				// Get the response
				BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
				StringBuffer sb = new StringBuffer();
				String line;
				while ((line = rd.readLine()) != null) {
					sb.append(line);
				}
				rd.close();
				result = sb.toString();
			} catch (Exception e) {
				e.printStackTrace();
			}
 
		return result;
	}
 
 
 
	public static void main(String args[]) throws Exception{
		String template = "templateDBScan.txt";
//		String template = "templateNOV_QRA.txt";
 
		StringWriter sw = new StringWriter();
		FileReader fr = new FileReader(new File(template));
 
		postData(fr , new URL(dataMinerUrl+"&gcube-token="+token), sw);
 
		fr.close();
 
		String answer = sw.toString();
 
		String statusLocation = answer.substring(answer.indexOf("statusLocation=\"")+"statusLocation=\"".length(), answer.indexOf("\">")); 
 
		System.out.println(sw.toString());
		System.out.println(statusLocation);
 
		String status = getStatus(statusLocation+"&gcube-token="+token);
 
		while (!(status.contains("wps:ProcessSucceeded") || status.contains("wps:ProcessFailed"))){
			System.out.println(status);
			status = getStatus(statusLocation+"&gcube-token="+token);
			Thread.sleep(5000);
		}
 
 
		System.out.println(status);
 
		if (status.contains("wps:ProcessFailed"))
			System.out.println("Process Failed!");
		else{
			String UrlToOutput = status.substring(status.lastIndexOf("<d4science:Data>")+"<d4science:Data>".length(), status.lastIndexOf("</d4science:Data>"));
			System.out.println("Url to output:"+UrlToOutput);
		}
	}
}


Another template file example is available here

WPS Client

A WPS Client is also available to invoke algorithms from R (R 3.3.1). Click this link to get the client and examples.

  • An example of model (BiOnym) invoked via POST and asynchronously with input data table embedding and uploading is available at this link
  • An example of long running model (Ichtyop) invoked via POST and asynchronously is available at this link.
  • An example of model (XMeans) invoked via POST and asynchronously with input data table embedding and uploading is available at this link


The template to invoke a process, copies the process description that can be obtained by building a link like this:

 https://<DataMiner Cluster>/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<your_token>&Identifier=<Process ID>

For example, the BiOnym process description is:

 https://dataminer-cloud1.d4science.org/wps/WebProcessingService?Request=DescribeProcess&Service=WPS&Version=1.0.0&gcube-token=<your_token>&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.generators.BIONYM

Note, that the following information is required in order to reconstruct a description like the one above in the VRE you are using:

1. Your token: you can find this in the home page of the VRE;

2. The DataMiner cluster (can change depending on the VRE): after executing your target process through the Web interface, press "Show". The DataMiner cluster address is the one reported after "https://";

3. The Process ID: in the link contained in the "Show" window, the Process ID is the value of the "Identifier=" parameter.

QGIS

QGIS supports a number of clients for WPS. We advice using the GeoLabs WPS plugin (add the plugin repository at http://geolabs.fr/plugins.xml) and enter the https://dataminer.d4science.org/wps/WebProcessingService endpoint.

QGIS WPS interface
Maps comparison with QGIS

Related Links