Difference between revisions of "GCube Data Catalogue for GRSF"

From Gcube Wiki
Jump to: navigation, search
(Common Metadata)
(GCube Data Catalogue: support for GRSF)
Line 4: Line 4:
 
= GCube Data Catalogue: support for GRSF =
 
= GCube Data Catalogue: support for GRSF =
  
In the following page are reported the relevant information about the GRSF Data Catalogue, which is available [https://i-marine.d4science.org/group/grsf/data-catalogue here]. This page is somehow an extension of the main [https://wiki.gcube-system.org/gcube/GCube_Data_Catalogue gCube Data Catalogue] guide, you are suggested to read before continue.  
+
In this page are reported the relevant information about the GRSF Data Catalogue, which is available [https://i-marine.d4science.org/group/grsf/data-catalogue here]. This page is somehow an extension of the main [https://wiki.gcube-system.org/gcube/GCube_Data_Catalogue gCube Data Catalogue] guide, you are suggested to read before continue.  
  
The GRFS Data Catalogue stores, as well as allows the publication of products of two types: Stock and Fishery. A part from the default set of metadata, each type of product will also have specific fields. Some of them will also become automatically tags of the product. The same reasoning applies for group associations. In fact a set of groups was already available and the product will be automatically associated to them during publication, if that is the case. Fields that fire tags creation or groups association will be reported in the following as well.
+
The GRFS Data Catalogue stores, as well as allows the publication of products of two types: ''Stock'' and ''Fishery''.  
 +
Apart from the default set of metadata, each type of product will also have specific fields. Some of them will also become automatically tags of the product. The same reasoning applies for group associations. In fact a set of groups was already available and each product will be automatically associated to them during publication, if that is the case. Fields that fire tags creation or groups association are documented below.
  
 
The publication phase is performed by means of a RESTful service whose publish methods accept JSON objects.
 
The publication phase is performed by means of a RESTful service whose publish methods accept JSON objects.
Line 13: Line 14:
 
=== Common Metadata ===
 
=== Common Metadata ===
  
The following table shows the set of ''core'' metadata, that is the ones shared by both Stock and Fishery types. Some of them are retrieved during publication and will be filled automatically so that you don't need to worry about. The values given to some fields are automatically used to tag the product. Check the Is Tag column of the table below. Other fields have a controlled vocabulary (that is, they can assume a limited set of values), and the value assigned to these field allow to automatically determine to which group assign the product. Check the Is Group column below.  
+
The following table shows the set of ''core'' metadata, that is the ones shared by both Stock and Fishery types. Some of them are automatically filled. The values given to some fields are automatically used to tag the product. Check the 'Is Tag' column of the table below. Other fields have a controlled vocabulary (that is, they can assume values selected from a defined set), and the value assigned to these fields allow to automatically determine to which group assign the product. Check the 'Is Group' column below.  
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 25: Line 26:
 
| Description
 
| Description
 
| description
 
| description
|  
+
| No
|  
+
| No
 
| This product contains attributes of ...
 
| This product contains attributes of ...
| Description of a few sentences, written in plain language. Should provide a sufficiently comprehensive overview of the resource for anyone to understand its content origins and any continuing work on it.
+
| A brief description of the dataset written in plain language. It should provide a sufficiently comprehensive overview of the resource for anyone to understand its content origins and any continuing work on it.
 
|-
 
|-
 
| License *
 
| License *
 
| license_id
 
| license_id
|  
+
| No
|  
+
| No
 
| CC-BY-SA-4.0
 
| CC-BY-SA-4.0
 
| style="font-style: italic;" | The list of licenses' ids can be retrieved by using the service (see below). By default the CC-BY-SA-4.0 will be used.
 
| style="font-style: italic;" | The list of licenses' ids can be retrieved by using the service (see below). By default the CC-BY-SA-4.0 will be used.
Line 39: Line 40:
 
| Author
 
| Author
 
| author
 
| author
|  
+
| No
|  
+
| No
 
| Bloggs, Joe
 
| Bloggs, Joe
| style="font-style: italic;" | This field is automatically evaluated. You do not need to pass it.
+
| style="font-style: italic;" | This field is automatically compiled by using the information of the caller entity.  
 
|-
 
|-
 
| Author contact
 
| Author contact
 
| author_contact
 
| author_contact
|  
+
| No
|  
+
| No
 
| joe.blogg@example.com
 
| joe.blogg@example.com
| style="font-style: italic;" | This field is automatically evaluated. You do not need to pass it.
+
| style="font-style: italic;" | This field is automatically compiled by using the information of the caller entity
 
|-
 
|-
 
| Maintainer
 
| Maintainer
 
| maintainer
 
| maintainer
|  
+
| No
|  
+
| No
 
| A person: Bloggs, Joe. An authority: D4Science
 
| A person: Bloggs, Joe. An authority: D4Science
 
| Mantainer of the dataset
 
| Mantainer of the dataset
Line 60: Line 61:
 
| Maintainer Contact
 
| Maintainer Contact
 
| maintainer_contact
 
| maintainer_contact
|  
+
| No
|  
+
| No
 
| joe@example.com
 
| joe@example.com
| Contact details of mantainer
+
| Contact details of the resource maintainer.
 
|-
 
|-
 
| Version
 
| Version
 
| version
 
| version
|  
+
| No
|  
+
| No
 
| 1.0
 
| 1.0
 
| Increase manually after editing
 
| Increase manually after editing
Line 74: Line 75:
 
| Catches or landings
 
| Catches or landings
 
| catches_or_landings
 
| catches_or_landings
|  
+
| No
|  
+
| No
 
| Catch - 18962 - ton - 2014
 
| Catch - 18962 - ton - 2014
 
| A combination of value, unit and date
 
| A combination of value, unit and date
Line 81: Line 82:
 
| Database Sources *
 
| Database Sources *
 
| database_sources
 
| database_sources
|  
+
| No
 
| Yes
 
| Yes
 
| {"name":"FIRMS", "description": "unknown", "url":"http://....."}
 
| {"name":"FIRMS", "description": "unknown", "url":"http://....."}
Line 89: Line 90:
 
| Source of Information *
 
| Source of Information *
 
| source_of_information
 
| source_of_information
|  
+
| No
|  
+
| No
 
| {"name":"...", "description": "...", "url":"http://....."}
 
| {"name":"...", "description": "...", "url":"http://....."}
 
| A list of elements of the type {"name": "a name", "description": "a description", "url": "http://...."}. Name and url are mandatory.
 
| A list of elements of the type {"name": "a name", "description": "a description", "url": "http://...."}. Name and url are mandatory.
Line 96: Line 97:
 
| Data owner  
 
| Data owner  
 
| data_owner
 
| data_owner
|  
+
| No
|  
+
| No
 
| IATTC
 
| IATTC
 
|  
 
|  
Line 117: Line 118:
 
| Stock Name *
 
| Stock Name *
 
| stock_name
 
| stock_name
|  
+
| No
|  
+
| No
 
| Skipjack tuna - Eastern Pacific
 
| Skipjack tuna - Eastern Pacific
| This will be the title of the product and a unique name will be generated starting from this.
+
| The title of the product. It is expected to be a unique name.
 
|-
 
|-
 
| Stock ID
 
| Stock ID
 
| stock_id
 
| stock_id
|  
+
| No
|  
+
| No
 
| SKJ- EPO
 
| SKJ- EPO
 
|  
 
|  
Line 139: Line 140:
 
| species_scientific_name
 
| species_scientific_name
 
| Yes
 
| Yes
|  
+
| No
 
| Katsuwonus pelamis (or SKJ)
 
| Katsuwonus pelamis (or SKJ)
 
|  
 
|  
Line 145: Line 146:
 
| Assessment distribution area *
 
| Assessment distribution area *
 
| assessment_distribution_area
 
| assessment_distribution_area
|  
+
| No
|  
+
| No
 
| East Pacific Ocean
 
| East Pacific Ocean
 
|  
 
|  
Line 152: Line 153:
 
| Exploiting Fishery
 
| Exploiting Fishery
 
| exploiting_fishery
 
| exploiting_fishery
|  
+
| No
|  
+
| No
 
| Tunas and billfishes fishery
 
| Tunas and billfishes fishery
 
|  
 
|  
Line 159: Line 160:
 
| Management entity
 
| Management entity
 
| management_entity
 
| management_entity
|  
+
| No
|  
+
| No
 
| DFO
 
| DFO
 
|  
 
|  
Line 166: Line 167:
 
| Assessment methods
 
| Assessment methods
 
| assessment_methods
 
| assessment_methods
|  
+
| No
|  
+
| No
 
| Analytical assessment
 
| Analytical assessment
 
|  
 
|  
Line 173: Line 174:
 
| State of marine resources
 
| State of marine resources
 
| state_of_marine_resource
 
| state_of_marine_resource
|  
+
| No
|  
+
| No
 
|  
 
|  
 
|  
 
|  
Line 180: Line 181:
 
| Exploitation Rate
 
| Exploitation Rate
 
| exploitation_rate
 
| exploitation_rate
|  
+
| No
 
| Yes
 
| Yes
 
| Moderate fishing mortality
 
| Moderate fishing mortality
Line 187: Line 188:
 
| Abundance level
 
| Abundance level
 
| abundance_level
 
| abundance_level
|  
+
| No
 
| Yes
 
| Yes
 
| Intermediate abundance
 
| Intermediate abundance
Line 194: Line 195:
 
| Narrative state and trend
 
| Narrative state and trend
 
| narrative_state_and_trend
 
| narrative_state_and_trend
|  
+
| No
|  
+
| No
 
| Stock size and fishing pressure are considered to be close to their value at MSY.
 
| Stock size and fishing pressure are considered to be close to their value at MSY.
 
| A textual description
 
| A textual description
Line 201: Line 202:
 
| Scientific advice
 
| Scientific advice
 
| scientific_advice
 
| scientific_advice
|  
+
| No
|  
+
| No
 
| The indices of abundance from two longline fleets available for this stock present divergent trends over the last few years, the differences observed in targeting are not fully explained.
 
| The indices of abundance from two longline fleets available for this stock present divergent trends over the last few years, the differences observed in targeting are not fully explained.
 
| A textual description
 
| A textual description
Line 208: Line 209:
 
| Reporting entity
 
| Reporting entity
 
| reporting_entity
 
| reporting_entity
|  
+
| No
|  
+
| No
 
| GRP3
 
| GRP3
 
|  
 
|  
Line 215: Line 216:
 
| Reporting year
 
| Reporting year
 
| reporting_year
 
| reporting_year
|  
+
| No
|  
+
| No
 
| 2005
 
| 2005
 
|  
 
|  
Line 223: Line 224:
 
| status
 
| status
 
| Yes
 
| Yes
|  
+
| No
 
| Pending
 
| Pending
 
| style="font-style: italic;" | Controlled vocabulary: Pending, Confirmed.  
 
| style="font-style: italic;" | Controlled vocabulary: Pending, Confirmed.  
Line 244: Line 245:
 
| Fishery Name *
 
| Fishery Name *
 
| fishery_name
 
| fishery_name
|  
+
| No
|  
+
| No
 
| NAFO Flemish Cap groundfish fisheries
 
| NAFO Flemish Cap groundfish fisheries
 
| This will be the title of the product and a unique name will be generated starting from this.
 
| This will be the title of the product and a unique name will be generated starting from this.
Line 251: Line 252:
 
| Fishery ID
 
| Fishery ID
 
| fishery_id
 
| fishery_id
|  
+
| No
|  
+
| No
 
| COD - 21.3.M - NAFO - OTB - CAN - Industrial
 
| COD - 21.3.M - NAFO - OTB - CAN - Industrial
 
|  
 
|  
Line 266: Line 267:
 
| scientific_name
 
| scientific_name
 
| Yes
 
| Yes
|  
+
| No
 
| Caribbean spiny lobster
 
| Caribbean spiny lobster
 
|  
 
|  
Line 272: Line 273:
 
| Fishing area
 
| Fishing area
 
| fishing_area
 
| fishing_area
|  
+
| No
|  
+
| No
 
| North Atlantic
 
| North Atlantic
 
| '''If missing then Jurisdiction Area cannot be null'''
 
| '''If missing then Jurisdiction Area cannot be null'''
Line 279: Line 280:
 
| Exploited stocks
 
| Exploited stocks
 
| exploited_stocks
 
| exploited_stocks
|  
+
| No
|  
+
| No
 
| Capelin - Southern Grand Bank
 
| Capelin - Southern Grand Bank
 
|  
 
|  
Line 287: Line 288:
 
| management_entity
 
| management_entity
 
| Yes
 
| Yes
|  
+
| No
 
| European Union
 
| European Union
 
|  
 
|  
Line 293: Line 294:
 
| Jurisdiction Area  
 
| Jurisdiction Area  
 
| jurisdiction_area
 
| jurisdiction_area
|  
+
| No
|  
+
| No
 
| Senegal
 
| Senegal
 
| '''If missing then Fishing Area cannot be null'''
 
| '''If missing then Fishing Area cannot be null'''
Line 300: Line 301:
 
| Production system type
 
| Production system type
 
| production_system_type
 
| production_system_type
|  
+
| No
 
| Yes
 
| Yes
 
| Industrial
 
| Industrial
Line 315: Line 316:
 
| fishing_gear
 
| fishing_gear
 
| Yes
 
| Yes
|  
+
| No
 
| PUN
 
| PUN
 
| style="font-style: italic;" |  
 
| style="font-style: italic;" |  
Line 321: Line 322:
 
| Environment
 
| Environment
 
| environment
 
| environment
|  
+
| No
|  
+
| No
 
|  
 
|  
 
|  
 
|  
Line 328: Line 329:
 
| Status *
 
| Status *
 
| status
 
| status
|  
+
| No
 
| Yes
 
| Yes
 
| Pending
 
| Pending

Revision as of 09:08, 13 October 2016

** THIS PAGE IS UNDER CONSTRUCTION **

GCube Data Catalogue: support for GRSF

In this page are reported the relevant information about the GRSF Data Catalogue, which is available here. This page is somehow an extension of the main gCube Data Catalogue guide, you are suggested to read before continue.

The GRFS Data Catalogue stores, as well as allows the publication of products of two types: Stock and Fishery. Apart from the default set of metadata, each type of product will also have specific fields. Some of them will also become automatically tags of the product. The same reasoning applies for group associations. In fact a set of groups was already available and each product will be automatically associated to them during publication, if that is the case. Fields that fire tags creation or groups association are documented below.

The publication phase is performed by means of a RESTful service whose publish methods accept JSON objects.

Metadata

Common Metadata

The following table shows the set of core metadata, that is the ones shared by both Stock and Fishery types. Some of them are automatically filled. The values given to some fields are automatically used to tag the product. Check the 'Is Tag' column of the table below. Other fields have a controlled vocabulary (that is, they can assume values selected from a defined set), and the value assigned to these fields allow to automatically determine to which group assign the product. Check the 'Is Group' column below.

Name Api Name (JSON) Is Tag Is Group Example Guidelines/Comments
Description description No No This product contains attributes of ... A brief description of the dataset written in plain language. It should provide a sufficiently comprehensive overview of the resource for anyone to understand its content origins and any continuing work on it.
License * license_id No No CC-BY-SA-4.0 The list of licenses' ids can be retrieved by using the service (see below). By default the CC-BY-SA-4.0 will be used.
Author author No No Bloggs, Joe This field is automatically compiled by using the information of the caller entity.
Author contact author_contact No No joe.blogg@example.com This field is automatically compiled by using the information of the caller entity
Maintainer maintainer No No A person: Bloggs, Joe. An authority: D4Science Mantainer of the dataset
Maintainer Contact maintainer_contact No No joe@example.com Contact details of the resource maintainer.
Version version No No 1.0 Increase manually after editing
Catches or landings catches_or_landings No No Catch - 18962 - ton - 2014 A combination of value, unit and date
Database Sources * database_sources No Yes {"name":"FIRMS", "description": "unknown", "url":"http://....."} A list of elements of the type {"name": "a name", "description": "a description", "url": "http://...."}. Name and url are mandatory.

For the attribute name there is a controlled vocabulary: FIRMS, RAM, FishSource

Source of Information * source_of_information No No {"name":"...", "description": "...", "url":"http://....."} A list of elements of the type {"name": "a name", "description": "a description", "url": "http://...."}. Name and url are mandatory.
Data owner data_owner No No IATTC

mandatory fields are marked with an asterisk (*)

Stock Metadata

The Stock product type also supports the following list of fields.

Name API Name Is Tag Is Group Example Guidelines/Comments
Stock Name * stock_name No No Skipjack tuna - Eastern Pacific The title of the product. It is expected to be a unique name.
Stock ID stock_id No No SKJ- EPO
Type type Yes Yes Assessment Unit Controlled vocabulary: Assessment Unit, Resource
Species Scientific Name * species_scientific_name Yes No Katsuwonus pelamis (or SKJ)
Assessment distribution area * assessment_distribution_area No No East Pacific Ocean
Exploiting Fishery exploiting_fishery No No Tunas and billfishes fishery
Management entity management_entity No No DFO
Assessment methods assessment_methods No No Analytical assessment
State of marine resources state_of_marine_resource No No
Exploitation Rate exploitation_rate No Yes Moderate fishing mortality Controlled vocabulary: Moderate fishing mortality, High fishing mortality, No or low fishing mortality
Abundance level abundance_level No Yes Intermediate abundance Controlled vocabulary: Intermediate abundance, Low abundance, Uncertain/Not assessed
Narrative state and trend narrative_state_and_trend No No Stock size and fishing pressure are considered to be close to their value at MSY. A textual description
Scientific advice scientific_advice No No The indices of abundance from two longline fleets available for this stock present divergent trends over the last few years, the differences observed in targeting are not fully explained. A textual description
Reporting entity reporting_entity No No GRP3
Reporting year reporting_year No No 2005
Status * status Yes No Pending Controlled vocabulary: Pending, Confirmed.

mandatory fields are marked with an asterisk (*)

Fishery Metadata

The Fishery product type also supports the following list of fields.

Name API Name Is Tag Is Group Example Guidelines/Comments
Fishery Name * fishery_name No No NAFO Flemish Cap groundfish fisheries This will be the title of the product and a unique name will be generated starting from this.
Fishery ID fishery_id No No COD - 21.3.M - NAFO - OTB - CAN - Industrial
Type type Yes Yes Fishery Activity Controlled vocabulary: Fishery Activity, Fishing Description
Scientific Name scientific_name Yes No Caribbean spiny lobster
Fishing area fishing_area No No North Atlantic If missing then Jurisdiction Area cannot be null
Exploited stocks exploited_stocks No No Capelin - Southern Grand Bank
Management entity management_entity Yes No European Union
Jurisdiction Area jurisdiction_area No No Senegal If missing then Fishing Area cannot be null
Production system type production_system_type No Yes Industrial Controlled vocabulary: Subsistence, Recreational, Commercial, Artisanal, Semi-industrial, Industrial, Exploratory_fishery, Unspecified
Flag state flag_state Yes ESP
Fishing gear fishing_gear Yes No PUN
Environment environment No No
Status * status No Yes Pending Controlled vocabulary: Pending, Confirmed.

mandatory fields are marked with an asterisk (*)

GRSF Publication Web Service

Publication of products is performed by means of a RESTful web service. Each call to the service requires the security token of the user for the context in which he wish to publish or exploit the other functionalities. Please note that in case of product publication it is needed that the user has enough privileges. The list of roles and associated privileges for the catalogue users is reported here. The VRE Manager assignes them.

Check Service Availability

Retrieve the licenses list

Stock Publication Example

Fishery Publication Example

How To Publish a GRSF product using JAVA

Below you find a simple Java client to publish a GRSF product (i.e. a stock).

package [YOUR PACKAGE];

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
import org.apache.commons.httpclient.methods.ByteArrayRequestEntity;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.http.HttpStatus;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * The Class GRSFPublishMetataTest.
 *
 * @author Francesco Mangiacrapa francesco.mangiacrapa@isti.cnr.it
 * Oct 12, 2016
 */
public class GRSFPublishMetataTest {

	public static final Logger logger = LoggerFactory.getLogger(GRSFPublishMetataTest.class);
	private static final String GRSF_PUBLISHER_REST_SERVICE_URL = "https://pre.d4science.org/grsf-publisher-ws/rest";
	private static enum PRODUCT_TYPE{stock, fishery}
	private static final String PUBLISH_PRODUCT_REQUEST = "publish-product";
	private static final String GCUBE_TOKEN_PARAMETER = "gcube-token";
	private static final String GCUBE_TOKEN_VALUE = "XYZ"; //***********SET YOUR TOKEN************
	private static final String CONTENTTYPE = "application/json";
	private HttpClient httpClient = null;
	public static final int TIME_OUT_REQUESTS = 5000; //5 sec

	/**
	 * Instantiates a new GRSF publish metata test.
	 */
	public GRSFPublishMetataTest() {
		MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
		connectionManager.getParams().setSoTimeout(TIME_OUT_REQUESTS);
		this.httpClient = new HttpClient(connectionManager);

	}


	/**
	 * Publish product.
	 *
	 * @param type the type
	 * @param body the body
	 * @return the string
	 * @throws Exception the exception
	 */
	public String publishProduct(PRODUCT_TYPE type, String body) throws Exception {
		// Create a method instance.
		String buildURL = GRSF_PUBLISHER_REST_SERVICE_URL + "/" + type.toString() +"/"+PUBLISH_PRODUCT_REQUEST +"?"+GCUBE_TOKEN_PARAMETER +"="+GCUBE_TOKEN_VALUE;
		PostMethod method = new PostMethod(buildURL);
		method.setRequestHeader("Content-type", CONTENTTYPE);
		logger.debug("call post to URI .... " + method.getURI());
		logger.debug("	the body is..." + body);
		method.setRequestEntity(new ByteArrayRequestEntity(body.getBytes()));
		byte[] responseBody = null;
		try {
			// Execute the method.
			int statusCode = httpClient.executeMethod(method);

			if (statusCode != HttpStatus.SC_OK && statusCode != HttpStatus.SC_CREATED) {
				logger.error("Method failed: " + method.getStatusLine()+"; Response bpdy: "+method.getResponseBody());
				method.releaseConnection();
				throw new Exception("Method failed: " + method.getStatusLine()+"; Response body: "+new String(method.getResponseBody()));
			}
			// Read the response body.
			responseBody = method.getResponseBody();

		} catch (HttpException e) {
			logger.error("Fatal protocol violation: ", e);
			method.releaseConnection();
			throw new Exception("Fatal protocol violation: " + e.getMessage());
		} catch (Exception e) {
			logger.error("Fatal transport error: ", e);
			method.releaseConnection();
			throw new Exception("Fatal transport error: " + e.getMessage());
		}
		method.releaseConnection();
		return new String(responseBody);
	}

	/**
	 * The main method.
	 *
	 * @param args the arguments
	 */
	public static void main(String[] args) {
		try {
			GRSFPublishMetataTest grsfP = new GRSFPublishMetataTest();
			String minimal_json_stock = "YOUR STOCK"; //***********SET YOUR STOCK************
			String response = grsfP.publishProduct(PRODUCT_TYPE.stock, minimal_json_stock);
			logger.info("The Response: "+response);
		}catch (Exception e) {
			e.printStackTrace();
		}
	}

}