DataMiner Algorithms

From Gcube Wiki
Revision as of 10:55, 30 November 2017 by Leonardo.candela (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

The complete list of currently active algorithms freely published by the DataMiner service is reported below.


Aquamaps-related Algorithms

ABSENCE_CELLS_FROM_AQUAMAPS

An algorithm producing cells and features (HCAF) for a species containing absence points taken by an Aquamaps Distribution

AQUAMAPS_NATIVE

Algorithm for Native Range by Aquamaps on a single node

AQUAMAPS_NATIVE_2050

Algorithm for Native Range in 2050 by Aquamaps on a single node

AQUAMAPS_NATIVE_NEURALNETWORK

Aquamaps Native Algorithm calculated by a Neural Network. A distribution algorithm that relies on Neural Networks and AquaMaps data for native distributions to generate a table containing species distribution probabilities on half-degree cells.

AQUAMAPS_SUITABLE

Algorithm by Aquamaps on a single node

AQUAMAPS_SUITABLE_2050

Algorithm for Suitable Range in 2050 by Aquamaps on a single node

AQUAMAPS_SUITABLE_NEURALNETWORK

Aquamaps Algorithm for Suitable Environment calculated by Neural Network. A distribution algorithm that relies on Neural Networks and AquaMaps data for suitable distributions to generate a table containing species distribution probabilities on half-degree cells.

AQUAMAPSNN

The AquaMaps model trained using a Feed Forward Neural Network. This is a method to train a generic Feed Forward Artifical Neural Network to be used by the AquaMaps Neural Network algorithm. Produces a trained neural network in the form of a compiled file which can be used later.

BIOCLIMATE_HCAF

A transducer algorithm that generates an Half-degree Cells Authority File (HCAF) dataset for a certain time frame, with environmental parameters used by the AquaMaps approach. Evaluates the climatic changes impact on the variation of the ocean features contained in HCAF tables

BIOCLIMATE_HSPEC

A transducer algorithm that generates a table containing an estimate of species distributions per half-degree cell (HSPEC) in time. Evaluates the climatic changes impact on species presence.

BIOCLIMATE_HSPEN

A transducer algorithm that generates a table containing species envelops (HSPEN) in time, i.e. models capturing species tolerance with respect to environmental parameters, used by the AquaMaps approach. Evaluates the climatic changes impact on the variation of the salinity values in several ranges of a set of species envelopes

HCAF_FILTER

An algorithm producing a HCAF table on a selected Bounding Box (default identifies Indonesia)

HCAF_INTERPOLATION

Evaluates the climatic changes impact on species presence

HSPEN

The AquMaps HSPEN algorithm. A modeling algorithm that generates a table containing species envelops (HSPEN), i.e. models capturing species tolerance with respect to environmental parameters, to be used by the AquaMaps approach.

PRESENCE_CELLS_GENERATION

An algorithm producing cells and features (HCAF) for a species containing presence points


Data clustering algorithms

DBSCAN

A clustering algorithm for real valued vectors that relies on the density-based spatial clustering of applications with noise (DBSCAN) algorithm. A maximum of 4000 points is allowed.

KMEANS

A clustering algorithm for real valued vectors that relies on the k-means algorithm, i.e. a method aiming to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. A Maximum of 4000 points is allowed.

LOF

Local Outlier Factor (LOF). A clustering algorithm for real valued vectors that relies on Local Outlier Factor algorithm, i.e. an algorithm for finding anomalous data points by measuring the local deviation of a given data point with respect to its neighbours. A Maximum of 4000 points is allowed.

XMEANS

A clustering algorithm for occurrence points that relies on the X-Means algorithm, i.e. an extended version of the K-Means algorithm improved by an Improve-Structure part. A Maximum of 4000 points is allowed.


Geospatial Data Analytics

ESRI_GRID_EXTRACTION

An algorithm to extract values associated to an environmental feature repository (e.g. NETCDF, ASC, GeoTiff files etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial repository ID (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) or a direct link to a file and the specification about time and space. The algorithm produces one ESRI GRID ASCII file containing the values associated to the selected bounding box.

MAPS_COMPARISON

An algorithm for comparing two OGC/NetCDF maps in seamless way to the user. The algorithm assesses the similarities between two geospatial maps by comparing them in a point-to-point fashion. It accepts as input the two geospatial maps (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) and some parameters affecting the comparison such as the z-index, the time index, the comparison threshold. Note: in the case of WFS layers it makes comparisons on the last feature column.

TIMEEXTRACTION

An algorithm to extract a time series of values associated to a geospatial features repository (e.g. NETCDF, ASC, GeoTiff files etc. ). The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram.

TIMEEXTRACTION_TABLE

An algorithm to extract a time series of values associated to a table containing geospatial information. The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram.

XYEXTRACTOR

An algorithm to extract values associated to an environmental feature repository (e.g. NETCDF, ASC, GeoTiff files etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial repository ID (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) or a direct link to a file and the specification about time and space. The algorithm produces one table containing the values associated to the selected bounding box.

XYEXTRACTOR_TABLE

An algorithm to extract values associated to a table containing geospatial features (e.g. Vessel Routes, Species distribution maps etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial table and the specification about time and space. The algorithm produces one table containing the values associated to the selected bounding box.

ZEXTRACTION

An algorithm to extract the Z values from a geospatial features repository (e.g. NETCDF, ASC, GeoTiff files etc. ). The algorithm analyses the repository and automatically extracts the Z values according to the resolution wanted by the user. It produces one chart of the Z values and one table containing the values.

ZEXTRACTION_TABLE

An algorithm to extract a time series of values associated to a table containing geospatial information. The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram.


Geospatial data publishing

RASTER_DATA_PUBLISHER

This algorithm publishes a raster file as a maps or datasets in the e-Infrastructure. NetCDF-CF files are encouraged, as WMS and WCS maps will be produced using this format. For other types of files (GeoTiffs, ASC etc.) only the raw datasets will be published. The resulting map or dataset will be accessible via the VRE GeoExplorer by the VRE participants.

SHAPEFILE_PUBLISHER

An algorithm to publish shapefiles under WMS and WFS standards in the e-Infrastructure. The produced WMS, WFS links are reported as output of this process. The map will be available in the VRE for consultation.


OBIS-related Algorithms

ABSENCE_GENERATION_FROM_OBIS

An algorithm to estimate absence records from survey data in OBIS. Based on the work in Coro, G., Magliozzi, C., Berghe, E. V., Bailly, N., Ellenbroek, A., & Pagano, P. (2016). Estimating absence locations of marine species from data of scientific surveys in OBIS. Ecological Modelling, 323, 61-76.

MOST_OBSERVED_SPECIES

An algorithm producing a bar chart for the most observed species in a certain years range (with respect to the OBIS database)

MOST_OBSERVED_TAXA

An algorithm producing a bar chart for the most observed taxa in a certain years range (with respect to the OBIS database)

SPECIES_OBSERVATION_LME_AREA_PER_YEAR

Algorithm returning most observed species in a specific years range (data collected from OBIS database).

SPECIES_OBSERVATION_MEOW_AREA_PER_YEAR

Algorithm returning most observed species in a specific years range (data collected from OBIS database).

SPECIES_OBSERVATIONS_PER_AREA

An algorithm producing a bar chart for the distribution of a species along a certain type of marine area (e.g. LME or MEOW)

SPECIES_OBSERVATIONS_TREND_PER_YEAR

An algorithm producing the trend of the observations for a certain species in a certain years range.

TAXONOMY_OBSERVATIONS_TREND_PER_YEAR

Algorithm returning most observations taxonomy trend in a specific years range (with respect to the OBIS database)


Occurrence Data Management Algorithms

OCCURRENCES_DUPLICATES_DELETER

A transducer algorithm that produces a duplicate free table of species occurrence points where duplicates have been identified via user defined comparison thresholds. Works with up to 100 000 points

OCCURRENCE_ENRICHMENT

An algorithm performing occurrences enrichment. Takes as input one table containing occurrence points for a set of species and a list of environmental layer, taken either from the e-infrastructure GeoNetwork (through the GeoExplorer application) or from direct HTTP links. Produces one table reporting the set of environmental values associated to the occurrence points.

OCCURRENCES_MERGER

A transducer algorithm that produces a duplicate-free table resulting from the union of two occurrence points tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two Ocurrence Sets, enrichs the Left Set with the elements of the Right Set that are not in the Left Set. Updates the elements of the Left Set with more recent elements in the Right Set. If one element in the Left Set corresponds to several recent elements in the Right Set, these will be all substituted to the element of the Left Set.

OCCURRENCES_INTERSECTOR

A transducer algorithm that produces a table of species occurrence points that are contained in both the two starting tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two occurrence sets, it keeps the elements of the Right Set that are similar to elements in the Left Set.

OCCURRENCES_SUBTRACTION

A transducer algorithm that produces a table resulting from the difference between two occurrence points tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two Ocurrence Sets, keeps the elements of the Left Set that are not similar to any element in the Right Set.

OCCURRENCES_MARINE_TERRESTRIAL

A transducer algorithm that produces a table containing occurrence points by filtering them by type of area, i.e. by recognising whether they are marine or terrestrial. Works with up to 10000 points per table.


RBDMS Inspection Algorithms

LISTDBINFO

Algorithm that allows to view information about one chosen resource of Database Type in the Infrastructure

LISTDBNAMES

Algorithm that allows to view the available database resources names in the Infrastructure

LISTDBSCHEMA

Algorithm that allows to view the schema names of a chosen database for which the type is Postgres

LISTTABLES

Algorithm that allows to view the table names of a chosen database


Other Algorithms

BIONYM

An algorithm implementing BiOnym, a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication.

BIONYM_BIODIV

An algorithm implementing BiOnym oriented to Biodiversity Taxa Names Matching with a predefined and optimized workflow. This version applies in sequence the following Matchers: GSay (thr:0.6, maxRes:10), FuzzyMatcher (thr:0.6, maxRes:10), Levenshtein (thr:0.4, maxRes:10), Trigram (thr:0.4, maxRes:10). BiOnym is a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication.

BIONYM_LOCAL

A fast version of the algorithm implementing BiOnym, a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication.

CMSY_2

The CMSY method for data-limited stock assessment. Described in Froese, R., Demirel, N., Coro, G., Kleisner, K. M., Winker, H. (2016). Estimating fisheries reference points from catch and resilience. Fish and Fisheries. Paper link: http://onlinelibrary.wiley.com/doi/10.1111/faf.12190/ Full Instructions and code: https://github.com/SISTA16/cmsy

CSQUARE_COLUMN_CREATOR

An algorithm that adds a column containing the CSquare codes associated to longitude and latitude columns.

DISCREPANCY_ANALYSIS

An evaluator algorithm that compares two tables containing real valued vectors. It drives the comparison by relying on a geographical distance threshold and a threshold for K-Statistic.

ECOPATH_WITH_ECOSIM

Ecopath with Ecosim (EwE) is a free ecological/ecosystem modeling software suite. This algorithm implementation expects a model and a configuration file as inputs; the result of the analysis is returned as a zip archive. References: Christensen, V., & Walters, C. J. (2004). Ecopath with Ecosim: methods, capabilities and limitations. Ecological modelling, 172(2), 109-139.

ESTIMATE_FISHING_ACTIVITY

An algorithm that estimates activity hours (fishing or other) from vessels trajectories, adds bathymetry information to the table and classifies (point-by-point) fishing activity of the involved vessels according to two algorithms: one based on speed (activity_class_speed output column) and the other based on speed and bathymetry (activity_class_speed_bath output column). The algorithm produces new columns containing this information. This algorithm is based on the paper 'Deriving Fishing Monthly Effort and Caught Species' (Coro et al. 2013, in proc. of OCEANS - Bergen, 2013 MTS/IEEE). Example of input table (NAFO anonymised data): http://goo.gl/3auJkM

ESTIMATE_MONTHLY_FISHING_EFFORT

An algorithm that estimates fishing exploitation at 0.5 degrees resolution from activity-classified vessels trajectories. Produces a table with csquare codes, latitudes, longitudes and resolution and associated overall fishing hours in the time frame of the vessels activity. Requires each activity point to be classified as Fishing or other. This algorithm is based on the paper 'Deriving Fishing Monthly Effort and Caught Species' (Coro et al. 2013, in proc. of OCEANS - Bergen, 2013 MTS/IEEE). Example of input table (NAFO anonymised data): http://goo.gl/3auJkM

FAOMSY

An algorithm to be used by Fisheries managers for stock assessment. Estimates the Maximum Sustainable Yield (MSY) of a stock, based on a catch trend. The algorithm has been developed by the Resource Use and Conservation Division of the FAO Fisheries and Aquaculture Department (contact: Yimin Ye, yimin.ye@fao.org). It is applicable to a CSV file containing metadata and catch statistics for a set of marine species and produces MSY estimates for each species. The CSV must follow a FAO-defined format (e.g. http://goo.gl/g6YtVx). The output is made up of two (optional) files: one for sucessfully processed species and another one for species that could not be processed because data were not sufficient to estimate MSY.

FAO_OCEAN_AREA_COLUMN_CREATOR

An algorithm that adds a column containing the FAO Ocean Area codes associated to longitude and latitude columns.

FAO_OCEAN_AREA_COLUMN_CREATOR_FROM_QUADRANT

An algorithm that adds a column containing the FAO Ocean Area codes associated to longitude, latitude and quadrant columns.

FEED_FORWARD_A_N_N_DISTRIBUTION

A Bayesian method using a Feed Forward Neural Network to simulate a function from the features space (R^n) to R. A modeling algorithm that relies on Neural Networks to simulate a real valued function. It accepts as input a table containing the training dataset and some parameters affecting the algorithm behaviour such as the number of neurons, the learning threshold and the maximum number of iterations.

FEED_FORWARD_ANN

A method to train a generic Feed Forward Artifical Neural Network in order to simulate a function from the features space (R^n) to R. Uses the Back-propagation method. Produces a trained neural network in the form of a compiled file which can be used in the FEED FORWARD NEURAL NETWORK DISTRIBUTION algorithm.

FEED_FORWARD_NEURAL_NETWORK_REGRESSOR

The algorithm simulates a real-valued vector function using a trained Feed Forward Artificial Neural Network and returns a table containing the function actual inputs and the predicted outputs

FEED_FORWARD_NEURAL_NETWORK_TRAINER

The algorithm trains a Feed Forward Artificial Neural Network using an online Back-Propagation procedure and returns the training error and a binary file containing the trained network

FIGIS_SDMX_DATA_CONVERTER

This tool allows to convert easily a SDMX dataset into CSV, by calling the rsdmx package for R

FIGIS_SPATIAL_REALLOCATION_GENERIC

The Spatial Reallocation algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation techniques and provides at now an area-weighted reallocation.

FIGIS_SPATIAL_REALLOCATION_SIMPLIFIED

The Spatial Reallocaton algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation technics and provides at now an area-weighted reallocation. This simplified algorithm is specifically targeting users from the FAO Fisheries and Aquaculture department, aims to facilitate its execution by doing abstraction of the intersections to provide.

FIGIS_SPATIAL_REALLOCATION_SIMPLIFIED_TABLE

The Spatial Reallocaton algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation technics and provides at now an area-weighted reallocation. This simplified algorithm is specifically targeting users from the FAO Fisheries and Aquaculture department, aims to facilitate its execution by doing abstraction of the intersections to provide.

GENERIC_CHARTS

An algorithm producing generic charts of attributes vs. quantities. Charts are displayed per quantity column. Histograms, Scattering and Radar charts are produced for the top ten quantities. A gaussian distribution reports overall statistics for the quantities.

GEO_CHART

An algorithm producing a charts that displays quantities as colors of countries. The color indicates the sum of the values recorded in a country.

GRID_CWP_TO_COORDINATES

An algorithm that adds longitude, latitude and resolution columns analysing a column containing FAO Ocean Area codes (CWP format).

HRS

An evaluator algorithm that calculates the Habitat Representativeness Score, i.e. an indicator of the assessment of whether a specific survey coverage or another environmental features dataset, contains data that are representative of all available habitat variable combinations in an area.

LWR

An algorithm to estimate Length-Weight relationship parameters for marine species, using Bayesian methods. Runs an R procedure. Based on the Cube-law theory.

MAX_ENT_NICHE_MODELLING

A Maximum-Entropy model for species habitat modeling, based on the implementation by Shapire et al. v 3.3.3k, Princeton University, http://www.cs.princeton.edu/schapire/maxent/. In this adaptation for the D4Science infrastructure, the software accepts a table produced by the Species Product Discovery service and a set of environmental layers in various formats (NetCDF, WFS, WCS, ASC, GeoTiff) via direct links or GeoExplorer UUIDs. The user can also establish the bounding box and the spatial resolution (in decimal deg.) of the training and the projection. The application will adapt the layers to that resolution if this is higher than the native one.The output contains: a thumbnail map of the projected model, the ROC curve, the Omission/Commission chart, a table containing the raw assigned values, a threshold to transform the table into a 0-1 probability distribution, a report of the importance of the used layers in the model, ASCII representations of the input layers to check their alignment.Other processes can be later applied to the raw values to produce a GIS map (e.g. the Statistical Manager Points-to-Map process) and results can be shared. Demo video: http://goo.gl/TYYnTO and instructions http://wiki.i-marine.eu/index.php/MaxEnt

POINTS_TO_MAP

A transducer algorithm to produce a GIS map of points from a set of points with x,y coordinates indications. A maximum of 259000 is allowed

POLYGONS_TO_MAP

A transducer algorithm to produce a GIS map of filled polygons associated to x,y coordinates and a certain resolution. A maximum of 259000 is allowed

QUALITY_ANALYSIS

An evaluator algorithm that assesses the effectiveness of a distribution model by computing the Receiver Operating Characteristics (ROC), the Area Under Curve (AUC) and the Accuracy of a model

SEADATANET_INTERPOLATOR

A connector for the SeaDataNet infrastructure. This algorithms invokes the Data-Interpolating Variational Analysis (DIVA) SeaDataNet service to interpolate spatial data. The model uses GEBCO bathymetry data and requires an estimate of the maximum spatial span of the correlation between points and the signal-to-noise ratio, among the other parameters. It can interpolate up to 10,000 points randomly taken from the input table. As output, it produces a NetCDF file with a uniform grid of values. This powerful interpolation model is described in Troupin et al. 2012, 'Generation of analysis and consistent error fields using the Data Interpolating Variational Analysis (Diva)', Ocean Modelling, 52-53, 90-101.

SGVM_INTERPOLATION

An interpolation method relying on the implementation by the Study Group on VMS (SGVMS). The method uses two interpolation approached to simulate vessels points at a certain temporal resolution. The input is a file in TACSAT format uploaded on the Statistical Manager. The output is another TACSAT file containing interpolated points.The underlying R code has been extracted from the SGVM VMSTools framework. This algorithm comes after a feasibility study (http://goo.gl/risQre) which clarifies the features an e-Infrastructure adds to the original scripts. Limitation: the input will be processed up to 10000 vessels trajectory points. Credits: Hintzen, N. T., Bastardie, F., Beare, D., Piet, G. J., Ulrich, C., Deporte, N., Egekvist, J., et al. 2012. VMStools: Open-source software for the processing, analysis and visualisation of fisheries logbook and VMS data. Fisheries Research, 115-116: 31-43. Hintzen, N. T., Piet, G. J., and Brunel, T. 2010. Improved estimation of trawling tracks using cubic Hermite spline interpolation of position registration data. Fisheries Research, 101: 108-115. VMStools, available as an add-on package for R. Documentation available at https://code.google.com/p/vmstools/. Build versions of VMStools for Window, Mac, Linux available at https://docs.google.com/. Authors: Niels T. Hintzen, Doug Beare

SPECIES_MAP_FROM_CSQUARES

A transducer algorithm to produce a GIS map from a probability distribution associated to a set of csquare codes. A maximum of 259000 is allowed

SPECIES_MAP_FROM_POINTS

A transducer algorithm to produce a GIS map from a probability distribution made of x,y coordinates and a certain resolution. A maximum of 259000 is allowed

SUPPORT_VECTOR_MACHINE_REGRESSOR

A simple algorithm for regression using and already trained Support Vector Machine

SUPPORT_VECTOR_MACHINE_TRAINER

A simple algorithm to train a Support Vector Machine

TIME_GEO_CHART

An algorithm producing an animated gif displaying quantities as colors in time. The color indicates the sum of the values recorded in a country.

TIME_SERIES_ANALYSIS

An algorithms applying signal processing to a non uniform time series. A maximum of 10000 distinct points in time is allowed to be processed. The process uniformly samples the series, then extracts hidden periodicities and signal properties. The sampling period is the shortest time difference between two points. Finally, by using Caterpillar-SSA the algorithm forecasts the Time Series. The output shows the detected periodicity, the forecasted signal and the spectrogram.

TIME_SERIES_CHARTS

An algorithm producing time series charts of attributes vs. quantities. Charts are displayed per quantity column and superposing quantities are summed.

WEB_APP_PUBLISHER

This algorithm publishes a zip file containing a Web site, based on html and javascript in the e-Infrastructure. It generates a public URL to the application that can be shared.