# DataMiner Algorithms

From Gcube Wiki

Revision as of 12:06, 9 November 2017 by Gianpaolo.coro (Talk | contribs)

The complete list of currently active algorithms freely published by the DataMiner service is reported below.

HCAF_INTERPOLATION
| |
---|---|

Description | Evaluates the climatic changes impact on species presence |

FEED_FORWARD_A_N_N_DISTRIBUTION
| |

Description | A Bayesian method using a Feed Forward Neural Network to simulate a function from the features space (R^n) to R. A modeling algorithm that relies on Neural Networks to simulate a real valued function. It accepts as input a table containing the training dataset and some parameters affecting the algorithm behaviour such as the number of neurons, the learning threshold and the maximum number of iterations. |

XYEXTRACTOR
| |

Description | An algorithm to extract values associated to an environmental feature repository (e.g. NETCDF, ASC, GeoTiff files etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial repository ID (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) or a direct link to a file and the specification about time and space. The algorithm produces one table containing the values associated to the selected bounding box. |

BIOCLIMATE_HSPEN
| |

Description | A transducer algorithm that generates a table containing species envelops (HSPEN) in time, i.e. models capturing species tolerance with respect to environmental parameters, used by the AquaMaps approach. Evaluates the climatic changes impact on the variation of the salinity values in several ranges of a set of species envelopes |

HSPEN
| |

Description | The AquMaps HSPEN algorithm. A modeling algorithm that generates a table containing species envelops (HSPEN), i.e. models capturing species tolerance with respect to environmental parameters, to be used by the AquaMaps approach. |

SUPPORT_VECTOR_MACHINE_REGRESSOR
| |

Description | A simple algorithm for regression using and already trained Support Vector Machine |

AQUAMAPS_NATIVE
| |

Description | Algorithm for Native Range by Aquamaps on a single node |

POLYGONS_TO_MAP
| |

Description | A transducer algorithm to produce a GIS map of filled polygons associated to x,y coordinates and a certain resolution. A maximum of 259000 is allowed |

LISTDBNAMES
| |

Description | Algorithm that allows to view the available database resources names in the Infrastructure |

BIOCLIMATE_HSPEC
| |

Description | A transducer algorithm that generates a table containing an estimate of species distributions per half-degree cell (HSPEC) in time. Evaluates the climatic changes impact on species presence. |

LOF
| |

Description | Local Outlier Factor (LOF). A clustering algorithm for real valued vectors that relies on Local Outlier Factor algorithm, i.e. an algorithm for finding anomalous data points by measuring the local deviation of a given data point with respect to its neighbours. A Maximum of 4000 points is allowed. |

TIME_GEO_CHART
| |

Description | An algorithm producing an animated gif displaying quantities as colors in time. The color indicates the sum of the values recorded in a country. |

SPECIES_OBSERVATIONS_PER_AREA
| |

Description | An algorithm producing a bar chart for the distribution of a species along a certain type of marine area (e.g. LME or MEOW) |

LWR
| |

Description | An algorithm to estimate Length-Weight relationship parameters for marine species, using Bayesian methods. Runs an R procedure. Based on the Cube-law theory. |

DBSCAN
| |

Description | A clustering algorithm for real valued vectors that relies on the density-based spatial clustering of applications with noise (DBSCAN) algorithm. A maximum of 4000 points is allowed. |

BIONYM
| |

Description | An algorithm implementing BiOnym, a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication. |

DISCREPANCY_ANALYSIS
| |

Description | An evaluator algorithm that compares two tables containing real valued vectors. It drives the comparison by relying on a geographical distance threshold and a threshold for K-Statistic. |

AQUAMAPS_SUITABLE
| |

Description | Algorithm by Aquamaps on a single node |

AQUAMAPS_NATIVE_NEURALNETWORK
| |

Description | Aquamaps Native Algorithm calculated by a Neural Network. A distribution algorithm that relies on Neural Networks and AquaMaps data for native distributions to generate a table containing species distribution probabilities on half-degree cells. |

ECOPATH_WITH_ECOSIM
| |

Description | Ecopath with Ecosim (EwE) is a free ecological/ecosystem modeling software suite. This algorithm implementation expects a model and a configuration file as inputs; the result of the analysis is returned as a zip archive. References: Christensen, V., & Walters, C. J. (2004). Ecopath with Ecosim: methods, capabilities and limitations. Ecological modelling, 172(2), 109-139. |

OCCURRENCES_INTERSECTOR
| |

Description | A transducer algorithm that produces a table of species occurrence points that are contained in both the two starting tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two ocurrence sets, it keeps the elements of the Right Set that are similar to elements in the Left Set. |

BIONYM_LOCAL
| |

Description | A fast version of the algorithm implementing BiOnym, a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication. |

TIME_SERIES_ANALYSIS
| |

Description | An algorithms applying signal processing to a non uniform time series. A maximum of 10000 distinct points in time is allowed to be processed. The process uniformly samples the series, then extracts hidden periodicities and signal properties. The sampling period is the shortest time difference between two points. Finally, by using Caterpillar-SSA the algorithm forecasts the Time Series. The output shows the detected periodicity, the forecasted signal and the spectrogram. |

OCCURRENCE_ENRICHMENT
| |

Description | An algorithm performing occurrences enrichment. Takes as input one table containing occurrence points for a set of species and a list of environmental layer, taken either from the e-infrastructure GeoNetwork (through the GeoExplorer application) or from direct HTTP links. Produces one table reporting the set of environmental values associated to the occurrence points. |

FAO_OCEAN_AREA_COLUMN_CREATOR_FROM_QUADRANT
| |

Description | An algorithm that adds a column containing the FAO Ocean Area codes associated to longitude, latitude and quadrant columns. |

POINTS_TO_MAP
| |

Description | A transducer algorithm to produce a GIS map of points from a set of points with x,y coordinates indications. A maximum of 259000 is allowed |

BIONYM_BIODIV
| |

Description | An algorithm implementing BiOnym oriented to Biodiversity Taxa Names Matching with a predefined and optimized workflow. This version applies in sequence the following Matchers: GSay (thr:0.6, maxRes:10), FuzzyMatcher (thr:0.6, maxRes:10), Levenshtein (thr:0.4, maxRes:10), Trigram (thr:0.4, maxRes:10). BiOnym is a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication. |

XMEANS
| |

Description | A clustering algorithm for occurrence points that relies on the X-Means algorithm, i.e. an extended version of the K-Means algorithm improved by an Improve-Structure part. A Maximum of 4000 points is allowed. |

QUALITY_ANALYSIS
| |

Description | An evaluator algorithm that assesses the effectiveness of a distribution model by computing the Receiver Operating Characteristics (ROC), the Area Under Curve (AUC) and the Accuracy of a model |

TIME_SERIES_CHARTS
| |

Description | An algorithm producing time series charts of attributes vs. quantities. Charts are displayed per quantity column and superposing quantities are summed. |

ABSENCE_CELLS_FROM_AQUAMAPS
| |

Description | An algorithm producing cells and features (HCAF) for a species containing absense points taken by an Aquamaps Distribution |

AQUAMAPS_SUITABLE_NEURALNETWORK
| |

Description | Aquamaps Algorithm for Suitable Environment calculated by Neural Network. A distribution algorithm that relies on Neural Networks and AquaMaps data for suitable distributions to generate a table containing species distribution probabilities on half-degree cells. |

SPECIES_MAP_FROM_POINTS
| |

Description | A transducer algorithm to produce a GIS map from a probability distribution made upf of x,y coordinates and a certain resolution. A maximum of 259000 is allowed |

MOST_OBSERVED_TAXA
| |

Description | An algorithm producing a bar chart for the most observed taxa in a certain years range (with respect to the OBIS database) |

LISTTABLES
| |

Description | Algorithm that allows to view the table names of a chosen database |

CSQUARE_COLUMN_CREATOR
| |

Description | An algorithm that adds a column containing the CSquare codes associated to longitude and latitude columns. |

FAO_OCEAN_AREA_COLUMN_CREATOR
| |

Description | An algorithm that adds a column containing the FAO Ocean Area codes associated to longitude and latitude columns. |

FEED_FORWARD_ANN
| |

Description | A method to train a generic Feed Forward Artifical Neural Network in order to simulate a function from the features space (R^n) to R. Uses the Back-propagation method. Produces a trained neural network in the form of a compiled file which can be used in the FEED FORWARD NEURAL NETWORK DISTRIBUTION algorithm. |

SPECIES_MAP_FROM_CSQUARES
| |

Description | A transducer algorithm to produce a GIS map from a probability distribution associated to a set of csquare codes. A maximum of 259000 is allowed |

HRS
| |

Description | An evaluator algorithm that calculates the Habitat Representativeness Score, i.e. an indicator of the assessment of whether a specific survey coverage or another environmental features dataset, contains data that are representative of all available habitat variable combinations in an area. |

ESRI_GRID_EXTRACTION
| |

Description | An algorithm to extract values associated to an environmental feature repository (e.g. NETCDF, ASC, GeoTiff files etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial repository ID (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) or a direct link to a file and the specification about time and space. The algorithm produces one ESRI GRID ASCII file containing the values associated to the selected bounding box. |

SPECIES_OBSERVATION_MEOW_AREA_PER_YEAR
| |

Description | Algorithm returning most observed species in a specific years range (data collected from OBIS database). |

SPECIES_OBSERVATION_LME_AREA_PER_YEAR
| |

Description | Algorithm returning most observed species in a specific years range (data collected from OBIS database). |

ABSENCE_GENERATION_FROM_OBIS
| |

Description | An algorithm to estimate absence records from survey data in OBIS. Based on the work in Coro, G., Magliozzi, C., Berghe, E. V., Bailly, N., Ellenbroek, A., & Pagano, P. (2016). Estimating absence locations of marine species from data of scientific surveys in OBIS. Ecological Modelling, 323, 61-76. |

GEO_CHART
| |

Description | An algorithm producing a charts that displays quantities as colors of countries. The color indicates the sum of the values recorded in a country. |

WEB_APP_PUBLISHER
| |

Description | This algorithm publishes a zip file containing a Web site, based on html and javascript in the e-Infrastructure. It generates a public URL to the application that can be shared. |

KMEANS
| |

Description | A clustering algorithm for real valued vectors that relies on the k-means algorithm, i.e. a method aiming to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. A Maximum of 4000 points is allowed. |

SGVM_INTERPOLATION
| |

Description | An interpolation method relying on the implementation by the Study Group on VMS (SGVMS). The method uses two interpolation approached to simulate vessels points at a certain temporal resolution. The input is a file in TACSAT format uploaded on the Statistical Manager. The output is another TACSAT file containing interpolated points.The underlying R code has been extracted from the SGVM VMSTools framework. This algorithm comes after a feasibility study (http://goo.gl/risQre) which clarifies the features an e-Infrastructure adds to the original scripts. Limitation: the input will be processed up to 10000 vessels trajectory points. Credits: Hintzen, N. T., Bastardie, F., Beare, D., Piet, G. J., Ulrich, C., Deporte, N., Egekvist, J., et al. 2012. VMStools: Open-source software for the processing, analysis and visualisation of fisheries logbook and VMS data. Fisheries Research, 115-116: 31-43. Hintzen, N. T., Piet, G. J., and Brunel, T. 2010. Improved estimation of trawling tracks using cubic Hermite spline interpolation of position registration data. Fisheries Research, 101: 108-115. VMStools, available as an add-on package for R. Documentation available at https://code.google.com/p/vmstools/. Build versions of VMStools for Window, Mac, Linux available at https://docs.google.com/. Authors: Niels T. Hintzen, Doug Beare |

TIMEEXTRACTION
| |

Description | An algorithm to extract a time series of values associated to a geospatial features repository (e.g. NETCDF, ASC, GeoTiff files etc. ). The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram. |

XYEXTRACTOR_TABLE
| |

Description | An algorithm to extract values associated to a table containing geospatial features (e.g. Vessel Routes, Species distribution maps etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial table and the specification about time and space. The algorithm produces one table containing the values associated to the selected bounding box. |

FEED_FORWARD_NEURAL_NETWORK_REGRESSOR
| |

Description | The algorithm simulates a real-valued vector function using a trained Feed Forward Artificial Neural Network and returns a table containing the function actual inputs and the predicted outputs |

GRID_CWP_TO_COORDINATES
| |

Description | An algorithm that adds longitude, latitude and resolution columns analysing a column containing FAO Ocean Area codes (CWP format). |

ESTIMATE_MONTHLY_FISHING_EFFORT
| |

Description | An algorithm that estimates fishing exploitation at 0.5 degrees resolution from activity-classified vessels trajectories. Produces a table with csquare codes, latitudes, longitudes and resolution and associated overall fishing hours in the time frame of the vessels activity. Requires each activity point to be classified as Fishing or other. This algorithm is based on the paper 'Deriving Fishing Monthly Effort and Caught Species' (Coro et al. 2013, in proc. of OCEANS - Bergen, 2013 MTS/IEEE). Example of input table (NAFO anonymised data): http://goo.gl/3auJkM |

FEED_FORWARD_NEURAL_NETWORK_TRAINER
| |

Description | The algorithm trains a Feed Forward Artificial Neural Network using an online Back-Propagation procedure and returns the training error and a binary file containing the trained network |

FIGIS_SPATIAL_REALLOCATION_GENERIC
| |

Description | The Spatial Reallocaton algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation technics and provides at now an area-weighted reallocation. |

TAXONOMY_OBSERVATIONS_TREND_PER_YEAR
| |

Description | Algorithm returning most observations taxonomy trend in a specific years range (with respect to the OBIS database) |

GENERIC_CHARTS
| |

Description | An algorithm producing generic charts of attributes vs. quantities. Charts are displayed per quantity column. Histograms, Scattering and Radar charts are produced for the top ten quantities. A gaussian distribution reports overall statistics for the quantities. |

BIOCLIMATE_HCAF
| |

Description | A transducer algorithm that generates an Half-degree Cells Authority File (HCAF) dataset for a certain time frame, with environmental parameters used by the AquaMaps approach. Evaluates the climatic changes impact on the variation of the ocean features contained in HCAF tables |

SHAPEFILE_PUBLISHER
| |

Description | An algorithm to publish shapefiles under WMS and WFS standards in the e-Infrastructure. The produced WMS, WFS links are reported as output of this process. The map will be available in the VRE for consultation. |

CMSY_2
| |

Description | The CMSY method for data-limited stock assessment. Described in Froese, R., Demirel, N., Coro, G., Kleisner, K. M., Winker, H. (2016). Estimating fisheries reference points from catch and resilience. Fish and Fisheries. Paper link: http://onlinelibrary.wiley.com/doi/10.1111/faf.12190/ Full Instructions and code: https://github.com/SISTA16/cmsy |

FIGIS_SDMX_DATA_CONVERTER
| |

Description | This tool allows to convert easily a SDMX dataset into CSV, by callingthe rsdmx package for R |

HCAF_FILTER
| |

Description | An algorithm producing a HCAF table on a selected Bounding Box (default identifies Indonesia) |

RASTER_DATA_PUBLISHER
| |

Description | This algorithm publishes a raster file as a maps or datasets in the e-Infrastructure. NetCDF-CF files are encouraged, as WMS and WCS maps will be produced using this format. For other types of files (GeoTiffs, ASC etc.) only the raw datasets will be published. The resulting map or dataset will be accessible via the VRE GeoExplorer by the VRE participants. |

AQUAMAPS_SUITABLE_2050
| |

Description | Algorithm for Suitable Range in 2050 by Aquamaps on a single node |

TIMEEXTRACTION_TABLE
| |

Description | An algorithm to extract a time series of values associated to a table containing geospatial information. The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram. |

FIGIS_SPATIAL_REALLOCATION_SIMPLIFIED_TABLE
| |

Description | The Spatial Reallocaton algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation technics and provides at now an area-weighted reallocation. This simplified algorithm is specifically targeting users from the FAO Fisheries and Aquaculture department, aims to facilitate its execution by doing abstraction of the intersections to provide. |

AQUAMAPS_NATIVE_2050
| |

Description | Algorithm for Native Range in 2050 by Aquamaps on a single node |

SPECIES_OBSERVATIONS_TREND_PER_YEAR
| |

Description | An algorithm producing the trend of the observations for a certain species in a certain years range. |

FAOMSY
| |

Description | An algorithm to be used by Fisheries managers for stock assessment. Estimates the Maximum Sustainable Yield (MSY) of a stock, based on a catch trend. The algorithm has been developed by the Resource Use and Conservation Division of the FAO Fisheries and Aquaculture Department (contact: Yimin Ye, yimin.ye@fao.org). It is applicable to a CSV file containing metadata and catch statistics for a set of marine species and produces MSY estimates for each species. The CSV must follow a FAO-defined format (e.g. http://goo.gl/g6YtVx). The output is made up of two (optional) files: one for sucessfully processed species and another one for species that could not be processed because data were not sufficient to estimate MSY. |

PRESENCE_CELLS_GENERATION
| |

Description | An algorithm producing cells and features (HCAF) for a species containing presence points |

ZEXTRACTION
| |

Description | An algorithm to extract the Z values from a geospatial features repository (e.g. NETCDF, ASC, GeoTiff files etc. ). The algorithm analyses the repository and automatically extracts the Z values according to the resolution wanted by the user. It produces one chart of the Z values and one table containing the values. |

MOST_OBSERVED_SPECIES
| |

Description | An algorithm producing a bar chart for the most observed species in a certain years range (with respect to the OBIS database) |

SEADATANET_INTERPOLATOR
| |

Description | A connector for the SeaDataNet infrastructure. This algorithms invokes the Data-Interpolating Variational Analysis (DIVA) SeaDataNet service to interpolate spatial data. The model uses GEBCO bathymetry data and requires an estimate of the maximum spatial span of the correlation between points and the signal-to-noise ratio, among the other parameters. It can interpolate up to 10,000 points randomly taken from the input table. As output, it produces a NetCDF file with a uniform grid of values. This powerful interpolation model is described in Troupin et al. 2012, 'Generation of analysis and consistent error ﬁelds using the Data Interpolating Variational Analysis (Diva)', Ocean Modelling, 52-53, 90-101. |

LISTDBINFO
| |

Description | Algorithm that allows to view information about one chosen resource of Database Type in the Infrastructure |

OCCURRENCES_MARINE_TERRESTRIAL
| |

Description | A transducer algorithm that produces a table containing occurrence points by filtering them by type of area, i.e. by recognising whether they are marine or terrestrial. Works with up to 10000 points per table. |

OCCURRENCES_DUPLICATES_DELETER
| |

Description | A transducer algorithm that produces a duplicate free table of species occurrence points where duplicates have been identified via user defined comparison thresholds. Works with up to 100 000 points |

FIGIS_SPATIAL_REALLOCATION_SIMPLIFIED
| |

Description | The Spatial Reallocaton algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation technics and provides at now an area-weighted reallocation. This simplified algorithm is specifically targeting users from the FAO Fisheries and Aquaculture department, aims to facilitate its execution by doing abstraction of the intersections to provide. |

LISTDBSCHEMA
| |

Description | Algorithm that allows to view the schema names of a chosen database for which the type is Postgres |

OCCURRENCES_SUBTRACTION
| |

Description | A transducer algorithm that produces a table resulting from the difference between two occurrence points tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two Ocurrence Sets, keeps the elements of the Left Set that are not similar to any element in the Right Set. |

OCCURRENCES_MERGER
| |

Description | A transducer algorithm that produces a duplicate-free table resulting from the union of two occurrence points tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two Ocurrence Sets, enrichs the Left Set with the elements of the Right Set that are not in the Left Set. Updates the elements of the Left Set with more recent elements in the Right Set. If one element in the Left Set corresponds to several recent elements in the Right Set, these will be all substituted to the element of the Left Set. |

ZEXTRACTION_TABLE
| |

Description | An algorithm to extract a time series of values associated to a table containing geospatial information. The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram. |

ESTIMATE_FISHING_ACTIVITY
| |

Description | An algorithm that estimates activity hours (fishing or other) from vessels trajectories, adds bathymetry information to the table and classifies (point-by-point) fishing activity of the involved vessels according to two algorithms: one based on speed (activity_class_speed output column) and the other based on speed and bathymetry (activity_class_speed_bath output column). The algorithm produces new columns containing this information. This algorithm is based on the paper 'Deriving Fishing Monthly Effort and Caught Species' (Coro et al. 2013, in proc. of OCEANS - Bergen, 2013 MTS/IEEE). Example of input table (NAFO anonymised data): http://goo.gl/3auJkM |

AQUAMAPSNN
| |

Description | The AquaMaps model trained using a Feed Forward Neural Network. This is a method to train a generic Feed Forward Artifical Neural Network to be used by the AquaMaps Neural Network algorithm. Produces a trained neural network in the form of a compiled file which can be used later. |

MAX_ENT_NICHE_MODELLING
| |

Description | A Maximum-Entropy model for species habitat modeling, based on the implementation by Shapire et al. v 3.3.3k, Princeton University, http://www.cs.princeton.edu/schapire/maxent/. In this adaptation for the D4Science infrastructure, the software accepts a table produced by the Species Product Discovery service and a set of environmental layers in various formats (NetCDF, WFS, WCS, ASC, GeoTiff) via direct links or GeoExplorer UUIDs. The user can also establish the bounding box and the spatial resolution (in decimal deg.) of the training and the projection. The application will adapt the layers to that resolution if this is higher than the native one.The output contains: a thumbnail map of the projected model, the ROC curve, the Omission/Commission chart, a table containing the raw assigned values, a threshold to transform the table into a 0-1 probability distribution, a report of the importance of the used layers in the model, ASCII representations of the input layers to check their alignment.Other processes can be later applied to the raw values to produce a GIS map (e.g. the Statistical Manager Points-to-Map process) and results can be shared. Demo video: http://goo.gl/TYYnTO and instructions http://wiki.i-marine.eu/index.php/MaxEnt |

MAPS_COMPARISON
| |

Description | An algorithm for comparing two OGC/NetCDF maps in seamless way to the user. The algorithm assesses the similarities between two geospatial maps by comparing them in a point-to-point fashion. It accepts as input the two geospatial maps (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) and some parameters affecting the comparison such as the z-index, the time index, the comparison threshold. Note: in the case of WFS layers it makes comparisons on the last feature column. |

SUPPORT_VECTOR_MACHINE_TRAINER
| |

Description | A simple algorithm to train a Support Vector Machine |