Difference between revisions of "DataMiner Algorithms"

From Gcube Wiki
Jump to: navigation, search
Line 3: Line 3:
 
|}
 
|}
  
The complete list of currently active algorithms published by the [[DataMiner_Manager | DataMiner]] service is reported below.  
+
The complete list of currently active algorithms freely published by the [[DataMiner_Manager | DataMiner]] service is reported below.  
  
 
{|border="1" cellpadding="5" cellspacing="0"
 
{|border="1" cellpadding="5" cellspacing="0"

Revision as of 11:06, 9 November 2017

The complete list of currently active algorithms freely published by the DataMiner service is reported below.

HCAF_INTERPOLATION
Description Evaluates the climatic changes impact on species presence
FEED_FORWARD_A_N_N_DISTRIBUTION
Description A Bayesian method using a Feed Forward Neural Network to simulate a function from the features space (R^n) to R. A modeling algorithm that relies on Neural Networks to simulate a real valued function. It accepts as input a table containing the training dataset and some parameters affecting the algorithm behaviour such as the number of neurons, the learning threshold and the maximum number of iterations.
XYEXTRACTOR
Description An algorithm to extract values associated to an environmental feature repository (e.g. NETCDF, ASC, GeoTiff files etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial repository ID (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) or a direct link to a file and the specification about time and space. The algorithm produces one table containing the values associated to the selected bounding box.
BIOCLIMATE_HSPEN
Description A transducer algorithm that generates a table containing species envelops (HSPEN) in time, i.e. models capturing species tolerance with respect to environmental parameters, used by the AquaMaps approach. Evaluates the climatic changes impact on the variation of the salinity values in several ranges of a set of species envelopes
HSPEN
Description The AquMaps HSPEN algorithm. A modeling algorithm that generates a table containing species envelops (HSPEN), i.e. models capturing species tolerance with respect to environmental parameters, to be used by the AquaMaps approach.
SUPPORT_VECTOR_MACHINE_REGRESSOR
Description A simple algorithm for regression using and already trained Support Vector Machine
AQUAMAPS_NATIVE
Description Algorithm for Native Range by Aquamaps on a single node
POLYGONS_TO_MAP
Description A transducer algorithm to produce a GIS map of filled polygons associated to x,y coordinates and a certain resolution. A maximum of 259000 is allowed
LISTDBNAMES
Description Algorithm that allows to view the available database resources names in the Infrastructure
BIOCLIMATE_HSPEC
Description A transducer algorithm that generates a table containing an estimate of species distributions per half-degree cell (HSPEC) in time. Evaluates the climatic changes impact on species presence.
LOF
Description Local Outlier Factor (LOF). A clustering algorithm for real valued vectors that relies on Local Outlier Factor algorithm, i.e. an algorithm for finding anomalous data points by measuring the local deviation of a given data point with respect to its neighbours. A Maximum of 4000 points is allowed.
TIME_GEO_CHART
Description An algorithm producing an animated gif displaying quantities as colors in time. The color indicates the sum of the values recorded in a country.
SPECIES_OBSERVATIONS_PER_AREA
Description An algorithm producing a bar chart for the distribution of a species along a certain type of marine area (e.g. LME or MEOW)
LWR
Description An algorithm to estimate Length-Weight relationship parameters for marine species, using Bayesian methods. Runs an R procedure. Based on the Cube-law theory.
DBSCAN
Description A clustering algorithm for real valued vectors that relies on the density-based spatial clustering of applications with noise (DBSCAN) algorithm. A maximum of 4000 points is allowed.
BIONYM
Description An algorithm implementing BiOnym, a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication.
DISCREPANCY_ANALYSIS
Description An evaluator algorithm that compares two tables containing real valued vectors. It drives the comparison by relying on a geographical distance threshold and a threshold for K-Statistic.
AQUAMAPS_SUITABLE
Description Algorithm by Aquamaps on a single node
AQUAMAPS_NATIVE_NEURALNETWORK
Description Aquamaps Native Algorithm calculated by a Neural Network. A distribution algorithm that relies on Neural Networks and AquaMaps data for native distributions to generate a table containing species distribution probabilities on half-degree cells.
ECOPATH_WITH_ECOSIM
Description Ecopath with Ecosim (EwE) is a free ecological/ecosystem modeling software suite. This algorithm implementation expects a model and a configuration file as inputs; the result of the analysis is returned as a zip archive. References: Christensen, V., & Walters, C. J. (2004). Ecopath with Ecosim: methods, capabilities and limitations. Ecological modelling, 172(2), 109-139.
OCCURRENCES_INTERSECTOR
Description A transducer algorithm that produces a table of species occurrence points that are contained in both the two starting tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two ocurrence sets, it keeps the elements of the Right Set that are similar to elements in the Left Set.
BIONYM_LOCAL
Description A fast version of the algorithm implementing BiOnym, a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication.
TIME_SERIES_ANALYSIS
Description An algorithms applying signal processing to a non uniform time series. A maximum of 10000 distinct points in time is allowed to be processed. The process uniformly samples the series, then extracts hidden periodicities and signal properties. The sampling period is the shortest time difference between two points. Finally, by using Caterpillar-SSA the algorithm forecasts the Time Series. The output shows the detected periodicity, the forecasted signal and the spectrogram.
OCCURRENCE_ENRICHMENT
Description An algorithm performing occurrences enrichment. Takes as input one table containing occurrence points for a set of species and a list of environmental layer, taken either from the e-infrastructure GeoNetwork (through the GeoExplorer application) or from direct HTTP links. Produces one table reporting the set of environmental values associated to the occurrence points.
FAO_OCEAN_AREA_COLUMN_CREATOR_FROM_QUADRANT
Description An algorithm that adds a column containing the FAO Ocean Area codes associated to longitude, latitude and quadrant columns.
POINTS_TO_MAP
Description A transducer algorithm to produce a GIS map of points from a set of points with x,y coordinates indications. A maximum of 259000 is allowed
BIONYM_BIODIV
Description An algorithm implementing BiOnym oriented to Biodiversity Taxa Names Matching with a predefined and optimized workflow. This version applies in sequence the following Matchers: GSay (thr:0.6, maxRes:10), FuzzyMatcher (thr:0.6, maxRes:10), Levenshtein (thr:0.4, maxRes:10), Trigram (thr:0.4, maxRes:10). BiOnym is a flexible workflow approach to taxon name matching. The workflow allows to activate several taxa names matching algorithms and to get the list of possible transcriptions for a list of input raw species names with possible authorship indication.
XMEANS
Description A clustering algorithm for occurrence points that relies on the X-Means algorithm, i.e. an extended version of the K-Means algorithm improved by an Improve-Structure part. A Maximum of 4000 points is allowed.
QUALITY_ANALYSIS
Description An evaluator algorithm that assesses the effectiveness of a distribution model by computing the Receiver Operating Characteristics (ROC), the Area Under Curve (AUC) and the Accuracy of a model
TIME_SERIES_CHARTS
Description An algorithm producing time series charts of attributes vs. quantities. Charts are displayed per quantity column and superposing quantities are summed.
ABSENCE_CELLS_FROM_AQUAMAPS
Description An algorithm producing cells and features (HCAF) for a species containing absense points taken by an Aquamaps Distribution
AQUAMAPS_SUITABLE_NEURALNETWORK
Description Aquamaps Algorithm for Suitable Environment calculated by Neural Network. A distribution algorithm that relies on Neural Networks and AquaMaps data for suitable distributions to generate a table containing species distribution probabilities on half-degree cells.
SPECIES_MAP_FROM_POINTS
Description A transducer algorithm to produce a GIS map from a probability distribution made upf of x,y coordinates and a certain resolution. A maximum of 259000 is allowed
MOST_OBSERVED_TAXA
Description An algorithm producing a bar chart for the most observed taxa in a certain years range (with respect to the OBIS database)
LISTTABLES
Description Algorithm that allows to view the table names of a chosen database
CSQUARE_COLUMN_CREATOR
Description An algorithm that adds a column containing the CSquare codes associated to longitude and latitude columns.
FAO_OCEAN_AREA_COLUMN_CREATOR
Description An algorithm that adds a column containing the FAO Ocean Area codes associated to longitude and latitude columns.
FEED_FORWARD_ANN
Description A method to train a generic Feed Forward Artifical Neural Network in order to simulate a function from the features space (R^n) to R. Uses the Back-propagation method. Produces a trained neural network in the form of a compiled file which can be used in the FEED FORWARD NEURAL NETWORK DISTRIBUTION algorithm.
SPECIES_MAP_FROM_CSQUARES
Description A transducer algorithm to produce a GIS map from a probability distribution associated to a set of csquare codes. A maximum of 259000 is allowed
HRS
Description An evaluator algorithm that calculates the Habitat Representativeness Score, i.e. an indicator of the assessment of whether a specific survey coverage or another environmental features dataset, contains data that are representative of all available habitat variable combinations in an area.
ESRI_GRID_EXTRACTION
Description An algorithm to extract values associated to an environmental feature repository (e.g. NETCDF, ASC, GeoTiff files etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial repository ID (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) or a direct link to a file and the specification about time and space. The algorithm produces one ESRI GRID ASCII file containing the values associated to the selected bounding box.
SPECIES_OBSERVATION_MEOW_AREA_PER_YEAR
Description Algorithm returning most observed species in a specific years range (data collected from OBIS database).
SPECIES_OBSERVATION_LME_AREA_PER_YEAR
Description Algorithm returning most observed species in a specific years range (data collected from OBIS database).
ABSENCE_GENERATION_FROM_OBIS
Description An algorithm to estimate absence records from survey data in OBIS. Based on the work in Coro, G., Magliozzi, C., Berghe, E. V., Bailly, N., Ellenbroek, A., & Pagano, P. (2016). Estimating absence locations of marine species from data of scientific surveys in OBIS. Ecological Modelling, 323, 61-76.
GEO_CHART
Description An algorithm producing a charts that displays quantities as colors of countries. The color indicates the sum of the values recorded in a country.
WEB_APP_PUBLISHER
Description This algorithm publishes a zip file containing a Web site, based on html and javascript in the e-Infrastructure. It generates a public URL to the application that can be shared.
KMEANS
Description A clustering algorithm for real valued vectors that relies on the k-means algorithm, i.e. a method aiming to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. A Maximum of 4000 points is allowed.
SGVM_INTERPOLATION
Description An interpolation method relying on the implementation by the Study Group on VMS (SGVMS). The method uses two interpolation approached to simulate vessels points at a certain temporal resolution. The input is a file in TACSAT format uploaded on the Statistical Manager. The output is another TACSAT file containing interpolated points.The underlying R code has been extracted from the SGVM VMSTools framework. This algorithm comes after a feasibility study (http://goo.gl/risQre) which clarifies the features an e-Infrastructure adds to the original scripts. Limitation: the input will be processed up to 10000 vessels trajectory points. Credits: Hintzen, N. T., Bastardie, F., Beare, D., Piet, G. J., Ulrich, C., Deporte, N., Egekvist, J., et al. 2012. VMStools: Open-source software for the processing, analysis and visualisation of fisheries logbook and VMS data. Fisheries Research, 115-116: 31-43. Hintzen, N. T., Piet, G. J., and Brunel, T. 2010. Improved estimation of trawling tracks using cubic Hermite spline interpolation of position registration data. Fisheries Research, 101: 108-115. VMStools, available as an add-on package for R. Documentation available at https://code.google.com/p/vmstools/. Build versions of VMStools for Window, Mac, Linux available at https://docs.google.com/. Authors: Niels T. Hintzen, Doug Beare
TIMEEXTRACTION
Description An algorithm to extract a time series of values associated to a geospatial features repository (e.g. NETCDF, ASC, GeoTiff files etc. ). The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram.
XYEXTRACTOR_TABLE
Description An algorithm to extract values associated to a table containing geospatial features (e.g. Vessel Routes, Species distribution maps etc. ). A grid of points at a certain resolution is specified by the user and values are associated to the points from the environmental repository. It accepts as one geospatial table and the specification about time and space. The algorithm produces one table containing the values associated to the selected bounding box.
FEED_FORWARD_NEURAL_NETWORK_REGRESSOR
Description The algorithm simulates a real-valued vector function using a trained Feed Forward Artificial Neural Network and returns a table containing the function actual inputs and the predicted outputs
GRID_CWP_TO_COORDINATES
Description An algorithm that adds longitude, latitude and resolution columns analysing a column containing FAO Ocean Area codes (CWP format).
ESTIMATE_MONTHLY_FISHING_EFFORT
Description An algorithm that estimates fishing exploitation at 0.5 degrees resolution from activity-classified vessels trajectories. Produces a table with csquare codes, latitudes, longitudes and resolution and associated overall fishing hours in the time frame of the vessels activity. Requires each activity point to be classified as Fishing or other. This algorithm is based on the paper 'Deriving Fishing Monthly Effort and Caught Species' (Coro et al. 2013, in proc. of OCEANS - Bergen, 2013 MTS/IEEE). Example of input table (NAFO anonymised data): http://goo.gl/3auJkM
FEED_FORWARD_NEURAL_NETWORK_TRAINER
Description The algorithm trains a Feed Forward Artificial Neural Network using an online Back-Propagation procedure and returns the training error and a binary file containing the trained network
FIGIS_SPATIAL_REALLOCATION_GENERIC
Description The Spatial Reallocaton algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation technics and provides at now an area-weighted reallocation.
TAXONOMY_OBSERVATIONS_TREND_PER_YEAR
Description Algorithm returning most observations taxonomy trend in a specific years range (with respect to the OBIS database)
GENERIC_CHARTS
Description An algorithm producing generic charts of attributes vs. quantities. Charts are displayed per quantity column. Histograms, Scattering and Radar charts are produced for the top ten quantities. A gaussian distribution reports overall statistics for the quantities.
BIOCLIMATE_HCAF
Description A transducer algorithm that generates an Half-degree Cells Authority File (HCAF) dataset for a certain time frame, with environmental parameters used by the AquaMaps approach. Evaluates the climatic changes impact on the variation of the ocean features contained in HCAF tables
SHAPEFILE_PUBLISHER
Description An algorithm to publish shapefiles under WMS and WFS standards in the e-Infrastructure. The produced WMS, WFS links are reported as output of this process. The map will be available in the VRE for consultation.
CMSY_2
Description The CMSY method for data-limited stock assessment. Described in Froese, R., Demirel, N., Coro, G., Kleisner, K. M., Winker, H. (2016). Estimating fisheries reference points from catch and resilience. Fish and Fisheries. Paper link: http://onlinelibrary.wiley.com/doi/10.1111/faf.12190/ Full Instructions and code: https://github.com/SISTA16/cmsy
FIGIS_SDMX_DATA_CONVERTER
Description This tool allows to convert easily a SDMX dataset into CSV, by callingthe rsdmx package for R
HCAF_FILTER
Description An algorithm producing a HCAF table on a selected Bounding Box (default identifies Indonesia)
RASTER_DATA_PUBLISHER
Description This algorithm publishes a raster file as a maps or datasets in the e-Infrastructure. NetCDF-CF files are encouraged, as WMS and WCS maps will be produced using this format. For other types of files (GeoTiffs, ASC etc.) only the raw datasets will be published. The resulting map or dataset will be accessible via the VRE GeoExplorer by the VRE participants.
AQUAMAPS_SUITABLE_2050
Description Algorithm for Suitable Range in 2050 by Aquamaps on a single node
TIMEEXTRACTION_TABLE
Description An algorithm to extract a time series of values associated to a table containing geospatial information. The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram.
FIGIS_SPATIAL_REALLOCATION_SIMPLIFIED_TABLE
Description The Spatial Reallocaton algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation technics and provides at now an area-weighted reallocation. This simplified algorithm is specifically targeting users from the FAO Fisheries and Aquaculture department, aims to facilitate its execution by doing abstraction of the intersections to provide.
AQUAMAPS_NATIVE_2050
Description Algorithm for Native Range in 2050 by Aquamaps on a single node
SPECIES_OBSERVATIONS_TREND_PER_YEAR
Description An algorithm producing the trend of the observations for a certain species in a certain years range.
FAOMSY
Description An algorithm to be used by Fisheries managers for stock assessment. Estimates the Maximum Sustainable Yield (MSY) of a stock, based on a catch trend. The algorithm has been developed by the Resource Use and Conservation Division of the FAO Fisheries and Aquaculture Department (contact: Yimin Ye, yimin.ye@fao.org). It is applicable to a CSV file containing metadata and catch statistics for a set of marine species and produces MSY estimates for each species. The CSV must follow a FAO-defined format (e.g. http://goo.gl/g6YtVx). The output is made up of two (optional) files: one for sucessfully processed species and another one for species that could not be processed because data were not sufficient to estimate MSY.
PRESENCE_CELLS_GENERATION
Description An algorithm producing cells and features (HCAF) for a species containing presence points
ZEXTRACTION
Description An algorithm to extract the Z values from a geospatial features repository (e.g. NETCDF, ASC, GeoTiff files etc. ). The algorithm analyses the repository and automatically extracts the Z values according to the resolution wanted by the user. It produces one chart of the Z values and one table containing the values.
MOST_OBSERVED_SPECIES
Description An algorithm producing a bar chart for the most observed species in a certain years range (with respect to the OBIS database)
SEADATANET_INTERPOLATOR
Description A connector for the SeaDataNet infrastructure. This algorithms invokes the Data-Interpolating Variational Analysis (DIVA) SeaDataNet service to interpolate spatial data. The model uses GEBCO bathymetry data and requires an estimate of the maximum spatial span of the correlation between points and the signal-to-noise ratio, among the other parameters. It can interpolate up to 10,000 points randomly taken from the input table. As output, it produces a NetCDF file with a uniform grid of values. This powerful interpolation model is described in Troupin et al. 2012, 'Generation of analysis and consistent error fields using the Data Interpolating Variational Analysis (Diva)', Ocean Modelling, 52-53, 90-101.
LISTDBINFO
Description Algorithm that allows to view information about one chosen resource of Database Type in the Infrastructure
OCCURRENCES_MARINE_TERRESTRIAL
Description A transducer algorithm that produces a table containing occurrence points by filtering them by type of area, i.e. by recognising whether they are marine or terrestrial. Works with up to 10000 points per table.
OCCURRENCES_DUPLICATES_DELETER
Description A transducer algorithm that produces a duplicate free table of species occurrence points where duplicates have been identified via user defined comparison thresholds. Works with up to 100 000 points
FIGIS_SPATIAL_REALLOCATION_SIMPLIFIED
Description The Spatial Reallocaton algorithm allows to estimate statistics for other areas from those where they were reported. The algorithm is based on spatial disaggregation technics and provides at now an area-weighted reallocation. This simplified algorithm is specifically targeting users from the FAO Fisheries and Aquaculture department, aims to facilitate its execution by doing abstraction of the intersections to provide.
LISTDBSCHEMA
Description Algorithm that allows to view the schema names of a chosen database for which the type is Postgres
OCCURRENCES_SUBTRACTION
Description A transducer algorithm that produces a table resulting from the difference between two occurrence points tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two Ocurrence Sets, keeps the elements of the Left Set that are not similar to any element in the Right Set.
OCCURRENCES_MERGER
Description A transducer algorithm that produces a duplicate-free table resulting from the union of two occurrence points tables where points equivalence is identified via user defined comparison thresholds. Works with up to 10000 points per table. Between two Ocurrence Sets, enrichs the Left Set with the elements of the Right Set that are not in the Left Set. Updates the elements of the Left Set with more recent elements in the Right Set. If one element in the Left Set corresponds to several recent elements in the Right Set, these will be all substituted to the element of the Left Set.
ZEXTRACTION_TABLE
Description An algorithm to extract a time series of values associated to a table containing geospatial information. The algorithm analyses the time series and automatically searches for hidden periodicities. It produces one chart of the time series, one table containing the time series values and possibly the spectrogram.
ESTIMATE_FISHING_ACTIVITY
Description An algorithm that estimates activity hours (fishing or other) from vessels trajectories, adds bathymetry information to the table and classifies (point-by-point) fishing activity of the involved vessels according to two algorithms: one based on speed (activity_class_speed output column) and the other based on speed and bathymetry (activity_class_speed_bath output column). The algorithm produces new columns containing this information. This algorithm is based on the paper 'Deriving Fishing Monthly Effort and Caught Species' (Coro et al. 2013, in proc. of OCEANS - Bergen, 2013 MTS/IEEE). Example of input table (NAFO anonymised data): http://goo.gl/3auJkM
AQUAMAPSNN
Description The AquaMaps model trained using a Feed Forward Neural Network. This is a method to train a generic Feed Forward Artifical Neural Network to be used by the AquaMaps Neural Network algorithm. Produces a trained neural network in the form of a compiled file which can be used later.
MAX_ENT_NICHE_MODELLING
Description A Maximum-Entropy model for species habitat modeling, based on the implementation by Shapire et al. v 3.3.3k, Princeton University, http://www.cs.princeton.edu/schapire/maxent/. In this adaptation for the D4Science infrastructure, the software accepts a table produced by the Species Product Discovery service and a set of environmental layers in various formats (NetCDF, WFS, WCS, ASC, GeoTiff) via direct links or GeoExplorer UUIDs. The user can also establish the bounding box and the spatial resolution (in decimal deg.) of the training and the projection. The application will adapt the layers to that resolution if this is higher than the native one.The output contains: a thumbnail map of the projected model, the ROC curve, the Omission/Commission chart, a table containing the raw assigned values, a threshold to transform the table into a 0-1 probability distribution, a report of the importance of the used layers in the model, ASCII representations of the input layers to check their alignment.Other processes can be later applied to the raw values to produce a GIS map (e.g. the Statistical Manager Points-to-Map process) and results can be shared. Demo video: http://goo.gl/TYYnTO and instructions http://wiki.i-marine.eu/index.php/MaxEnt
MAPS_COMPARISON
Description An algorithm for comparing two OGC/NetCDF maps in seamless way to the user. The algorithm assesses the similarities between two geospatial maps by comparing them in a point-to-point fashion. It accepts as input the two geospatial maps (via their UUIDs in the infrastructure spatial data repository - recoverable through the Geoexplorer portlet) and some parameters affecting the comparison such as the z-index, the time index, the comparison threshold. Note: in the case of WFS layers it makes comparisons on the last feature column.
SUPPORT_VECTOR_MACHINE_TRAINER
Description A simple algorithm to train a Support Vector Machine