Difference between revisions of "Statistical Algorithms Importer: Docker Support"

From Gcube Wiki
Jump to: navigation, search
(The Algorithm Docker Image Executor (DIE))
Line 2: Line 2:
 
||__TOC__
 
||__TOC__
 
|}
 
|}
+
 
This page explains how to create and run Docker Images in the D4Science infrastructure through the [[DataMiner_Manager|DataMiner Manager]] service and the algorithms developed with the [[Statistical_Algorithms_Importer|Statistical Algorithms Importer (SAI)]]. Currently for this purpose there is the [https://services.d4science.org/group/rprototypinglab/data-miner?OperatorId=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.DOCKER_IMAGE_EXECUTOR Docker Image Executor (DIE)] algorithm. More information on Docker can be found [https://www.docker.com/ here].
+
A Docker image represents an easy-way to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files and thus they may contribute to simplifying the configuration of the D4Science infrastructure.
 +
D4Science delivers a solution allowing to exploit Docker while preserving the main features of the D4Science infrastructure: replicability, reusability, sharing, accounting of the execution will all be preserved by following and exploiting the Docker Image Executor.
 +
 
 +
This page explains how to create and run Docker Images in the D4Science infrastructure through the [[DataMiner_Manager|DataMiner Manager]] service and the algorithms developed with the [[Statistical_Algorithms_Importer|Statistical Algorithms Importer (SAI)]]. Currently, for this purpose there is the [https://services.d4science.org/group/rprototypinglab/data-miner?OperatorId=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.DOCKER_IMAGE_EXECUTOR Docker Image Executor (DIE)] algorithm. More information on Docker can be found [https://www.docker.com/ here].
  
 
== The Algorithm Docker Image Executor (DIE) ==
 
== The Algorithm Docker Image Executor (DIE) ==
Line 27: Line 30:
 
=== How Create A Docker Image ===
 
=== How Create A Docker Image ===
  
An example of how create a Docker Image suitable for running via the DIE algorithm is shown [https://code-repo.d4science.org/gCubeSystem/sortapp here]. This image is built starting from the base python:3.6-alpine image and installing the sortapp application written in python3.6 (see [https://code-repo.d4science.org/gCubeSystem/sortapp/src/branch/master/Dockerfile Dockerfile]).
+
An example of how to create a Docker Image suitable for running via the DIE algorithm is shown [https://code-repo.d4science.org/gCubeSystem/sortapp here]. This image is built starting from the base python:3.6-alpine image and installing the sortapp application written in python3.6 (see [https://code-repo.d4science.org/gCubeSystem/sortapp/src/branch/master/Dockerfile Dockerfile]).
  
:The image is available on Docker Hub here: [https://hub.docker.com/repository/docker/d4science/sortapp d4science/sortapp]
+
: The image is available on Docker Hub here: [https://hub.docker.com/repository/docker/d4science/sortapp d4science/sortapp]
  
 
The sortapp application built in this example simply does the sorting of strings. The strings are contained in the file indicated by the FileParam parameter.
 
The sortapp application built in this example simply does the sorting of strings. The strings are contained in the file indicated by the FileParam parameter.

Revision as of 13:49, 24 September 2020

A Docker image represents an easy-way to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files and thus they may contribute to simplifying the configuration of the D4Science infrastructure. D4Science delivers a solution allowing to exploit Docker while preserving the main features of the D4Science infrastructure: replicability, reusability, sharing, accounting of the execution will all be preserved by following and exploiting the Docker Image Executor.

This page explains how to create and run Docker Images in the D4Science infrastructure through the DataMiner Manager service and the algorithms developed with the Statistical Algorithms Importer (SAI). Currently, for this purpose there is the Docker Image Executor (DIE) algorithm. More information on Docker can be found here.

The Algorithm Docker Image Executor (DIE)

The Docker Image Executor (DIE) algorithm allows you to retrieve and run a image in D4Science Swarm cluster, from a Docker Hub repository (Attention only public repositories are supported). The DIE is already present and accessible on the D4Science infrastructure:

Docker Image Executor (DIE), Docker Support

To run the DIE algorithm the user must enter:

  • Image, the name of the repository (e.g. d4science/sortapp)
  • CommandName, the name of the command to invoke when the service is started (e.g. sortapp)
  • FileParam, a file present in the user's workspace to be passed as an input parameter along with the run command (e.g. sortableelements.txt)

This algorithm will take care of retrieving the user token and passing the parameters to the Docker Service in this format:

<command-name> <token> <file-item-id> <temp-dir-item-id>

In addition to passing the token and the input file, the algorithm also passes the id of the temporary folder that was created on the StorageHub service to contain the computation results. The service created from the chosen image will be responsible for saving the data of its own computation in the folder indicated by the algorithm interacting with the StorageHub service. When the execution of the Docker Service is finished, that is, it will have created the results and saved on the temporary folder, then the Docker Image Executor (DIE) algorithm will take care of returning the result as a zip file of the temporary folder. So, it is important that the Docker Image is written with these constraints in mind.

How Create A Docker Image

An example of how to create a Docker Image suitable for running via the DIE algorithm is shown here. This image is built starting from the base python:3.6-alpine image and installing the sortapp application written in python3.6 (see Dockerfile).

The image is available on Docker Hub here: d4science/sortapp

The sortapp application built in this example simply does the sorting of strings. The strings are contained in the file indicated by the FileParam parameter.

So in general, the image could also be constructed using other languages and different base images. What remains binding is that the image that is created accepts the parameters as passed by the DIE and respects the constraint to save the results in the temporary folder on StorageHub as indicated.