Data Transfer Scheduler & Agent components

Intro

Overview

This class of components manages transfer capabilities among gCube infrastructure nodes, in particular but not only it can handles data transfer between Data Sources and Data Storages exploiting the interfaces and the services implemented under the Data Access and Storage Facilities subsystem.

This document outlines the design rationale, key features, and high-level architecture, the options for their deployment and as well some use cases.

Key features

The components belonging to this class are responsible for:

reliable data transfer between Infrastructure Data Sources and Data Storages

by exploiting the uniform access interfaces provided by gCube and standard transfer protocols

structured and unstructured Data Transfer

it guarantees both Tree based and File based transfer to cover all possible iMarine use-cases

transfers to local nodes for data staging

data staging for particular use cases can be enabled on each node of the infrastructure

advanced transfer scheduling and transfer optimization

a dedicated gCube service responsible fot data transfer scheduling combined to transfer optimization at the level of protocols and Access interfaces.

transfer statistics availability

transfers are traced by the system and make available to interested consumers.

transfer shares per scopes and users

a management interface is used to configure transfer shares per scopes and users at the level of Data Sources and Storages.

Design

Philosophy

Data transfer on a distributed infrastructure has to guarantee in first place transfer reliability and optimization in the sense of the resource usage (minimize network load while not causing storage overload). In addition compared to most of the solution developed for data transfer, the solution designed has to take into account not only the standard "unstructured" data transfer ( file transfer) but the capability of "structured" data transfer peculiar to the iMarine data infrastructure.

Architecture

The main components forming this class of Data transfer facilities are two gCube service plus the related libraries:

The gCube Data Transfer Scheduler service ( gDT Scheduler)

The gCube Data Transfer Agent service ( gDT Agent)

The gCube Data Transfer Scheduler Library ( gDT Scheduler Lib)

The gCube Data Transfer Agent Library ( gDT Agent Lib)

Deployment

Large Deployment

figure

Small Deployment

figure