Difference between revisions of "Data Transfer Agent"

Revision as of 17:22, 1 June 2012

Data Transfer Agent Service

The Data Transfer Agent Service has been implemented with the aim of facilitating the transfer of data ( both structured and unstructured ) for the following use cases:

• Transfer of Local files from an external/internal client to a remote GHN
• File Transfer from a remote Data Source to a remote GHN using standard protocol ( HTTP, FTP, BitTorrent) thanks to the integration with the URLResolution Library.
• File Transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocol ( HTTP, FTP, BitTorrent) thanks to the integration with the URLResolution Library.
• Tree based data transfer from a remote Data Source to a remote Tree-Based Storage.

The service make use of gRS to implement Local Transfer, Tree based Data Transfer and delivery of Transfer Outcomes to clients.

In addition the service exploit the Messaging infrastructure in order to publish transfer statistics, that can be consumed by:

• Accounting statistics consumers
• the Data Transfer Scheduler Service, which use messaging in order to consume Agent transfer results.

Data Transfer Agent Library

The Data Transfer Agent Library is the CL implementing the API for Data Transfer. In particular the Library implement the API to contact the Data Transfer Agent Service

The latest version of the library (1.1.0-SNAPSHOT) and the related javadoc artifacs can be downloaded from maven repository at [1]

In case of integration with a maven components the dependency to be included in the pom file is:

<dependency>
  <groupId>org.gcube.data.transfer</groupId>
  <artifactId>agent-library</artifactId>
  <version>1.1.0-SNAPSHOT</version>
</dependency>

The Library offers both sync and asynchronous data transfer operations.

This is a simple code snippet which invokes a synchronous  data transfer of a list of URIs locally to a GHN which runs an instance of the Data Transfer agent service:

ScopeProvider.instance.set(/gcube/devsec/);
 
AgentLibrary library = new AgentLibrary("test.research-infrastructures.eu",9090);
 
URL [] input = new URL [] {
        new URL("ftp://pcd4science3.cern.ch/test.txt"),
	new URL("http://dl.dropbox.com/u/8704957/commons-dbcp-1.3.jar")};
 
String outPath = "/tmp";
 
ArrayList<TransferOutcome> outcomes = library.startTransferSync(input, outPath, StorageType.LocalGHN, false);
 
for (TransferOutcome outcome : outcomes)
{
		logger.debug("Exception: "+outcome.getException());
		logger.debug("Success?: "+ outcome.isSuccess());
		logger.debug("Failure?: "+ outcome.isFailure());
}