Difference between revisions of "Storage Manager"

From Gcube Wiki
Jump to: navigation, search
(Operations)
Line 22: Line 22:
 
== Operations ==
 
== Operations ==
  
The StorageManger library implements many operations on remote files, the methods LFile and RFile refer to, respectively,  Local File (if needed) and Remote File. In the call's signature, they are preceded by a method that identify an operation, see examples below:
+
The StorageManger library implements many operations on remote files, the methods LFile and RFile refer to, respectively,  Local File (if needed) and Remote File. The remote file could be identified by path or by id. The id is returned when the file is created. In the call's signature, these methods are preceded by a method that identify an operation, see examples below:
  
 
;put(boolean replace)   
 
;put(boolean replace)   
 
:put a local file in a remote directory. If replace is true then the remote file, if exists, will be replaced.
 
:put a local file in a remote directory. If replace is true then the remote file, if exists, will be replaced.
 
: If the remote directory does not exist it will be automatically  
 
: If the remote directory does not exist it will be automatically  
:Example: put(true).LFile(“localPath”).RFile(“remotePath”)
+
:Example: put(true).LFile(“localPath”).RFile(“remotePath”);
 +
:Example: put(true).LFile(“localPath”).RFile(id);
 +
 
  
 
;get()  
 
;get()  
 
:downloads a file from a remote path to a local directory
 
:downloads a file from a remote path to a local directory
 
:Example: get().LFile(“localPath”).RFile(“remotePath”);
 
:Example: get().LFile(“localPath”).RFile(“remotePath”);
 +
:Example: get().LFile(“localPath”).RFile(id);
  
 
;remove()  
 
;remove()  
 
:removes a remote file
 
:removes a remote file
 
:Example: remove().RFile(“remotePath”);
 
:Example: remove().RFile(“remotePath”);
 +
Example: remove().RFile(id);
  
 
;removeDir()  
 
;removeDir()  
Line 66: Line 70:
  
 
;getUrl()  
 
;getUrl()  
:returns the url of a file stored in the repository
+
:returns the smp url of a file stored in the repository
 
:this method is not compatible with private access mode  
 
:this method is not compatible with private access mode  
 
:Example: getUrl().RFile(“remotePath”);
 
:Example: getUrl().RFile(“remotePath”);
 +
:Example: getUrl().RFile(id);
 +
 +
;getHttpUrl()
 +
:returns the http url of a file stored in the repository
 +
:this method is not compatible with private access mode
 +
:Example: getUrl().RFile(“remotePath”);
 +
:Example: getUrl().RFile(id);
 +
  
 
;copy()  
 
;copy()  
Line 115: Line 127:
 
or alternatively, download the file by Id
 
or alternatively, download the file by Id
 
<code>
 
<code>
  client.get().LFile("/home/rcirillo/FilePerTest/pippo.jpg").RFileById(id);
+
  client.get().LFile("/home/rcirillo/FilePerTest/pippo.jpg").RFile(id);
 
</code>
 
</code>
  
Line 148: Line 160:
 
or alternatively, remove a file by Id
 
or alternatively, remove a file by Id
 
<code>
 
<code>
  client.remove()..RFileById(id);
+
  client.remove()..RFile(id);
 
</code>
 
</code>
  
Line 228: Line 240:
 
         <Interface>
 
         <Interface>
 
                  
 
                  
             <Endpoint EntryName="server1">146.48.123.71</Endpoint>
+
             <Endpoint EntryName="server1">146.48.123.01</Endpoint>
 
              
 
              
 
         </Interface>
 
         </Interface>
Line 246: Line 258:
 
               <Name>host</Name>
 
               <Name>host</Name>
 
                      
 
                      
               <Value encrypted="false">node41.d.d4science.research-infrastructures.eu</Value>
+
               <Value encrypted="false">nodeXX.d.d4science.research-infrastructures.eu</Value>
 
                  
 
                  
 
             </Property>
 
             </Property>
Line 276: Line 288:
 
         <Interface>
 
         <Interface>
 
                  
 
                  
             <Endpoint EntryName="server2">146.48.123.72</Endpoint>
+
             <Endpoint EntryName="server2">146.48.123.02</Endpoint>
 
              
 
              
 
         </Interface>
 
         </Interface>
Line 294: Line 306:
 
               <Name>host</Name>
 
               <Name>host</Name>
 
                      
 
                      
               <Value encrypted="false">node42.d.d4science.research-infrastructures.eu</Value>
+
               <Value encrypted="false">nodeYY.d.d4science.research-infrastructures.eu</Value>
 
                  
 
                  
 
             </Property>
 
             </Property>
Line 335: Line 347:
 
     <groupId>org.gcube.contentmanagement</groupId>
 
     <groupId>org.gcube.contentmanagement</groupId>
 
     <artifactId>storage-manager-core</artifactId>
 
     <artifactId>storage-manager-core</artifactId>
     <version>[1.0.0, 2.0.0)</version>
+
     <version>[2.0.0, 3.0.0)</version>
 
   </dependency>
 
   </dependency>
 
   <dependency>
 
   <dependency>
 
     <groupId>org.gcube.contentmanagement</groupId>
 
     <groupId>org.gcube.contentmanagement</groupId>
 
     <artifactId>storage-manager-wrapper</artifactId>
 
     <artifactId>storage-manager-wrapper</artifactId>
     <version>[1.0.0, 2.0.0)</version>
+
     <version>[2.0.0, 3.0.0)</version>
 
   </dependency>
 
   </dependency>
 
</source>
 
</source>
Line 390: Line 402:
 
</code>
 
</code>
  
= storage-manager v 1.0.1 =
+
= VOLATILE area =
  
;VOLATILE area
+
;
The 1.0.1 version  of storage-manager has a new storage area called VOLATILE.
+
Since 1.0.1 version  of storage-manager there is a new storage area called VOLATILE.
 
The files inserted on this special area are automatically '''removed after a period of 7 days''' from the first upload.
 
The files inserted on this special area are automatically '''removed after a period of 7 days''' from the first upload.
 
The use of this area is intended for all clients that need to use storage for temporary data.
 
The use of this area is intended for all clients that need to use storage for temporary data.
Line 411: Line 423:
 
Another instance of the same service, s2 (identify by: ServiceClass: sc2, ServiceName: sn2) running in the same scope but on another node, download it.
 
Another instance of the same service, s2 (identify by: ServiceClass: sc2, ServiceName: sn2) running in the same scope but on another node, download it.
  
The following code shows the file uploading operation:
+
The following code shows the upload operation:
  
 
<code>
 
<code>
Line 419: Line 431:
 
</code>
 
</code>
  
The following code shows the file downloading operation:
+
The following code shows the download operation:
  
 
<code>
 
<code>

Revision as of 15:57, 14 July 2016


Role

Library for access and storage of unstructured bytestreams, or files, can be provided through a standards-based, POSIX-like API which supports the organisation and operations normally associated with local file systems whilst offering scalable and fault-tolerant remote storage.

The library acts a facade to the service and allows clients to download, upload, remove, add and list files. Files have owners and owners may define access rights to files, allowing private, public, or group-based access.

Access Mode

There are 3 ways to access remote files:

Private
Accessible only to the owner of the file
Public
Accessible to all users of the community
Shared
Accessible to all users of the same group of the owner

These types of access will be specified when instantiating the client. If the client is instantiated for example,with the private access mode, all the operations done with this client instance will be in private access mode.

Operations

The StorageManger library implements many operations on remote files, the methods LFile and RFile refer to, respectively, Local File (if needed) and Remote File. The remote file could be identified by path or by id. The id is returned when the file is created. In the call's signature, these methods are preceded by a method that identify an operation, see examples below:

put(boolean replace)
put a local file in a remote directory. If replace is true then the remote file, if exists, will be replaced.
If the remote directory does not exist it will be automatically
Example: put(true).LFile(“localPath”).RFile(“remotePath”);
Example: put(true).LFile(“localPath”).RFile(id);


get()
downloads a file from a remote path to a local directory
Example: get().LFile(“localPath”).RFile(“remotePath”);
Example: get().LFile(“localPath”).RFile(id);
remove()
removes a remote file
Example: remove().RFile(“remotePath”);

Example: remove().RFile(id);

removeDir()
removes a remote directory and all files in it. This operation is recursive.
Example removeDir().RDir(“remoteDirPath”);
showDir()
shows the content of a directory
Example showDir.RFile(“remoteDirPath”);
lock()
locks a remote file to prevent multiple access. It returns an id string for unlock the file. Remember that every lock has a TTL associated (Default: 180000 ms).
Example lock().RFile(“remotePath”);
Otherwise: Download the file in "localFilePath" and Lock the remote file stored on backend server
Example: lock().LFile("localFilePath").RFile("remotePath")
unlock(String key4Unlock)
if the file is locked, unlocks the remote file.
Example unlock(“key4Unlock”).RFile(“remotePath”);
Otherwise: Upload the new File stored in the local FileSystem at: "localFilePath" and unlock the file on the backend server
Example: unlock("key4Unlock").LFile("localFilePath").RFile("remotePath")
getTTL()
returns the TimeToLive associated to a remote file if the file is currently locked
Example getTTL().RFile("remoteDirPath");
setTTL(int milliseconds)
resets the TimeToLive to a remote file that is currently locked. This operation is consented max 5 times for any file.
Example setTTL(180000).RFile("remoteDirPath");
getUrl()
returns the smp url of a file stored in the repository
this method is not compatible with private access mode
Example: getUrl().RFile(“remotePath”);
Example: getUrl().RFile(id);
getHttpUrl()
returns the http url of a file stored in the repository
this method is not compatible with private access mode
Example: getUrl().RFile(“remotePath”);
Example: getUrl().RFile(id);


copy()
copies a remote file from a remote path to another remote path
return the id of the new file stored in the repository
this method is available only in version 2.0.0
Example: copy().RFile(“remotePath1”).Rfile(“remotePath2”);
moveFile()
moves a remote file from a remote path to another remote path
return the id of the file stored in the repository
this method is available only in version 2.0.0 or later
Example: moveFile().from(“remoteFilePath1”).to(“remoteFilePath2”);
moveDir()
moves a remote directory from a remote directory path to another remote directory path
this method is available only in version 2.0.0 or later
Example: moveDir().from(“remoteDirPath1”).to(“remoteDirPath2”);
link()
links a remote file from a remote path to another remote path as an hard link in Unix system
return the id of the new file linked in the repository
this method is available only in version 2.0.0
Example: link().RFile(“remotePath1”).Rfile(“remotePath2”);

Sample Usage

The following code explains how to instatiate a client for do remote operations on the storage repository:

GCUBEScope scope = GCUBEScope.getScope("/gcube");
 // The Access type can be private | shared | public
IClient client=new StorageClient("ServiceClass", "ServiceName", "owner", AccessType.SHARED, scope).getClient();


The following code explains how to upload a new file in the storage repository:

String id=client.put(true).LFile("/home/rcirillo/FilePerTest/pippo.jpg").RFile("/img/pippo.jpg");


The following code explains how to download a file from the storage repository:

client.get().LFile("/home/rcirillo/FilePerTest/pippo.jpg").RFile("/img/pippo.jpg");

or alternatively, download the file by Id

client.get().LFile("/home/rcirillo/FilePerTest/pippo.jpg").RFile(id);

or alternatively, download the file by stream

InputStream is = client.get().RFileAStream("/img/pippo.jpg");


The following code explains how to show the contents of a remote directory on the storage repository:

List<StorageObject> list=client.showDir().RDir("img");


The following code explains how to lock a file on the storage repository:

String id=client.lock().RFile("/img/pluto.jpg");


The following code explains how to unlock a file on the storage repository:

client.unlock(id).RFile("/img/pluto.jpg");


The following code explains how to remove a file on the storage repository:

client.remove().RFile("/img/pluto.jpg");

or alternatively, remove a file by Id

client.remove()..RFile(id);


The following code explains how to remove a dir on the storage repository:

client.removeDir().RDir("/img");

The following code explains how to move a dir on the storage repository from a remote path to another remote path:

client.moveDir().from("/img/test1").to("/img/test2");

The following code explains how to move a file from a remote path to another remote path:

client.moveFile().from("/img/test1.xml").to("/img/test2.xml");

Deployment

To release the library at a particular gcube scope you need to create only a runtime resource that specifies the address of the server used for storing data. This is an example of runtime resource used.

<Resource version="0.4.x">
 
   <ID>46083b40-a5e3-11e2-a210-9a433747c17a</ID>
 
   <Type>RuntimeResource</Type>
 
   <Scopes>
 
      <Scope>/gcube/devsec</Scope>
 
      <Scope>/gcube</Scope>
 
      <Scope>/gcube/devsec/devVRE</Scope>
    </Scopes>    <Profile>       <Category>DataStorage</Category>
 
      <Name>StorageManager</Name>
 
      <Description>Backend description  for StorageManager library</Description>
 
      <Platform>
 
         <Name>storage-manager</Name>
 
         <Version>2</Version>
 
         <MinorVersion>0</MinorVersion>
 
         <RevisionVersion>0</RevisionVersion>
 
         <BuildVersion>0</BuildVersion>
 
      </Platform>
 
      <RunTime>
 
         <HostedOn>d4science.org</HostedOn>
 
         <GHN UniqueID="" />
 
         <Status>READY</Status>
 
      </RunTime>
 
      <AccessPoint>
 
         <Description>MongoDB server</Description>
 
         <Interface>
 
            <Endpoint EntryName="server1">146.48.123.01</Endpoint>
 
         </Interface>
 
         <AccessData>
 
            <Username />
 
            <Password>6vW1u92cpdgHzYAgIurn9w==</Password>
 
         </AccessData>
 
         <Properties>
 
            <Property>
 
               <Name>host</Name>
 
               <Value encrypted="false">nodeXX.d.d4science.research-infrastructures.eu</Value>
 
            </Property>
 
            <Property>
 
               <Name>priority</Name>
 
               <Value encrypted="false">default</Value>
 
            </Property>
 
            <Property>
 
               <Name>type</Name>
 
               <Value encrypted="false">MongoDB</Value>
 
            </Property>
 
         </Properties>
 
      </AccessPoint>
 
      <AccessPoint>
 
         <Description>MongoDB server</Description>
 
         <Interface>
 
            <Endpoint EntryName="server2">146.48.123.02</Endpoint>
 
         </Interface>
 
         <AccessData>
 
            <Username />
 
            <Password>6vW1u92cpdgHzYAgIurn9w==</Password>
 
         </AccessData>
 
         <Properties>
 
            <Property>
 
               <Name>host</Name>
 
               <Value encrypted="false">nodeYY.d.d4science.research-infrastructures.eu</Value>
 
            </Property>
 
            <Property>
 
               <Name>priority</Name>
 
               <Value encrypted="false">default</Value>
 
            </Property>
 
            <Property>
 
               <Name>type</Name>
 
               <Value encrypted="false">MongoDB</Value>
 
            </Property>
 
         </Properties>
 
      </AccessPoint>
 
   </Profile>
 
</Resource>

The "endpoint" tag specifies the ip of server. The resource can specify one or more servers. In this case there are three servers. If the servers are more than one then the second server has attribute value: "server2" and so on. The values of entryName attributes of endpoint tag, in this case, are: server1, server2, server3. The property named "type", specifies what kind of backend, in this case is a MongoDB server.

Maven Coordinates

   <dependency>
     <groupId>org.gcube.contentmanagement</groupId>
     <artifactId>storage-manager-core</artifactId>
     <version>[2.0.0, 3.0.0)</version>
   </dependency>
   <dependency>
     <groupId>org.gcube.contentmanagement</groupId>
     <artifactId>storage-manager-wrapper</artifactId>
     <version>[2.0.0, 3.0.0)</version>
   </dependency>

Example

Scenario 1

In this example a GCUBE service s1 (identify by: ServiceClass: sc, ServiceName: sn), running in the GCUBEScope /gcube upload the file: /home/xxx/FilePerTest/test.jpg in shared mode and another instance of the same service, s2 (identify by: ServiceClass: sc2, ServiceName: sn2) running in the same scope but on another node, download it.

The following code shows the file uploading operation:

GCUBEScope scope = GCUBEScope.getScope("/gcube");
 // The Access type can be private | shared | public
IClient client=new StorageClient("sc", "sn", "s1", AccessType.SHARED, scope).getClient();
String id=client.put(true).LFile("/home/xxx/FilePerTest/test.jpg").RFile("/img/test.jpg");

The following code shows the file downloading operation:

GCUBEScope scope = GCUBEScope.getScope("/gcube");
 // The Access type can be private | shared | public
IClient client=new StorageClient("sc2", "sn2", "s2", AccessType.SHARED, scope).getClient();
String id=client.get().LFile("/home/yyy/FilePerTest/test.jpg").RFile("/img/test.jpg");

The file is downloaded in the local directory: /home/yyy/FilePerTest/ where s2 is running and the file name is test.jpg


Scenario 2

In this example a GCUBE service s1 (identify by: ServiceClass: sc, ServiceName: sn), running in the GCUBEScope /gcube uploads the file: /home/xxx/FilePerTest/test2.jpg in shared mode, retrieve the url of the file uploaded by the method getUrl and passes it to another service s2 that is in running on another node. The s2 service retrieves the inputStream of the file that the service s1 has uploaded by the url:

IClient client=new StorageClient("sc", "sn", "s1", AccessType.SHARED, scope).getClient(); 
String id=client.put(true).LFile("/home/xxx/FilePerTest/test.jpg").RFile("/img/test1.jpg"); 
String url=client.getUrl().RFile("/img/test1.jpg"); 

The s2 service retrieves the inputStream of the file by the url stored in the "url" variable:

Handler.activateProtocol();
URL smsHome = new URL("smp://img/test1.jpg5ezvFfBOLqbElsh5xYVgBTxt8lwt4fl");
URLConnection uc = null;
uc = ( URLConnection ) smsHome.openConnection();
InputStream is=uc.getInputStream();

VOLATILE area

Since 1.0.1 version of storage-manager there is a new storage area called VOLATILE. The files inserted on this special area are automatically removed after a period of 7 days from the first upload. The use of this area is intended for all clients that need to use storage for temporary data. The use of VOLATILE area is very simple and very similar to the standard usage of storage-manager. The only difference is that using the VOLATILE area the client must add an input parameter on the storage-manager client constructor. The new input parameter is an enumerated type and is called MemoryType, it can have only 2 values:

  • VOLATILE: for storing in the VOLATILE area: the files are deleting automatically after 7 days
  • PERMANENT: for storing in the standard area: the files are deleting only manually

The VOLATILE area is suggested to all clients that do not need to maintain data on the storage permanently. Through the use of this area the client doesn't need to perform the delete operation manually. It is strongly discouraged the usage of the VOLATILE storage area for storing data that will need to last more than seven days.


Scenario 1

In this example a GCUBE service s1 (identify by: ServiceClass: sc, ServiceName: sn), running in the GCUBEScope /gcube upload the file: /home/xxx/FilePerTest/test.jpg in shared mode in the VOLATILE area. Another instance of the same service, s2 (identify by: ServiceClass: sc2, ServiceName: sn2) running in the same scope but on another node, download it.

The following code shows the upload operation:

GCUBEScope scope = GCUBEScope.getScope("/gcube");
IClient client=new StorageClient("sc", "sn", "s1", AccessType.SHARED, scope, MemoryType.VOLATILE).getClient();
String id=client.put(true).LFile("/home/xxx/FilePerTest/test.jpg").RFile("/img/test.jpg");

The following code shows the download operation:

GCUBEScope scope = GCUBEScope.getScope("/gcube");
IClient client=new StorageClient("sc2", "sn2", "s2", AccessType.SHARED, scope, MemoryType.VOLATILE).getClient();
String id=client.get().LFile("/home/yyy/FilePerTest/test.jpg").RFile("/img/test.jpg");

The file is downloaded in the local directory: /home/yyy/FilePerTest/ where s2 is running and the file name is test.jpg

Scenario 2

In this example a GCUBE service s1 (identify by: ServiceClass: sc, ServiceName: sn), running in the GCUBEScope /gcube uploads the file: /home/xxx/FilePerTest/test2.jpg in shared mode in VOLATILE area. It then retrieves the url of the file uploaded by the method getUrl and passes it to another service s2 that is in running on another node. The s2 service retrieves the inputStream of the file that the service s1 had uploaded by the url. Remember, getUrl() method is not compatible with private access mode:

IClient client=new StorageClient("sc", "sn", "s1", AccessType.SHARED, scope, MemoryType.VOLATILE).getClient(); 
String id=client.put(true).LFile("/home/xxx/FilePerTest/test.jpg").RFile("/img/test1.jpg"); 
String url=client.getUrl().RFile("/img/test1.jpg"); 

The s2 service retrieves the inputStream of the file by the url stored in the "url" variable:

Handler.activateProtocol();
URL smsHome = new URL("smp://img/test1.jpg5ezvFfBOLqbElsh5xYVgBTxt8lwt4fl");
URLConnection uc = null;
uc = ( URLConnection ) smsHome.openConnection();
InputStream is=uc.getInputStream();

storage-manager v 2.0.0

The version 2.0.0 of storage-manager is based on FWS. With the version 2.0.0, the scope is not passed as constructor argument anymore. Rather, the scope must be setted before the instantiation of the client. This is the most important change that affects the clients exploiting it.

The following code shows an example of download operation:

String scope="/gcube/devsec";
ScopeProvider.instance.set(scope);
IClient client=new StorageClient("sc", "sn", "own", AccessType.SHARED, MemoryType.VOLATILE).getClient();
String id=client.get().LFile("/home/yyy/FilePerTest/test.jpg").RFile("/img/test.jpg");

where "sc" is the serviceClass, "sn" is the serviceName, "own" is the owner name.


Furthermore, version 2.0.0 includes a new client constructor that avoids the specification of ServiceClass and ServiceName. Default values will be automatically taken in this case.

The following code shows an example:

String scope="/gcube/devsec";
ScopeProvider.instance.set(scope);
IClient client=new StorageClient("own", AccessType.SHARED, MemoryType.VOLATILE).getClient();

where "own" is the owner name.


Media:Example.ogg