Manage data in the Grid using GridFTP

GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. The GridFTP protocol is based on FTP, the highly-popular Internet file transfer protocol. We have selected a set of protocol features and extensions defined already in IETF RFCs and added a few additional features to meet requirements from current data grid projects.

Globsu white paper, GridFTP: Universal Data Transfer for the Grid , describes the motivations behind the use of GridFTP

GridFTP provides the following protocol features.

• GSI security on control and data channels
• Multiple data channels for parallel transfers
• Partial file transfers
• Third-party (direct server-to-server) transfers
• Authenticated data channels
• Reusable data channels
• Command pipelining

The low-level tools can be used to allow users to perform actions an the GridFTP server of a SE.
There are 7 low level commands. All 7 commands shall be covered in this section.

edg-gridftp-* commands

All the edg-gridftp-* commands take as a parameter one or more URL with the following format:

protocol://hostname(:port)/filepath

where

• protocol refers to the protocol to access the SE (gsiftp generally)
• hostname refers to the host name of the SE you want to access to
• port, optional, refers to the listening socked port of the GridFTP server. The default one is 2811
• path, the filepath to the directory of file one wishes to peruse

Here an example of a real URL:

gsiftp://aliserv6.ct.infn.it/tmp/tony/bigfile.dat

edg-gridftp-exists command

edg-gridftp-exists determines if the file or directory associated with the given URL exists

The command is as follows:
$ edg-gridftp-exists URL [URL ...]

This command will return a status of 0 if the file or directory exists. In all other cases, it will return a non-zero value and print an error message to the standard error.

If more than one URL is given, then a successful return will only occur if all of the URLs exist. The processing will stop with the first failure.

_Try out: Check the existence of a directory_

[morgan@localhost ~]$ edg-gridftp-exists    gsiftp://grid2.fe.infn.it/tmp/.font-unix
[morgan@localhost ~]$ echo $?
0

edg-gridftp-ls

edg-gridftp-ls will list (to the standard output) a file or the contents of a directory for the given URL.

The command is as follows:

edg-gridftp-ls [--proxy=proxy] [--verbose] [--noauth] URL

It will return a zero status on success. In all other cases, it will return a non-zero value and print an error message to the standard error.

[morgan@localhost ~]$ edg-gridftp-ls  gsiftp://grid2.fe.infn.it/tmp
.
..
crlpem-B10253
crlpem-u27893
crlpem-T16248
crlpem-v22698
[...]

edg-gridftp-mkdir

edg-gridftp-mkdir creates a directory on a GridFTP server

The command is as follows:

edg-gridftp-mkdir [--proxy=proxy] [--parents] URL [URL ...]

This command will return a status of 0 if the directory was created successfully. In all other cases, it will return a non-zero value and print an error message to the standard error.

If the --parents option is given any parent directories of the given URL will also be created.

If more than one URL is given, then the URLs are processed sequentially. The processing will stop with the first failure.

globus-url-copy

This command allows to transfer files to, from and between GridFTP servers.
It handles local-to-remote, remote-to-local, and remote-to-remote (third party) transfers.

The simplest way to use this command is as follows:

globus-url-copy [-vb] sourceURL destURL

When the source or the destination file is local, you have to use =*file://*= protocol. Full path have to be given.
The -vb_ flag is used to display, during the transfer, the number of bytes transferred and the transfer rate per second.

[morgan@localhost ~]$ globus-url-copy -vb  file:/home/morgan/delme.txt gsiftp://grid2.fe.infn.it/tmp/delme
Source: file:/home/morgan/
Dest:   gsiftp://grid2.fe.infn.it/tmp/
  delme.txt  ->  delme
            6 bytes         0.00 MB/sec avg         0.00 MB/sec inst
[morgan@localhost ~]$ edg-gridftp-ls gsiftp://grid2.fe.infn.it/tmp/delme --verbose
-rw-r--r--   1 lanck002   planck            6 Aug 18 19:16 delme

globsu-url-copy supports a multichannel file transfer that optimize the file transfer on WAN. Using -n option we set the number of parallel transfer channels.

[morgan@localhost ~]$ globus-url-copy -vb -p 16 file:/home/morgan/delme.txt gsiftp://grid2.fe.infn.it/tmp/delme2
Source: file:/home/morgan/
Dest:   gsiftp://grid2.fe.infn.it/tmp/
  delme.txt  ->  delme2
            6 bytes         0.00 MB/sec avg         0.00 MB/sec inst

globsu-url-copy supports third party transfer

[morgan@localhost ~]$ globus-url-copy -vb -p 16 gsiftp://grid2.fe.infn.it/tmp/delme gsiftp://gridse.ilc.cnr.it/tmp/delme
Source: gsiftp://grid2.fe.infn.it/tmp/
Dest:   gsiftp://gridse.ilc.cnr.it/tmp/
  delme
            6 bytes         0.00 MB/sec avg         0.00 MB/sec inst
[morgan@localhost ~]$ edg-gridftp-ls --verbose  gsiftp://gridse.ilc.cnr.it/tmp/delme
-rw-r--r--   1  inaf008     inaf            6 Aug 18 19:19 /tmp/delme

edg-gridftp-rm

edg-gridftp-rm removes the file from a GridFTP server.

The command is as follows:

edg-gridftp-rm [--proxy=proxy] URL [URL ...]

This command will return a status of 0 if the file was successfully removed. In all other cases, it will return a non-zero value and print an error message to the standard error.

If more than one URL is given, then a successful return will only occur if all of the URLs were successfully deleted. The processing will stop with the first failure.

[morgan@localhost ~]$ edg-gridftp-rm gsiftp://grid2.fe.infn.it/tmp/delme 
[morgan@localhost ~]$ edg-gridftp-rm gsiftp://gridse.ilc.cnr.it/tmp/delme

--+++ edg-gridftp-rename

edg-gridftp-rename renames a file or directory on a GridFTP server.

The command is as follows:

edg-gridftp-rename [--proxy=proxy] SourceURL DestinationURL

Everything but the filename is ignored on the destination URL meaning that files will not be moved between servers. This command will return a status of 0 if the file or directory was renamed. In all other cases, it will return a non-zero value and print an error message to the standard error.

edg-gridftp-mkdir

edg-gridftp-mkdir creates a directory on a GridFTP server

The command is as follows:

edg-gridftp-mkdir [--proxy=proxy] [--parents] URL [URL ...]

This command will return a status of 0 if the directory was created successfully. In all other cases, it will return a non-zero value and print an error message to the standard error.

If the --parents option is given any parent directories of the given URL will also be created.

If more than one URL is given, then the URLs are processed sequentially. The processing will stop with the first failure.

NOTE: The creation is only attempted if the URL does not already exist. If the given file is a regular file, this command will return success.

edg-gridftp-rmdir

edg-gridftp-rmdir removes the directory from a GridFTP server.

The command is as follows:

edg-gridftp-rmdir [--proxy=proxy] URL [URL ...]

This command will return a status of 0 if the directory was successfully removed. In all other cases, it will return a non-zero value and print an error message to the standard error.

If more than one URL is given, then a successful return will only occur if all of the URLs were successfully deleted. The processing will stop with the first failure.

The uberftp client

uberftp is a GridFTP -enabled client that supports both interactive use and FTP commands on the uberftp command line to transfer files between two computers.
It provides many commands similar to the standard FTP clients (cat, cd, get, put, mget, mput, dir, ls, mkdir).

[morgan@localhost ~]$ uberftp grid2.fe.infn.it
220 grid2.fe.infn.it GridFTP Server 2.3 (gcc32dbg, 1144436882-63) ready.
230 User planck002 logged in.
uberftp> ls
-rw-r--r--    1  lanck002  planck   5619 May 20 12:01  .canna
drwxr-xr-x  824      root    root  20480 May 20 12:02  ..
-rw-r--r--    1  lanck002  planck    124 May 20 12:01  .bashrc
-rw-r--r--    1  lanck002  planck    658 May 20 12:01  .zshrc
drwx------    2  lanck002  planck   4096 May 20 12:01  .
-rw-r--r--    1  lanck002  planck    120 May 20 12:01  .gtkrc
-rw-r--r--    1  lanck002  planck    191 May 20 12:01  .bash_profile
-rw-r--r--    1  lanck002  planck     24 May 20 12:01  .bash_logout
-rw-r--r--    1  lanck002  planck    383 May 20 12:01  .emacs
uberftp> 
uberftp> put delme.txt  
delme.txt:  6 bytes in 0.26 seconds. 0.02 KB/sec
uberftp> ls
-rw-r--r--    1  lanck002  planck   5619 May 20 12:01  .canna
drwxr-xr-x  824      root    root  20480 May 20 12:02  ..
-rw-r--r--    1  lanck002  planck    124 May 20 12:01  .bashrc
-rw-r--r--    1  lanck002  planck    658 May 20 12:01  .zshrc
drwx------    2  lanck002  planck   4096 Aug 18 19:54  .
-rw-r--r--    1  lanck002  planck    120 May 20 12:01  .gtkrc
-rw-r--r--    1  lanck002  planck      6 Aug 18 19:54  delme.txt
-rw-r--r--    1  lanck002  planck    191 May 20 12:01  .bash_profile
-rw-r--r--    1  lanck002  planck     24 May 20 12:01  .bash_logout
-rw-r--r--    1  lanck002  planck    383 May 20 12:01  .emacs
uberftp> rm delme.txt
uberftp> 

-- TaffoniGiuliano - 18 Aug 2008