When the submit host and the execution host do not have access to a common, shared file system or are simply using different root directories, the necessary job data must be copied (via rcp or a similar utility) to the execution host, and the job results must be copied back to the submit host.
Because such a file staging mechanism must be very flexible and fully configurable, it is desirable to let the actual copying be handled by the system itself, such as by a script written by the administrator, so that one can focus on just the necessary data. Because the actual work of copying is delegated to the system, i.e. the script, the term "delegated file staging" is used. With delegated file staging in the Grid Engine 6.0 release two scripts are needed. The first is called in the prolog and copies the input data to the selected execution host. The second is called in the epilog and copies the results to the submit host.
The scripts must, at a minimum, copy the stdin, stdout, and stderr files either from the submit host to the execution host or vice versa.
The additional needed information is obtained by the script from command-line parameters, which can be set in the cluster configuration using the "qconf -mconf" command. Under "prolog", after the path to the prolog script(s), the following variables can be given to be replaced automatically by the DRM with the appropriate, corresponding values when the prolog is called:
The values of these variables at the time the prolog is called will either be "1" or "0", according to whether file staging is enabled for that particular data type.
The values of these variables at the time the prolog is called will be the names of the hosts from which the data is to be copied.
The values of these variables at the time the prolog is called will be the path (relative to the respective hosts) to the data to be copied.
The values of these variables at the time the prolog is called will be the path to where the data is to be copied.
A typical call to a prolog script, i.e. what would appear in "qconf -mconf" would look like:
prolog /..../scripts/fs_prolog.sh $fs_stdin_file_staging \ $fs_stdin_host $fs_stdin_path $fs_stdin_tmp_path
and a typical call to an epilog script would look like:
epilog /..../scripts/fs_epilog.sh $fs_stdout_file_staging \ $fs_stdout_host $fs_stdout_path $fs_stdout_tmp_path \ $fs_stderr_file_staging $fs_stderr_host $fs_stderr_path $fs_stderr_tmp_path
The following must then appear in the prolog script:
... if( $1 ) then rcp $2:$3 $4 endif ...
and similarly in the epilog script:
... if( $1 ) then rcp $4 $2:$3 endif if( $5 ) then rcp $8 $6:$7 endif ...
Almost. The administrator must add the following entry to the cluster configuration:
Now delegated file staging is available for use.
From this point forward, we are talking specifically about using delegated file staging with DRMAA. For more information on DRMAA, see the DRMAA howto.
Because drmaa_transfer_files is an optional attribute in DRMAA, the first
thing an application must do is use drmaa_get_attribute_names() to
determine if the attribute is supported by the DRMAA implementation in
use. The DRMAA implementation provided with Grid Engine 6.0 only supports
the drmaa_transfer_files attribute when
true is enabled in the cluster configuration. (See the previous
If the drmaa_transfer_files attribute is available, the application can set this attribute's value to one of following: "i", "o", "e", or a concatinated combination of the three, such as "oi". Each letter, if present in the attribute's value indicated that the corresponding data source should be copied using delegated file tranferal.
Example scripts are available:
A general review of file staging can be found in the general file staging howto.