Archiving MINOS Data in the Tape Robot
As part of the ongoing effort to outsource MINOS MonteCarlo production to clusters located at the member institutions and in order to simplify institutions participation, it is useful to establish a unified approach to store the resulting output in the Fermilab tape robot and make the data available to the collaboration at large. The following set of instructions should enable your site to archive produced data into the tape robot. It is assumed your archiver machine (e.g. your cluster head node) has kerberos client applications installed, namely the krb5-workstation and krb5-ftp packages, and some flavor of Python installed. You should also hold an account on the MINOS Linux Cluster at FNAL.
Installing Fermilab's kftp package
kftp was written at Fermilab by Igor Mandrichenko in Python. It includes a GSS-kerberized FTP client Python module and simple command line ftp client programs for transferring files and listing directories.
Obtaining the binaries
To install kftp at your home institution, download the following tar files containing the binaries:
The GSS module should be compatible with your python version. The link above works for python 2.3. If you have an earlier version installed, look in here and download the appropriate version.
To be able to run the kftp programs you should unwind the tar files in their respective directory structures:
where $KFTP_HOME points to any path of your choice.
Finally, you need to make your environment aware of kftp. To do so, just modify and source the configuration script provided here.
At the end of the script you can find an example usage line. You can learn more about kftp by looking at the $KFTP_DIR/doc/README file.
Obtaining write permissions for Tape Robot access
Before being able to use kftp to transfer files to ENSTORE, you will need to obtain writing permissions for your archiver machine. The recommended way to do so is to request a kerberos project principal and create a keytab file for that project principal.
Requesting a project principal
To obtain your kerberos project principal you should e-mail Liz Buckley-Geer stating your reasons to request writing privileges into ENSTORE.You also need to include the DNS hostname of your archiver machine and the account name that will be doing the transfers, which has to match the name of a MINOS user account on the Linux Cluster.
Pending approval of your request, Liz will ask FNAL's Computing Security Team for the principal. After a few days, you should be sent your principal along with a one-time use password. The principal should look something like
You can then proceed to the next step:
Generating a keytab file for a project principal
The keytab file is necessary to create kerberos tickets that give you permissions to write into the tape robot.
Keep your keytab in a reasonably safe location in a local file system not shared over a network (i.e. not on AFS space or any NFS mounted storage). A possible location would be $KFTP_HOME/.krb5, if $KFTP_HOME complies with the aforementioned requirements. The keytab file must be readable only by the user account that will generate the kerberos tickets and perform the data storage.
You should verify you have the command "kadmin" available. Sometimes it will be located in /usr/sbin/, which is normally not in a user's PATH.
Here is an example of the command that creates a keytab file for the principal mcarchiver/minos/cluster01.tccs.tufts.edu@FNAL.GOV
/usr/sbin/kadmin -r FNAL.GOV \ -p mcarchiver/minos/cluster01.tccs.tufts.edu@FNAL.GOV \ -q "ktadd -k /home/testbed/asousa01/.krb5/mcarchiver.keytab \ mcarchiver/minos/cluster01.tccs.tufts.edu@FNAL.GOV"
After issuing the command corresponding to your principal, enter the provided password and your keytab will be created. Note that after executing the "kadmin" command, the password is no longer valid for that principal (it can be used only once to generate the keytab file).
More detailed information on creating a keytab file is available in this page.
Generating kerberos tickets
With the keytab file in place, the last step before file transfer is to generate kerberos tickets for your project principal. This is done with a command such as:
kinit -A -k -t /home/testbed/asousa01/.krb5/mcarchiver.keytab \ mcarchiver/minos/cluster01.tccs.tufts.edu@FNAL.GOV
You can check that you got the proper tickets via the "klist" command.
Using kftp for file transfers
Before transferring any files, a directory must be created to hold them in "/pnfs/minos". If you have an account on the MINOS Linux Cluster, you already have write permissions in "/pnfs/minos". Supposing you created a directory "mydir", you can now easily look at the contents of that directory from your archiver machine by issuing:
And perform a transfer from the current directory in your local machine by doing:
The "-p 24127" option tells kftp to connect to port 24127, the only port that allows writing, and the "-m p" switch defines passive mode behavior, as opposed to "-m a", useful if your archiver is behind a firewall. Your file transfers can be monitored at the ENSTORE page, more specifically in the dCache transfer summary page. If all is working up to here, you should be in position to use the "kftpcp" command in the output stage of your MC generation jobs to do automated transfers to ENSTORE or alternatively archive large quantities of data that you wish to make available to the whole MINOS Collaboration.
Send suggestions or comments to - The Pagemaster