Streaming access to Minos Enstore files via xrootd


Background

From the Fermilab cluster, one can stream Minos data files directly from DCache disk to loon or root sessions using installed dcap software and the TDCacheFile "plugin" provided with ROOT. The file will look like a local root file to loon or root, when opened with:

Loon job

loon myscript.C "dcap://fndca1.fnal.gov:24125/pnfs/fnal.gov/usr/minos/fardet_data/2003-01/F00011732_0000.mdaq.root"

ROOT session

root [19] TFile *myfile=TFile::Open("dcap://fndca1.fnal.gov:24125/pnfs/fnal.gov/usr/minos/fardet_data/2003-01/F00011732_0000.mdaq.root");

This is practical as well from other computers located at Fermilab, provided the dcap software has been installed on them. However, dcap is not built on "unsupported" platforms, such as MacOSX. At remote sites it is often difficult to use dcap. In particular, the firewalls at most sites do not permit a dcap connection to be established with Fermilab.

At sites away from Fermilab, when one needs to read a Minos data file repeatedly, one is most likely to copy the file from Fermilab to a local disk at the remote site. However, it is sometimes appropriate to read the file in streaming mode over the network from Fermilab. This is true when one is reading a file only once, when one wishes to read only part of a file (as in debugging or event display), when one does not have enough bandwidth to copy a multi-megabyte file from Fermilab to the remote computer or when one doesn't have enough local disk space at the remote site for an entire data file.

ROOT provides an alternative mechanism to dcap for streaming access to enstore data. It involves an xrootd server, kept running at Fermilab, to serve Minos data files to clients running root or loon sessions. This extends the functionality of dcap-like streaming to clients without dcap access. The xrootd server machine is addressed via a Fermilab alias minos-xrootd.fnal.gov, obviating the need to specify a concrete node name.


Using XROOTD with ROOT/Loon

One can access Minos enstore files from a loon job or from a root prompt:

Loon job

loon myscript.C "root://minos-xrootd.fnal.gov//stage/F00011732_0000.mdaq.root"
or:

JobC j;
j.Input.AddFile("root://minos-xrootd.fnal.gov//stage/F00032605_0000.spill.cand.R1_18.0.root");

ROOT session

root [19] TFile *myfile=TFile::Open("root://minos-xrootd.fnal.gov//stage/F00011732_0000.mdaq.root");

Note that one does not prefix the file name with the PNFS path. Instead, one uses the prefix "//stage/". This signals to the XROOTD server that it is to "stage" the file from enstore to a cache disk on minos-xrootd before serving the data to the remote client. The xrootd server queries the Minos sam database to determine whether the file exists in the enstore system. Files not yet declared to sam are identified through a text file catalog of /pnfs/minos contents, which is refreshed daily.

As with dcap, if the file is already on the stage disk, then the job starts immediately. If it is only on tape, then the file will be staged and the job will wait until this is complete before it starts to read the file. As with dcap, one should not list a large number of files on the loon command line because Loon tries to open them all at the same time. Instead, one should use the AddFile syntax in JobControl. One should also include the streams to read; otherwise Loon tries to open all the files to find the available streams. See the offline manual for information about Job Control.


Caveats

There are a couple of caveats:

a) There is a period of very old ROOT versions which don't work with
   xrootd and loon.  One should use either ROOT 4.04/02n (n=any) or
   ROOT v5, starting Sept 6, 2005.  Earlier versions of
   ROOT v5 don't work properly with xrootd in the loon context.

b) Use Loon development updated after September 25, 2005, or with the
   latest version of Persistency/PerFile.cxx (CVS revision >= 1.17).

Questions to: George Irwin(gmieg@slac.stanford.edu)
Last modified: Monday, November 14, 2005