MINOS Data Handling Advice

The MINOS Data Handling project uses the Enstore system for permanent data storage (tape), and DCache for working copies on disk.

The BlueArc NFS served areas are mounted as /minos/data and /minos/scratch. These volatile working areas are not backed up.

BLUEARC LOCKING 2009/05/04:

Starting on 2009 April 01, Minos batch jobs accessing /minos/data files have repeatedly overloaded BlueArc file servers, causing global Fermigrid outages. Until the root causes are understood, and we know the limits of the system, we must serialize our Bluearc reads and writes. We have provided a lock file and a script for this purpose.

All batch jobs must use /grid/fermiapp/minos/scripts/cp1 to move files to or from the /minos/data or /minos/scratch areas.
Batch jobs must not access the Bluearc areas directly.

Grid jobs should use the dynamic local area ${_CONDOR_SCRATCH_DIR}/work .
You must mkdir ${_CONDOR_SCRATCH_DIR}/work .
DO NOT put file directly into ${_CONDOR_SCRATCH_DIR}, as these will be copied back to /tmp on minos25, filling the disk and causing global failures.

CP1=/grid/fermiapp/minos/scripts/cp1
mkdir -p ${_CONDOR_SCRATCH_DIR}/work
${CP1} /minos/data/beam_data/2007-07/B070717_160001.mbeam.root ${_CONDOR_SCRATCH_DIR}/work/

This is a crude, single threaded lock at present.
The lock file is /minos/data/LOCK1.
A lock history is kept in /minos/data/LOCKS.

Several heavy users are already using this and related scripts. We will make locking mandatory very soon.

Enstore characterisics that you should know :

Target files to be at least 1 GB in size.

Tape drive per-file overheads are about 200 MBytes per file, and about 5 seconds of tape positioning time.

Target under about 100 files per directory

There are PNFS backend database overheads of over .1 second per file probed.

The tape operational lifetime is about 2000 mounts by Fermilab policy. The vendor limit is 4000. When a tape has been mounted 2000 times, the administrative staff take the tape offline, and move the files to new media., discarding the old tape. The default --delayed-dismount parameter for encp is around 30 seconds, and can not be set over 10 minutes. So copying files in directly with encp every 15 minutes is deadly.

Therefore, use dccp to read files from disk, never use encp to read directly from tape.

Likewise, write data using dccp , so that you will not have to wait for a tape mount. The general write pools flush data to tape about every 4 hours.

Reading is generally done without authentication, setup dccp -q unsequred.

The farms are writing via Grid certificates, by doing setup dccp -q x509.

When writing data, use File Families to specify dedicated sets of tapes,keeping related data together.

Security, Privacy, Legal