Database Distribution: Distributing Update Files using rsync

Where a Master and a Slave cannot access a common publish directory some mechanism has to be introduced to replicate the Master's publish directory on a disk which the Slave has access to. One method that appears to work well is rsync and this is the service currently used for replication. To resynchronise a local mirror:-

rsync -rvt user@publish_directory mirror

• -r recurse into directories (needed if specifying a directory)
• -v verbose so not really needed.
• -t use date and size to skip unchanged files (makes things much faster)

  minos-db1.fnal.gov::out

  rsync -rvt nwest@minos-db1.fnal.gov::out /userdisk/west/work/minos/rsync

Brett Viren suggests the following cron script. When run from cron it will be silent unless there is an error. On error it will speak up which will trigger cron to send you email.

An example crontab entry might look like (shown with maximum keystrokes to satisfy the cron neophytes)

        shell> crontab -e
        (... now type ...)
        # Run once a day at midnight
        0 0 * * * /path/to/dbm_update_from_cron.sh
        (... and save/quit editor ...)

#!/bin/sh
# dbm_update_from_cron.sh

# set to your user name, eg "bv"
user=SET_ME_TO_YOUR_USERNAME

# set to directory to dump the files ending in a "/"
# eg "/path/to/rsync/"
dest=SET_ME_TO_WHERE_FILES_GO__SHOULD_END_IN_A_SLASH


log=/tmp/dbm.rsync.log.$$
src=${user}@minos-db1.fnal.gov::out
rsync -rvt $src $dest > $log 2>&1
err=$?
if [ $err -ne 0 ] ; then
        echo "DBM rsync failed with error $err (see rsync man page)"
        cat $log
fi
#### end dbm_update_from_cron.sh

There is a recommendation on the scheduling of cron jobs. If the job involves access to AFS disks under KERBEROS control see Using KERBEROS.