MINOS Offline Documentation
[home] [installing MINOS software] [supported platforms] [package links] [mailing list] [HyperNews] [FAQ] [computing help] [MINOS glossary] [archives]

Doing parallel and caching builds with msrt

( [main msrt page] | [help output] | [msrt and your test releases] | [SRT_CXX] | [msrt and multiple compilers] | [msrt and parallel builds] )


Parallel builds can greatly speed up building and caching can additionally speed up re-building a code base. In the case of MINOSSOFT, SRT puts some constraints on parallel builds which may surprise the unwary. Simply doing "make -j 4" will not work. Support for parallel and caching builds has been put in to msrt after R1.12.

Builds can be made even faster by using more than one computer to do the build. Support for distcc, a distributed compiler wrapper, has been added to msrt. This works by passing the post-CPP processed file across the network to the compile node, which compiles it and returns the object code.

A caching build is supported by a compiler wrapper such as distcc's sister program ccache. This works by passing the post-CPP processed file to the cache. If the file has already been seen it will simply return the previously compiled version if not it will compile it (potentially via distcc) and store the result for future cache hits.

This document explains how to use these features with msrt. Also see this page for how to use distcc and ccache in the context of a test release.


This explains how to set up things to use distcc and/or ccache and SRT's parallel make mechanisms. They are all orthogonal so you can pick and choose what features you want. While it is rather pointless to use distcc but not turn on parallel builds, the reverse is not true (assuming a multi-cpu system). Likewise, ccache can help re-builds with or without parallel builds.
  1. Get the source for distcc and/or ccache from the above links. They are packaged for Debian and possibly for your distribution as well. For distcc you will need to run the distccd daemon on every node that will accept compilation jobs. Each node should be configured with a list of IP numbers from which it will accept jobs.
  2. Set up needed environment variables:
  3. Set up msrt. This can be done on a per invocation basis or for all invocations by editing the users ~/.msrtrc file. The command line options are:
        -d       (use_distcc)    use distcc
        -c       (use_ccache)    use ccache
        -j       (max_make_jobs) set max make jobs (ie make -j XX)
    The ~/.msrtrc file variables (in "()" above) are described:
    # Whether or not to also use ccache.  Use of this is independent from
    # parallel build settings below.  This will speed up re-building of
    # files that were built previously (via ccache) but haven't changed,
    # ie. a "msrt clean && msrt build" command chain.
    # Whether or not to also use distcc.  You should externally set
    # DISTCC_HOSTS to the list of hosts to use.  If not set MSRT will set
    # it to localhost, which pretty much negates any speed up.  If you set
    # this then do NOT set CCACHE_PREFIX=distcc.
    # Max number of make threads allowed.  A good choice is the number of
    # CPUs available.  If this is >1 you should externally set either
    # PARALLEL or PARALLEL_TOP, or both (any value will do).  If neither
    # are set, MSRT will set both.  You may also want to set MAXLOAD to
    # limit how much the local host gets hammered.

Invocation of msrt

Invoke msrt as usual unless you elected to use the command line options to set the above msrt variables.

Monitoring distcc parallel builds

Distcc provides a text based (distccmon-text) and GUI based (distccmon-gnome) monitor which will show what hosts are compiling what files and what their level of activity is.

Some useful ccache related notes

Ccache provides some statistics (ccache -s) of the cache. It also lets you clear the cache (ccache -C) which is useful when getting things to work as you can force a compilation. This is particularly useful in conjunction with watching distccmon-* output.


Please contact me, bv@bnl.gov, with any problems, suggestions or comments.

Last Modified: $Date: 2004/12/15 20:05:58 $
Contact: bv@bnl.gov
Page viewed from http://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/msrt-parallel.html
Security, Privacy, Legal Fermi National Accelerator Laboratory