CCP4-JHBuild is JHBuild customized for building CCP4.
JHBuild is a Python program that can pull a number of modules from a variety of sources (CVS, Subversion, Bazaar, tarballs...) and build them in the correct order. It was originally developed to build GNOME, but now is also used for a few other projects.
Modules (packages) are downloaded, patched if necessary, configured, built and installed, one by one.
For example, when one asks to build Refmac:
$ ./cj build refmac
first its dependencies are downloaded and built (Lapack is downloaded from Netlib, CCP4 libraries from Bazaar) and finally Refmac is fetched from the Refmac website and built. The user can skip compilation of Lapack if it is already installed.
New packages that use popular build systems like autotools or cmake can be easily added by specifying the download location and dependencies in the moduleset file.
CCP4-JHBuild is intended for developers, but can also be used by users who are familiar with compiling software and want to test new features.
Prerequisites:
On Linux, here are corresponding commands for Debian/Ubuntu, Fedora/RHEL, and Suse:
apt-get install bzr cmake wget g++ gfortran m4 libqt4-dev python-dev \
default-jdk libxml2-dev
yum install bzr cmake wget gcc-gfortran gcc-c++ m4 qt-devel python-devel \
java-devel libxml2-devel
zypper install bzr cmake wget gcc-fortran gcc-c++ m4 qt-devel python-devel \
java-devel libxml2-devel
On OSX, either install separately C, C++ and Fortran compilers, bzr, CMake, Qt4.8 or use package manager such as Homebrew or MacPorts.
On Windows MinGW/MSYS is recommended, see the Windows section below.
In the Harwell Campus the proxy must be configured:
$ export http_proxy=http://wwwcache.rl.ac.uk:8080/ $ export no_proxy=ccp4.ac.uk,rc-harwell.ac.uk
Download "devtools" from bzr:
$ bzr co http://fg.oisin.rc-harwell.ac.uk/anonscm/bzr/devtools/trunk devtools $ cd devtools/
Check if jhbuild works (list default modules):
$ ./cj list
It's also possible to work in a different directory and call cj using full path.
Before you start building CCP4 you may want to change some options in the conf.py configuration file. Some options can be also set through environmental variables. Read the conf.py file for details. In particular, if you have multiple compilers installed, choose which one should be used.
If you can't install cmake 2.8.7+ system-wide, build it locally:
$ ./cj build cmake
To use Qt installed in a non-default location, set $QTDIR.
The Tcl library and headers are needed for building ccp4mapwish and (optionally) diff-image. Although any Tcl version (8.4 - 8.6) would do, we use 8.4, because that is the only version iMosflm works with. Since iMosflm and ccp4i require several Tcl/Tk libraries, we've prepared a bundle with all of them for easy building:
$ # on Unix Xlib headers must be installed (e.g. libX11-dev on Debian) $ curl -O http://devtools.fg.oisin.rc-harwell.ac.uk/files/tcltk++_8.4.19.2.tar.gz $ tar xzf tcltk++_8.4.19.2.tar.gz # (90MB unpacked) $ tcltk++_8.4.19.2/build.sh TclTk84 # the argument is dir for installation
If you have some of the popular libraries used by CCP4 (zlib, libxml2, libjpeg, LAPACK) already installed, add them to the skip list in conf.py to not waste time on building them from sources.
Then you may build selected module(s) with dependencies:
$ ./cj build refmac
or default set of modules (packages that don't require Tcl, Qt, Java):
$ ./cj build
or everything:
$ ./cj build all
The list command shows what packages would be built:
$ ./cj list refmac $ ./cj list default $ ./cj list all
and the info command shows additional info:
$ ./cj info refmac
The build command first checks for updates in the repository and then builds the program. You can separate the two steps:
$ ./cj update [program] $ ./cj build -n [program] # -n == --no-network
Other useful build options are -f (forces build even if the module has not been updated) and -c (runs make clean before make).
There are also equivalents of make clean and make uninstall:
$ ./cj clean [program] $ ./cj uninstall [program]
To build a module without dependencies, replace build with buildone. The same with update/updateone and clean/cleanone.
The build command creates three directories (checkout, build and install) in the current directory. They are used for, respectively, downloads, building and installing. These locations can be changed in conf.py (variables checkoutroot, buildroot and prefix).
Other settings in conf.py include compiler and linker flags, number of parallel jobs (jobs=N) and options for individual programs (like Tcl version or OpenMP support). These settings can be changed either in conf.py or in $HOME/.cjrc.
All JHBuild commands can be listed with:
$ ./cj help
Some of them do not work with CPP4 module set yet.
Option --help shows details about each command:
$ ./cj build --help
Some CCP4 programs expect data in lib/ and lib/data/ directory. In Linux Filesystem Hierarchy Standard as well as in BSD/Darwin hierarchy architecture-independent data go to share/ (see man hier). We gradually move in this direction. For now some symbolic links may be needed to make all programs work:
$ ./symlinks.sh install-dir
You can run the newly compiled version without uninstalling other CCP4 versions and without setting any variables, but it must be started using:
$ ./run-ccp4.py install-dir
This script internally changes environmental variables (without affecting other CCP4 versions) and offers to start either ccp4i or a new shell.
Update devtools, then update and compile modules:
$ bzr update $ ./cj build
It is possible to compile the suite on Windows or to cross-compile it for Windows on other system. In both cases we use MinGW (port of GCC).
C library used by MinGW has no regex. Regex is needed for the ccif library. mingw-libgnurx is a possible replacement.
Before compiling cctbx python extensions with MinGW make sure you have so-called import library for pythonNN.dll. Programs that use cctbx from C++ (like pointless) don't need Python extensions. To skip building the extensions (and reduce compilation time) uncomment --build-boost-python-extensions=False in conf.py.
Bash or similar shell with basic Unix tools is needed to run autoconf-generated configure scripts. Below we describe three setups.
One way to install MinGW with MSYS is using mingw-get-inst.
Install necessary packages:
$ mingw-get install mingw32-gcc mingw32-gcc-fortran mingw32-gcc-g++ \
msys-make msys-wget mingw32-libz
mingw-libgnurx needs to be installed manually. After unpacking .h file goes to include under C:\MinGW, .a to lib and .dll to bin.
Outside of MSYS install CMake, Python 2.7 and Bazaar (Python-based installer will do, but for 64-bit Python see recent comments here). Make sure all three are in PATH, e.g. add to ~/.profile:
PATH=$PATH:/c/Python27:/c/Python27/Scripts:"/c/Program Files (x86)/CMake 2.8/bin" export PATH
Note
MSYS shell allows to specify paths in different ways. For example /home, /c/MinGW/msys/1.0/home, C:/MinGW/msys/1.0/home and C:\MinGW\msys\1.0\home point to the same directory. MSYS is converting paths that are passed between MSYS and non-MSYS programs, using heuristics to decide which strings are paths. Since paths are passed a lot between shell (MSYS), Python (non-MSYS), CMake (non-MSYS), make (MSYS), compilers (non-MSYS) and other programs during the build, the whole thing is fragile.
In default MinGW installation user's home directory is in C:\MinGW\msys\1.0\home\username. In our setup all directories used by the build system are in our home directory. Other configurations have not been tested much. It was spotted that make/libtool fail if checkoutroot is outside of the MinGW directory (C:/MinGW by default).
When environment variables go through Python they are converted to uppercase, and wget ignores uppercase *_PROXY variables, so if you are behind proxy you need to additionally configure it in ~/.wgetrc. In Harwell add lines:
http_proxy=http://wwwcache.rl.ac.uk:8080/ no_proxy=ccp4.ac.uk,rc-harwell.ac.uk
Thanks to automatic path conversion in MSYS, it is even possible to run unix tests.
Cygwin provides more tools and is more convenient for debugging. Cctbx does not compile under cygwin (I don't have time to work on it). All other modules can be compiled.
Native Windows programs can be built under Cygwin using MinGW (cross-)compiler, in a similar way as on Linux (see the next section):
$TARGET=i686-w64-mingw32 ./cj build clipper-progs
Most of Linux distribution have MinGW already packaged (note that g++ and gfortran are also needed).
To easily cross-compile Tcl extensions, download Tcl/Tk 8.4 compiled on Windows (the same that is distribued with CCP4).
To build ccif, install mingw-libgnurx library (on Fedora and OpenSuse in mingw32-libgnurx* RPM).
Use the same commands as for native compilation, but set environmental variable $TARGET to the prefix of cross-compiler, e.g.:
$ TARGET=i686-pc-mingw32 ./cj build
By default, directories for building and installing are build-$TARGET and install-$TARGET.
missing bits
Cross-compilation of cctbx python extensions is not supported, so we automatically set --build-boost-python-extensions=False flag when cross-compiling (see conf.py). Fortunately, these extensions are not needed to build pointless, aimless and phaser.
Unless you have configured wine and binfmt_misc, ccif build script will not generate automatically cif_mmdic.lib. It can be generated later on Windows:
cifdic_to_symtab %CCP4%\share\ccif\cif_mm.dic %CLIB%\ccp4\cif_mmdic.lib 199
If multiple compilers are installed, autoconf and cmake may have different preferences. Mixing compilers may cause errors. In such case choose compilers explicitly in the compilers section in conf.py.
If phaser compilation fails (e.g. when compiling src/ListEditing.cc) with internal compiler error, either use newer compiler or comment out the --enable-openmp-if-possible=True option.
If cmake gives a warning about libxml2.so.2 version, probably you have built libxml2 (rapper dependency) and cmake is indirectly (via libarchive) linked to another libxml2 version. While this warning should not cause any harm, it can be avoided by using libxml2 from the system instead of building it. Set the skip option in conf.py and do ./cj uninstall libxml2.
Each change in a library triggers make install, which reinstalls also not modified headers, changing their timestamps and triggering not necessary recompilations. To avoid it, jhbuild has a small wrapper around the install program. This wrapper, called install-check, uses cmp to compare headers before calling install. It works only on Unix. To enable it, compile the wrapper in devtools:
cc -O2 install-check.c -o install-check