24 Feb 2014

Current update preparation procedure

This document reflects the current state of the update preparation procedure, which will be gradually changing to include more automation.
An update can be reiterated several times before it is released. Each iteration includes
  • Update of the release branch (involving a substantial amount of manual work)
  • Builds from the release branch
  • Generation of a patch directory (including manual correction of an automatic patch)
  • Annotation (involving manual editing of an html-file)
  • Generation of the update bundle which can be put on hold and tested, or released

1. Update of the release branch

A mock example is used to demonstrates the procedure. Steps (1-3) prepare mock trunk and release branches. Steps (4-7) correspond to the actual procedure.

Steps (6a) and (6b) demonstrate alternative protocols and are mutually exclusive. Run the whole tutorial twice to exercise both possibilities.

(1) Generate a trunk branch:

mkdir -p /tmp/bzr/demo/trunk
cd /tmp/bzr/demo/trunk
echo "file 1" > demo1.txt
bzr init
bzr add .
bzr status
bzr commit -m "added demo1.txt"

(2) Generate a release branch

mkdir /tmp/bzr/series-64
bzr branch /tmp/bzr/demo/trunk /tmp/bzr/series-64/demo

(3) Add a couple of commits to the trunk

cd /tmp/bzr/demo/trunk
echo "file 2" > demo2.txt
bzr add demo2.txt
bzr commit -m "added demo2.txt"
echo "file 3" > demo3.txt
bzr add demo3.txt
bzr commit -m "added demo3.txt"

(4) Set shell variables (to simplify the following description)


For a real project these would have been, for example


(5) Generate a local copy of the release branch

mkdir /tmp/workarea
cd /tmp/workarea
bzr branch $RELEASE demo_local
cd demo_local

• Suppose we need to skip revision 2 and to take only the revision 3 to the release branch (and to the update). In this case proceed as follows.

(6a) Merge required revisions

bzr missing $TRUNK
bzr merge $TRUNK -r2..3
bzr commit -m "merged rev 3 (added file demo3.txt)"
bzr missing $TRUNK
bzr push $RELEASE

– The commands bzr missing does not change anything in the repository and only shows revisions missing in the release branch.
– It is a good idea to indicate at least the revision No and possibly duplicate the original revision message in the current commit message. Then this information will be visible with future invocations of bzr log or bzr missing.

• For some projects (e.g. aimless, pointless, documentation) the release branch is currently in sync with the trunk. For these repositories use bzr pull instead of bzr merge, as demonstrated below.

(6b) Pull the release branch

bzr missing $TRUNK
bzr pull $TRUNK
bzr missing $TRUNK
bzr push $RELEASE

– As opposed to bzr merge, no bzr commit is required after bzr pull.
– It is also possible to pull all sequential revisions up to a certain revision using e.g. bzr pull $TRUNK -r2.
– In both (6a) and (6b) nothing changes in the central repository before bzr push.

(7) To double check whether everything is fine

cd ..
rm -r demo_local
bzr branch $RELEASE demo_local
cd demo_local

2. Release-branch builds

Release branch builds are available for all four supported platforms.
  • Ask Marcin to trigger the release builds for Linux-32, Linux-64 and Windows.
  • There are nightly release builds for Mac.
  • To trigger a new release build for Mac at day time,

3. Generation of patch directory

The working directory and its subdirectories are as follows.



The patch directory current_auto is generated automatically. The patch directory current is generated from current_auto according to instructions contained in the file current.diff and is used downstream for the generation of update bundles. The file current.diff and the directory current re the only two nodes of /sw/update_studio to be handled manually in any way.

(1) Generation of automatic patch directory

ssh -C ccp4um@fg.oisin.rc-harwell.ac.uk
. aliases

– Directory builds/reference_builds contains builds for the four systems at the time when previous update was released.
– The script um_auto downloads latest available builds for the four systems into builds/current_builds
– Then it compares the current and reference builds to generate the patch directory current_auto.
– In addition, it copies current/annotation.xml to current_auto/annotation.xml.

(2) Putting in place
– Normally, the script um_comb should be used for merging selected changes from current_auto to current.
– File current.diff is generated after the first run of um_comb; it contains explanations on how to edit it.
– In short, the first character in an uncommented line means:
     0 - remove this item from current,
     1 - copy the item from current_auto to current and
     2 - do not change the existing item in current.

um_comb -h
um_comb -v
vi current.diff
um_comb -v

– Please read the rest of this section before any manual adjustment to current, should such adjustment be needed.

(3) The following example explains two features of the patch-directory "format".


linux-x86/ccp4-6.4.0/html -> ../../common/ccp4-6.4.0/html
linux-x86_64/ccp4-6.4.0/html -> ../../common/ccp4-6.4.0/html
macosx-x86_64/ccp4-6.4.0/html -> ../../common/ccp4-6.4.0/html
macosx-x86_64/ccp4-6.4.0/share/ccp4i/etc/WINDOWS -> _del_
windows-x86/6.4/html -> ../../common/ccp4-6.4.0/html

– The html directories in this patch are identical for four systems and are linked to a single directory. (The latter is located outside of the systems' subdirectories.) Such links are not generated automatically by um_auto. I wonder if they can still be of any use at all.
– The directory WINDOWS is marked for deletion as it is linked to the file _del_. Such links are generated by um_auto and also can be added manually if necessary. The file _del_ itself will not go to the update. However, a broken link to a file with this basename will also do the job.

Known problems
• symlinks. The program ccp4um that works on users' computers replaces relative paths in symlinks by absolute paths. By itself this is not a critical problem. However the same classes seem to be used in mkupdate, and, in particular, for generation of old-style binary patches, for which such a replacement is critical. Therefore um_auto runs um_walk with the option --follow-links and replaces symlinks with the files to which the symlinks point.
• permissions. The patch format provides a mechanism (via symlinks to _exe_ and _dat_) to include change-of-permission actions into updates. However, the permission functionality seems to be broken in ccp4um, on users' side. Therefore, at this moment, the only way to update permissions of a file is to update the whole file.

4. Annotation of changes

• Annotations of changes should be put in the file annotation.xml.
   – This file stores annotations for all platforms.
   – The data from this file will go automatically to update manager and update web-page, and also should be reformatted and e-mailed to CCP4BB.
   – A template annotation.xml file is copied into the right place when the previous update is out,


• Edit this file to get something like this

<platform alias='mac' name='macosx-x86_64'/>
<platform alias='l64' name='linux-x86_64'/>
<platform alias='l32' name='linux-x86'/>
<platform alias='win' name='windows-x86'/>
<component name='Documentation'>
  <fix>Added documentation for othercell program</fix>
<component name='CCP4I'>
  <fix include='mac'>Cleaned up <i>etc</i> directory</fix>

• Available elements and their attributes:


'blocking' A blocking update requires restart of ccp4um for accessing further updates. Such updates are used for updating ccp4um itself or related components. To generate a blocking update, set this attribute to 'yes'. The default value is 'no'.

'date' This attribute defines the date of update and defaults to the current date. This attribute should not be used in normal circumstances.

'name' This attribute defines the official name of the platform in the update system.

'alias' This is a short arbitrary name used as a value of 'include' and 'exclude' attributes

'name' An arbitrary name best suited for description of what has been changed.

'ifarpwarp' This attribute defaults to 'no' and should be set to 'yes' if the component is ARP/wARP, in which case ccp4um will not show descriptions of fixes for this component for lone ccp4 installations.

This element wraps the description of a particular fix.

These (mutually exclusive) attributes define platforms affected by the fix. Their values are take comma separated lists of platform aliases. For example, the following two expressions are equivalent and mean that both Linux installations are affected: include='l32,l64', exclude='mac,win'. Default - all platforms are affected.

If this element is defined, ccp4um will pop up a window after the update installation with a message wrapped into this element .

  – Use cases can be found in ./archive/6.4.0-00?/annotation.xml files
  – Both <fix> and <message> elements can wrap simple html tags such as <i> as in the example above.

5. Generation of update bundle

• Further operations are performed using um_local script.
  – The user ccp4 on mini-mac has um_local script in the path.
  – There is no need to add all components being updated in one go. A partial update can be put on-hold, tested and extended later.

>> um_local -h
usage: um_local [-h] [--dry] [--yes] [--topdir <dir>] [--release | --clear]

This program (re)generates, releases or resets a currently prepared update. By
default, the update is (re)generated and put on hold. Use --release to release
the update and --clear to reset the update and remove all accumulated files.
By default, the program shows update tables and asks for confirmation before
any further action. Use --yes to skip the question. Use --dry for a dry run.

optional arguments:
  -h, --help      show this help message and exit
  --dry           dry run
  --yes           do not ask confirmation
  --topdir <dir>  used in tests; default ./update_studio is the right value
  --release       prepare and release an accumulated update
  --clear         reset current update and empty current directory

• A dry run is recommended before putting an update on hold or releasing it. This is an auto-test, which also shows what is going to happen during the real run,

>> um_local --dry

Update preparation script
Mode: onhold

Working directory for all steps        /sw/update_studio/
Path to mkupdate binary file           bin/mkupdate
Session log will be appended to        current.log
Destination of packed updates          ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/
Destination of summaries              ccp4null@ccp4serv2.dl.ac.uk:/public/ccp4/www/download/updates/
Destination of patch tarballs          ccp4null@ccp4serv2.dl.ac.uk:/public2/ftp/pub/ccp4/6.4.0/updates/

history                         linux-x86    linux-x86_64   macosx-x86_64     windows-x86

25.10.13:    6.4.0-001           released        released        released        released
20.11.13:    6.4.0-002           released        released        released        released
20.11.13:    6.4.0-003           released        released        released        released
20.11.13:    6.4.0-004           released        released        released        released

current update - previous state

20.11.13:    6.4.0-005                n/a             n/a             n/a             n/a
files to add/ replace                   -               -               -               -
files to remove                         -               -               -               -
new permissions                         -               -               -               -

current update - new state and action

27.11.13:    6.4.0-005            on hold         on hold         on hold         on hold
files to add/ replace                   2               2               2               2
files to remove                         -               -               1               -
new permissions                         -               -               -               -

Continue? [y|N]y

# generating description files
writing file desc/6.4.0/linux-x86/6.4.0-005.dsc
writing file desc/6.4.0/linux-x86_64/6.4.0-005.dsc
writing file desc/6.4.0/macosx-x86_64/6.4.0-005.dsc
writing file desc/6.4.0/windows-x86/6.4.0-005.dsc

# preparing new update
bin/mkupdate desc/6.4.0/linux-x86/6.4.0-005.dsc updates/6.4.0/linux-x86
bin/mkupdate desc/6.4.0/linux-x86_64/6.4.0-005.dsc updates/6.4.0/linux-x86_64
bin/mkupdate desc/6.4.0/macosx-x86_64/6.4.0-005.dsc updates/6.4.0/macosx-x86_64
bin/mkupdate desc/6.4.0/windows-x86/6.4.0-005.dsc updates/6.4.0/windows-x86

# syncing updates to server
scp -p updates/6.4.0/linux-x86/6.4.0-005.pck ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/linux-x86
scp -p updates/6.4.0/linux-x86/updates.xml ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/linux-x86
scp -p updates/6.4.0/linux-x86_64/6.4.0-005.pck ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/linux-x86_64
scp -p updates/6.4.0/linux-x86_64/updates.xml ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/linux-x86_64
scp -p updates/6.4.0/macosx-x86_64/6.4.0-005.pck ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/macosx-x86_64
scp -p updates/6.4.0/macosx-x86_64/updates.xml ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/macosx-x86_64
scp -p updates/6.4.0/windows-x86/6.4.0-005.pck ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/windows-x86
scp -p updates/6.4.0/windows-x86/updates.xml ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/windows-x86

# saving history file
writing file updates/history.xml

# done

• It did not crash and the table current update - new state and action shows correct figures (we use the example of current from §3).
• Hence it is safe to proceed with the real run

>> um_local --yes

• At this point the new update should become available with ccp4um -expert command.
• However, the description files can be cached. Open the following pages in a web-browser and Shift-LeftMouse on the address field to refresh them,


• To start from scratch if ./current has been messed up,

>> um_local --clear

• Eventually, when all planned components are added to the patch directory, the on-hold version of the update passed all relevant tested, and all fixes are described in annotation.xml,

>> um_local --release --dry
>> um_local --release

• The standard output from um_local --release shows the actions that the script has performed.

  Actions noticeable to users
  • The update bundles are placed on server and made accessible by users.
  • The update annotation is put on-line.
  • The old-style binary patches are updated.
  • On-line copy of ccp4 documentation is updated if necessary.
  Technical actions
  • The current release branch of the ccp4 repository is tagged.
  • The directory builds/current_builds is renamed to builds/reference_builds.
  • The directory current is moved to archive and a new current directory with an annotation template is generated.
  • The file current.log is moved to archive.
• The last thing to do is an e-mail to CCP4BB