Current update preparation procedure
This document reflects the current state of the update
preparation procedure, which will be gradually changing to
include more automation.
An update can be reiterated several times before it is
released. Each iteration includes
- Update of the release branch (involving a
substantial amount of manual work)
- Builds from the release branch
- Generation of a patch directory (including manual
correction of an automatic patch)
- Annotation (involving manual editing of an
html-file)
- Generation of the update bundle which can be put on
hold and tested, or released
1. Update of the release branch
A mock example is used to demonstrates the procedure.
Steps (1-3) prepare mock trunk and release branches. Steps
(4-7) correspond to the actual procedure.
ATTENTION:
Steps (6a) and (6b) demonstrate alternative protocols and
are mutually exclusive. Run the whole tutorial twice to
exercise both possibilities.
(1) Generate a trunk branch:
mkdir -p /tmp/bzr/demo/trunk
cd /tmp/bzr/demo/trunk
echo "file 1" > demo1.txt
bzr init
bzr add .
bzr status
bzr commit -m "added demo1.txt"
ls
|
(2) Generate a release branch
mkdir /tmp/bzr/series-64
bzr branch /tmp/bzr/demo/trunk /tmp/bzr/series-64/demo
|
(3) Add a couple of commits to the trunk
cd /tmp/bzr/demo/trunk
echo "file 2" > demo2.txt
bzr add demo2.txt
bzr commit -m "added demo2.txt"
echo "file 3" > demo3.txt
bzr add demo3.txt
bzr commit -m "added demo3.txt"
ls
|
(4) Set shell variables (to simplify the following
description)
TRUNK=/tmp/bzr/demo/trunk
RELEASE=/tmp/bzr/series-64/demo
|
For a real project these would have been, for example
#TRUNK=bzr+ssh://andrey@fg.oisin.rc-harwell.ac.uk/var/lib/gforge/chroot/scmrepos/bzr/molrep/trunk
#RELEASE=bzr+ssh://andrey@fg.oisin.rc-harwell.ac.uk/var/lib/gforge/chroot/scmrepos/bzr/series-64/molrep
|
(5) Generate a local copy of the release branch
mkdir /tmp/workarea
cd /tmp/workarea
bzr branch $RELEASE demo_local
cd demo_local
ls
|
• Suppose we need to skip revision 2 and to take only the
revision 3 to the release branch (and to the update). In
this case proceed as follows.
(6a) Merge required revisions
bzr missing $TRUNK
bzr merge $TRUNK -r2..3
bzr commit -m "merged rev 3 (added file demo3.txt)"
bzr missing $TRUNK
bzr push $RELEASE
|
– The commands bzr missing does not change
anything in the repository and only shows revisions
missing in the release branch.
– It is a good idea to indicate at least the revision No
and possibly duplicate the original revision message in
the current commit message. Then this information will be
visible with future invocations of bzr log or bzr
missing.
• For some projects (e.g. aimless, pointless,
documentation) the release branch is currently in
sync with the trunk. For these repositories use bzr
pull instead of bzr merge, as
demonstrated below.
(6b) Pull the release branch
bzr missing $TRUNK
bzr pull $TRUNK
bzr missing $TRUNK
bzr push $RELEASE
|
– As opposed to bzr merge, no bzr commit
is required after bzr pull.
– It is also possible to pull all sequential revisions up
to a certain revision using e.g. bzr pull $TRUNK -r2.
– In both (6a) and (6b) nothing changes in the central
repository before bzr push.
(7) To double check whether everything is fine
cd ..
rm -r demo_local
bzr branch $RELEASE demo_local
cd demo_local
ls
|
2. Release-branch builds
Release branch builds are available for all four supported
platforms.
- Ask Marcin to trigger the release builds for
Linux-32, Linux-64 and Windows.
- There are nightly release builds for Mac.
- To trigger a new release build for Mac at day time,
3. Generation of patch directory
The working directory and its subdirectories are as follows.
ccp4um@ccp4serv5:/var/lib/gforge/chroot/home/groups/series-??/update_studio:
archive/
bin/
builds/
builds/current_builds/
builds/current_builds/linux-x86/
builds/current_builds/linux-x86_64/
builds/current_builds/macosx-x86_64/
builds/current_builds/windows-x86/
builds/reference_builds/
builds/reference_builds/linux-x86/
builds/reference_builds/linux-x86_64/
builds/reference_builds/macosx-x86_64/
builds/reference_builds/windows-x86/
current.log
current/
current/annotation.xml
current/linux-x86/
current/linux-x86_64/
current/macosx-x86_64/
current/windows-x86/
current_auto/
current_auto/annotation.xml
current_auto/linux-x86/
current_auto/linux-x86_64/
current_auto/macosx-x86_64/
current_auto/windows-x86/
desc/
tests/
updates/
|
The patch directory current_auto is generated
automatically. The patch directory current is
generated from current_auto according to
instructions contained in the file current.diff
and is used downstream for the generation of update
bundles.
The file current.diff and the directory current
re the only two nodes of /sw/update_studio to be
handled manually in any way.
(1) Generation of automatic patch directory
ssh -C ccp4um@fg.oisin.rc-harwell.ac.uk
. aliases
cu
um_auto
|
– Directory builds/reference_builds contains
builds for the four systems at the time when previous
update was released.
– The script um_auto downloads latest available
builds for the four systems into builds/current_builds
– Then it compares the current and reference builds to
generate the patch directory current_auto.
– In addition, it copies current/annotation.xml
to current_auto/annotation.xml.
(2) Putting in place
– Normally, the script um_comb should be used
for merging selected changes from current_auto
to current.
– File current.diff is generated after the first
run of um_comb; it contains explanations on how
to edit it.
– In short, the first character in an uncommented line
means:
0 - remove this item from current,
1 - copy the item from current_auto
to current and
2 - do not change the existing
item in current.
um_comb -h
um_comb -v
vi current.diff
um_comb -v
|
– Please read the rest of this section before any manual
adjustment to current, should such adjustment be
needed.
(3) The following example explains two features of the
patch-directory "format".
./current:
annotation.xml
common/ccp4-6.4.0/html/INDEX.html
common/ccp4-6.4.0/html/othercell.html
linux-x86/ccp4-6.4.0/html -> ../../common/ccp4-6.4.0/html
linux-x86_64/ccp4-6.4.0/html -> ../../common/ccp4-6.4.0/html
macosx-x86_64/ccp4-6.4.0/html -> ../../common/ccp4-6.4.0/html
macosx-x86_64/ccp4-6.4.0/share/ccp4i/etc/_del_
macosx-x86_64/ccp4-6.4.0/share/ccp4i/etc/WINDOWS -> _del_
windows-x86/6.4/html -> ../../common/ccp4-6.4.0/html
|
– The html directories in this patch are
identical for four systems and are linked to a single
directory. (The latter is located outside of the systems'
subdirectories.) Such links are not generated
automatically by um_auto. I wonder if they can
still be of any use at all.
– The directory WINDOWS is marked for deletion
as it is linked to the file _del_. Such links
are generated by um_auto and also can be added
manually if necessary. The file _del_ itself
will not go to the update. However, a broken link to a
file with this basename will also do the job.
Known problems
• symlinks. The program ccp4um that works on
users' computers replaces relative paths in symlinks by
absolute paths. By itself this is not a critical problem.
However the same classes seem to be used in mkupdate,
and, in particular, for generation of old-style binary
patches, for which such a replacement is critical.
Therefore um_auto runs um_walk with
the option --follow-links and replaces symlinks
with the files to which the symlinks point.
• permissions. The patch format provides a mechanism (via
symlinks to _exe_ and _dat_) to
include change-of-permission actions into updates.
However, the permission functionality seems to be broken
in ccp4um, on users' side. Therefore, at this
moment, the only way to update permissions of a file is to
update the whole file.
4. Annotation of changes
• Annotations of changes should be put in the file annotation.xml.
– This file stores annotations for all
platforms.
– The data from this file will go
automatically to update manager and update web-page, and
also should be reformatted and e-mailed to CCP4BB.
– A template annotation.xml file is
copied into the right place when the previous update is
out,
• Edit this file to get something like this
<update>
<platform alias='mac' name='macosx-x86_64'/>
<platform alias='l64' name='linux-x86_64'/>
<platform alias='l32' name='linux-x86'/>
<platform alias='win' name='windows-x86'/>
<component name='Documentation'>
<fix>Added documentation for othercell program</fix>
</component>
<component name='CCP4I'>
<fix include='mac'>Cleaned up <i>etc</i> directory</fix>
</component>
</update>
|
• Available elements and their attributes:
| <update> |
|
|
|
'blocking' |
A blocking update requires restart
of ccp4um for accessing further updates.
Such updates are used for updating ccp4um
itself or related components. To generate a
blocking update, set this attribute to 'yes'.
The default value is 'no'.
|
|
'date' |
This attribute defines the date of
update and defaults to the current date. This
attribute should not be used in normal
circumstances.
|
<platform>
|
|
|
|
'name' |
This attribute defines the official
name of the platform in the update system. |
|
'alias' |
This is a short arbitrary name used
as a value of 'include' and 'exclude'
attributes
|
<component>
|
|
|
|
'name' |
An arbitrary name best suited for
description of what has been changed. |
|
'ifarpwarp' |
This attribute defaults to 'no'
and should be set to 'yes' if the
component is ARP/wARP, in which case ccp4um
will not show descriptions of fixes for this
component for lone ccp4 installations. |
<fix>
|
|
This element wraps the description
of a particular fix. |
|
'include'
'exclude'
|
These (mutually exclusive)
attributes define platforms affected by the fix.
Their values are take comma separated lists of
platform aliases. For example, the following two
expressions are equivalent and mean that both
Linux installations are affected: include='l32,l64',
exclude='mac,win'. Default - all
platforms are affected. |
<message>
|
|
If this element is defined, ccp4um
will pop up a window after the update installation
with a message wrapped into this element .
|
– Use cases can be found in ./archive/6.4.0-00?/annotation.xml
files
– Both <fix> and <message>
elements can wrap simple html tags such as <i>
as in the example above.
5. Generation of update bundle
• Further operations are performed using um_local
script.
– The user ccp4 on mini-mac has um_local
script in the path.
– There is no need to add all components being
updated in one go. A partial update can be put on-hold,
tested and extended later.
>> um_local -h
usage: um_local [-h] [--dry] [--yes] [--topdir <dir>] [--release | --clear]
This program (re)generates, releases or resets a currently prepared update. By
default, the update is (re)generated and put on hold. Use --release to release
the update and --clear to reset the update and remove all accumulated files.
By default, the program shows update tables and asks for confirmation before
any further action. Use --yes to skip the question. Use --dry for a dry run.
optional arguments:
-h, --help show this help message and exit
--dry dry run
--yes do not ask confirmation
--topdir <dir> used in tests; default ./update_studio is the right value
--release prepare and release an accumulated update
--clear reset current update and empty current directory
|
• A dry run is recommended before putting an update on
hold or releasing it. This is an auto-test, which also
shows what is going to happen during the real run,
>> um_local --dry
Update preparation script
Mode: onhold
Working directory for all steps /sw/update_studio/
Path to mkupdate binary file bin/mkupdate
Session log will be appended to current.log
Destination of packed updates ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/
Destination of summaries ccp4null@ccp4serv2.dl.ac.uk:/public/ccp4/www/download/updates/
Destination of patch tarballs ccp4null@ccp4serv2.dl.ac.uk:/public2/ftp/pub/ccp4/6.4.0/updates/
history linux-x86 linux-x86_64 macosx-x86_64 windows-x86
25.10.13: 6.4.0-001 released released released released
20.11.13: 6.4.0-002 released released released released
20.11.13: 6.4.0-003 released released released released
20.11.13: 6.4.0-004 released released released released
current update - previous state
20.11.13: 6.4.0-005 n/a n/a n/a n/a
files to add/ replace - - - -
files to remove - - - -
new permissions - - - -
current update - new state and action
27.11.13: 6.4.0-005 on hold on hold on hold on hold
files to add/ replace 2 2 2 2
files to remove - - 1 -
new permissions - - - -
Continue? [y|N]y
# generating description files
writing file desc/6.4.0/linux-x86/6.4.0-005.dsc
writing file desc/6.4.0/linux-x86_64/6.4.0-005.dsc
writing file desc/6.4.0/macosx-x86_64/6.4.0-005.dsc
writing file desc/6.4.0/windows-x86/6.4.0-005.dsc
# preparing new update
bin/mkupdate desc/6.4.0/linux-x86/6.4.0-005.dsc updates/6.4.0/linux-x86
bin/mkupdate desc/6.4.0/linux-x86_64/6.4.0-005.dsc updates/6.4.0/linux-x86_64
bin/mkupdate desc/6.4.0/macosx-x86_64/6.4.0-005.dsc updates/6.4.0/macosx-x86_64
bin/mkupdate desc/6.4.0/windows-x86/6.4.0-005.dsc updates/6.4.0/windows-x86
# syncing updates to server
scp -p updates/6.4.0/linux-x86/6.4.0-005.pck ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/linux-x86
scp -p updates/6.4.0/linux-x86/updates.xml ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/linux-x86
scp -p updates/6.4.0/linux-x86_64/6.4.0-005.pck ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/linux-x86_64
scp -p updates/6.4.0/linux-x86_64/updates.xml ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/linux-x86_64
scp -p updates/6.4.0/macosx-x86_64/6.4.0-005.pck ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/macosx-x86_64
scp -p updates/6.4.0/macosx-x86_64/updates.xml ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/macosx-x86_64
scp -p updates/6.4.0/windows-x86/6.4.0-005.pck ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/windows-x86
scp -p updates/6.4.0/windows-x86/updates.xml ccp4null@ccp4serv2.dl.ac.uk:/public2/www/update/6.4.0/windows-x86
# saving history file
writing file updates/history.xml
# done
|
• It did not crash and the table current update - new
state and action shows correct figures (we use the
example of current from §3).
• Hence it is safe to proceed with the real run
• At this point the new update should become available
with ccp4um -expert command.
• However, the description files can be cached. Open the
following pages in a web-browser and Shift-LeftMouse
on the address field to refresh them,
http://www.ccp4.ac.uk/update/6.4.0/linux-x86_64/updates.xml
http://www.ccp4.ac.uk/update/6.4.0/linux-x86/updates.xml
http://www.ccp4.ac.uk/update/6.4.0/macosx-x86_64/updates.xml
http://www.ccp4.ac.uk/update/6.4.0/windows-x86/updates.xml
• To start from scratch if ./current
has been messed up,
• Eventually, when all planned components are added to the
patch directory, the on-hold version of the update passed
all relevant tested, and all fixes are described in annotation.xml,
>> um_local --release --dry
>> um_local --release
|
• The standard output from um_local --release
shows the actions that the script has performed.
Actions noticeable to users
- The update bundles are placed on server and made
accessible by users.
- The update annotation is put on-line.
- The old-style binary patches are updated.
- On-line copy of ccp4 documentation is updated if
necessary.
Technical actions
- The current release branch of the ccp4 repository is
tagged.
- The directory builds/current_builds is
renamed to builds/reference_builds.
- The directory current is moved to archive
and a new current directory with an
annotation template is generated.
- The file current.log is moved to archive.
• The last thing to do is an e-mail to CCP4BB
|