Baubles: Making the World a Better Place for Logfile Viewing

Peter Briggs^* and Kevin Cowtan^**

^*CCP4, CSE Department, STFC Daresbury Laboratory, Warrington WA4 4AD
Email: p.j.briggs@dl.ac.uk

^**Structural Biology Laboratory, University of York, Heslington, York YO10 5YW
Email: cowtan at ysbl.york.ac.uk

Introduction

It is a perennial problem with running many CCP4 programs, that in order to understand the results of a program run it is often necessary to look at the log file from that program and this can sometimes be quite challenging. Traditional methods such as viewing the "raw" log file using the UNIX/Linux more command in a terminal window (see figure 1a) have given way to tools such as the file viewer and loggraph programs in CCP4i (figure 1b). However the log file can still be difficult to navigate, particularly if it consists of the output from many programs run in sequence - as is often the case for CCP4i tasks.


Figure 1a) Log file viewed using `more` in a Linux terminal window.	Figure 1b) Log file displayed using CCP4i viewer, with graphs shown in separate loggraph window.	Figure 1c) Log file reformatted by `baubles` and displayed in web browser with inline graphs in JLogGraph.

Baubles is an experimental program that attempts to make log file navigation and intepretation easier, by reformatting them into HTML so they can be viewed in a web browser (figure 1c).

It aims to work with the existing log file structure (for example, the standard program banners, warning and termination messages, and summary tags) so that it can work with the existing CCP4 programs without requiring substantial modifications. It uses a (substantially improved) version of the JLogGraph Java applet to show log graphs inline, and uses JavaScript to add interactivity (for example, the ability to toggle between "summary" and "full" logfile displays) using JavaScript.

Baubles is written in Python and uses the smartie module to do the work of interpreting log files (see the article in newsletter 46: http://www.ccp4.ac.uk/newsletters/newsletter46/articles/smartie.html). It runs on Linux and Windows systems and the generated output should be viewable using both Firefox and Internet Explorer.

Some examples of logfiles reformatted using baubles

The best way to get a feel for what baubles does is to look at a few examples:

Scala: 57_scala.html (see the original log file)
Buccaneer with Refmac5: 393_buccaneer_ref.html (see the original log file)
Refmac5: 156_refmac5.html (see the original log file)
Truncate: 28_truncate.html (see the original log file)

The Scala and Buccaneer examples each include a "results" section, which is transformed by baubles and placed in a prominent position at the start of each program summary. Similarly the Truncate example includes some warning messages that have been extracted and displayed prominently by baubles.

Some other examples can also be found at http://www.ccp4.ac.uk/peter/logfiles/.

Limitations

Baubles is still very much an experiment and there are a number of limitations, for example:

Some programs don't (for whatever reason) produce particularly useful summaries in their log files, so the reformatted version output from baubles isn't always that much more useful. You can't make a silk purse from a sow's ear.
Some programs aren't readily recognised by baubles because they don't output "standard" CCP4 program banners or other features that baubles can interpret.
The reformatted log file from baubles doesn't always help with understanding the outcomes of a task, particularly when many programs are run together in a single script.

Trying it out

You can give baubles a try now - a version of baubles is available for download from:

ftp://ftp.ccp4.ac.uk/pjx/ccp4/smartie/baubles-0.0.5.tar.gz

This file includes an installer script that will update CCP4 6.0.2 to make baubles available from CCP4i (sorry it only installs into Linux and UNIX-type systems at the moment - see the README file for instructions on how to manually install the update for Windows).

Once baubles has been installed, any new jobs that are run in CCP4i will also run baubles at the end to generate an "annotated" version of the log file which can then be accessed via a "View Annotated Log in Web Browser" option in the "View Files from Job" menu (see figure 2).

Figure 2) The "View Annotated Log..." option for accessing HTML versions of log files generated by baubles.

If you're a program developer

You can make your program output more "baubles-compatible" by following a few guidelines:

Use the standard CCP4 banners to flag the start (and end, if you're adventurous) of the output from your program - cut and paste from an existing program (but be careful not to add extra #'s!).
Use the $TABLE markup for tables of data that should be displayed as graphs.
Write summary tags <!-SUMMARY_BEGIN--> and  into your program output to enclose any sections that should be particularly interesting to users, for example the key results at the end of the program run.
Use the $TEXT markup for warnings or informational messages. For example, warning messages can be flagged using:
```
	$TEXT:Warning: $$ Stuff here is ignored $$
        Something very important that the user really ought to know about!
        $$
```
while a "result" section can be denoted by:
```
	$TEXT:Result: $$ Stuff here is ignored $$
        Put details of the major results here.
        $$
```
(Note that there should only be a single "result" section per program log). In both cases, the messages will be extracted and displayed prominently within the baubles-annotated log file. In the case of the "results" section, additional "magic" formatting is applied in baubles to make the information more readable (for example by putting tabulated data into a HTML table).

Examples of warning messages can be seen in the Truncate example (28_truncate.html) and of the results section in the Scala and Buccaneer/Refmac5 examples (57_scala.html and 393_buccaneer_ref.html).

Information on the $TABLE and $TEXT markup formats can be found in the CCP4 documentation, for example at http://www.ccp4.ac.uk/dist/html/loggraphformat.html. In addition, the article on smartie in the previous newsletter has some information about the kind of log file features that baubles is able to recognise. You can also email us and we'll try to help make your program output more baubles-compliant and (hopefully) more user-friendly.

If you're a program user

Give baubles a try on your favourite programs, and send feedback to us about how we might improve it. Also send feedback to program developers suggesting which bits of program log file you would like to see in a summary.

The Future of Baubles

Baubles is still quite experimental, however we intend to incorporate it in some form in the next release of CCP4. Beyond that we hope to look at ways of improving baubles reformatting to make it more useful operating on "standard" logfiles, and at the same time to put pressure on program developers to make improvements to the "raw" log files output from their programs. Beyond that we would also like to investigate ways to link to other relevant resources and possibly include "commentaries" on the program output.

Credits

Baubles was originally written by Peter Briggs and is based on the "smartie" Python module. It has since undergone several iterations of development by both Peter Briggs and Kevin Cowtan. Kevin also produced the original changes to CCP4i to integrate baubles into it, along with the installer script. Both have subsequently been adapted by Peter.

The JLogGraph Java applet was written by Kevin Cowtan who has also made several recent substantial improvements.

Phil Evans has provided some invaluable feedback on both baubles and JLogGraph, for which PJB and KC are both very grateful.