CCP4I2 Generating Reports

Introduction

Handling XML

CCP4ReportParser classes

The report definition will have a tree structure with a Report object as the root node. The child objects may be either container objects that have children or content objects that represent features such as text or tables.Report is a sub-class of Container (which should not be used directly). The other useful sub-classes are Fold which creates a 'fold' in the report whose contents can be displayed or hidden by clicking on a title line and GraphGroup or PictureGroup which groups together multiple graphs or pictures to be displayed in the same widget. There are several other Container types that are used to support the logic in the XRT report definitions but these are not necessary to support Python definitions.

The __init__ method of all classes have three arguments that pass along some basic data:

There are also optional arguments to specify the id or class attribute for the html element that may be useful for customising the appearance of the report. Note that class is a reserved word in Python so the Python argument is called class_ and also some report features have a fixed class attribute needed to support the standard functionality.

The Python report definition should usually begin:

from CCP4ReportParser import *
from CCP4ErrorHandling import *

class my_task_report(Report):
  # Specify which gui task and/or pluginscript this applies to
  TASKNAME = 'my_task'
  def __init__(self,xmlnode=None,jobInfo={},**kw):
    Report. __init__(self,xmlnode=xmlnode,jobInfo=jobInfo,**kw)
    if self.errorReport().maxSeverity()>SEVERITY_WARNING:
      print 'FAILED instantiating my_task report generator'
      self.errorReport().report()
      return
    graph = self.addGraph( title="The best results" , select="//results/best" )

The task-specific report object will be instantiated with appropriate xmlnode and jobInfo data by the ccp4i2 framework and must pass these arguments onto the Report sub-class which will hand them on to its child objects. The xrtnode argument is not used in the Python api and can be ignored.

File paths

Specification of file paths can begin '$CCP4I2' for files in the CCP4i2 distribution directory.

Tables

The table layout and headers are specified by the Container.addTable() and Table.addData() methods, the values are extracted from the task XML. The table object has two optional arguments, transpose changes the orientation of the table, and select, an XPATH expression which extracts a list of elements containing row data. Each extracted element is expected to contain all the information for a single table row. Occasionally the data in the task XML may not be in a suitable form for this, in which case the argument is omitted. The addData() method can take four arguments: title, subtitle, expr and select. The title argument gives the column header. The subtitle argument allows tables with nested headers. The select argument specifies an xpath expression to select the data to be displayed. If the addTable() select argument has selected a list of row elements then this should specify a unique element within that row element. If addTable() has no select argument, the select should return a list of elements which will become the column values. In the latter form there is no guarantee columns will be of the same length. The expr argument enables a python expression to be applied to the value before display.

Alternatively the Python script can extract and process the data as it needs and then load it using the data argument of the addData() method; this should be a list of values for a column of data.

Graphs

This is an interface to the pimple graph display program which will be shown inline within the report. pimple can display multiple plots in one window with two combo-boxes to allow selection of the current plot. The left-hand combo-box will select a data table and the right-hand box will select between views on that table. The Graph object can only specify one data table with, optionally, multiple views specified by the addPlot block. Multiple Graph objects can be grouped in a GraphGroup object to be displayed in the same pimple instance with the title arguments from each addGraph() appearing in the left-hand combo-box.

This makes a picture in the report with buttons to launch either CCP4mg or Coot with the same view. The picture is generated by CCP4mg which runs in background without appearing on the computer screen. CCP4mg needs a scene.xml file to specify what it is to display and save as a picture. The scene file specifies which data files (either coordinates or maps) are to be loaded and how they should be drawn. The drawing specification can be either a simple scene or generated using the CCP4mg Picture Wizard. Many other aspects of the display can also be specified using CCP4mg's usual XML. The template for the scene.xml file should be given as the scene argument or provided in a separate file that is specified by the sceneFile argument. Typically some parameters such as filenames need to be substituted into the scene.xml template and this should be done using database or select attributes to specify the source of the substituted data.

Insert a button in the report to launch another viewer. Currently supports CCP4mg and Coot. The initial view in CCP4mg can be set by specifying a pictureDefinitionFile argument.

Help

This puts a Help link by tables, graphs etc. and is highly recommended! The ref argument gives the link reference which can be a file path starting with an environment variable (eg $CCP4I2) and the appropriate full path will be substituted into the html.

Bibliographic references

Accessing other program output

Customising Report Appearance

The appearance of reports is determined by css and Javascript files that are referenced in the header of the generated html file. Individual report html elements or groups of html elements for selection by Javascript or css can be given id and/or class attributes by using id or class_ arguments to the element's __init__ method or to the Container.addWhatever methods. Note the underscore in the class_ argument that is necessary to differentiate from a Python reserved word. For example in the buccaneer_build_refine_report.py there is the line:

graph = parent.addGraph( title="Completeness by build-refine cycle", select="//BuccaneerBuildRefineResult/BuildRefineCycle", id = "completenessGraph")

This will create a graph element in the html with the id completenessGraph.

By default the css xreport.css and Javascript xreport.js files are referenced in generated html files. These files have a pathname in the CCP4i2 release such as ccp4i2/docs/report_files/x.y.z/xreport.css where x.y.z is a version directory.Developers may add alternative css or js files but please consider and discuss how your requirements might be met by updating the default files. If you wish to use alternative files then theg the filename (e.g. whatever.css) must be passsed as argument cssFile or jsFile to Report.__init__().

Future releases of CCP4i2 will retain old report_files version directories in order to support any old report files. By default the report parser will generate html to use the latest version available unless Report.__init__() class has a cssVersion argument which is set to an earlier version number. Developers are recommended to set this parameter to the version that they have used in development of the prd and to only update the value after they have tested against any more recent version of report_files. If the cssVersion is set but there is a higher minor version number available then that will be used on the assumption that this minor version number increments are bug-fixes.

CCP4i2 uses an internal HTTP server to serve the files from report_files and it usually expects these to have a URL beginning http://127.0.0.1:43434/report_files. The port number (default 43434) may change - reports are edited at display time to correct the port number. Old report files which do not use the HTTP server are also be updated to do so.

Real-time updating reports

A report can be generated and updated as a job is running - a drawback is that at the moment the entire report is redone so speed is a possible issue and the 'running' report should be not too long - usually an updating graph is ideal. The report class should have the the class parameter RUNNING set true so the system knows to call the report generator as the job is running. The Report.__init__() method is passed a jobStatus argument which will be Running and the appropriate report should be returned.

By default the system watches a running job's program.xml file and when this file changes there is a redraw of the job report. If you require to watch an alternative file then set the class attribute MyReport.WATCHED_FILE to the name of the watched file or a relative path to the file from the job directory.

Creating Reports for Pipelines

In principle a report from a pipeline is no different from a report from a wrapper but a pipeline report may want to incorporate some elements from the sub-job reports. For this to be possible the sub-job report should be written in modular fashion with separate methods for each element in the report. The Report.__init__() method calls the appropriate methods to generate the report dependent on the jobStatus argument which is usually either Finished, Unsatisfactory or Running. The Report.__init__() method should also handle a jobStatus value of noOutput for which it returns without creating any report elements. For example:

class MyWrapperReport(Report):
  def __init__(self,xmlnode=None,jobInfo={},jobStatus=None,**kw):
    Report. __init__(self,xmlnode=xmlnode,jobInfo=jobInfo,**kw)
    if self.errorReport().maxSeverity()>SEVERITY_WARNING:
      print 'FAILED instantiating MyWrapper report generator'
      self.errorReport().report()
      return

    # 'nooutput' mode would be used by another report class that wanted
    # to use some method(s) from this class for its own report
    if jobStatus is not None and jobStatus.lower() == 'nooutput':
      return
    elif jobStatus == 'Running':
      self.completenessGraph(self)
    else:
      self.completenessGraph(self)
      self.details(self)
      self.picture(self)

  def completenessGraph(self,parent):
    graph = parent.addGraph( title="Completeness by cycle", select="//MyWrapper/Cycle" )
    graph.addData (title="Cycle",  select="Number" )
    ...
    ...

Note that here the method completenessGraph is passed a reference to self so it will (in the first line) add a graph to self and then go on to add details to that graph.

A pipeline that uses the MyWrapper wrapper and wanted to incorporate elements from the MyWrapperReport could then do:

   import MyWrapperReport
   myWrapperXml = self.xmlnode.xpath0('foo/bar')
   myWapperReport = MyWrapperReport.MyWrapperReport(xmlnode=myWrapperXml,jobStatus='noOutput')
   myWapperReport.picture(parent=self)

Here we are assuming that the program xml for MyWrapper is embedded in the pipeline xml and can be selected by the xpath 'foo/bar' and this is then passed to MyWrapperReport.__init__() as the argument xmlnode. Alternatively data may come from a separate file that should be passed as the xmlFile argument. MyWrapperReport.__init__() is called with jobStatus='noOutput' so that it creates no output. The pipeline report generator can then call the appropriate methods for the report elements that it requires. It passes self as the parent argument so the elements are appended to the pipeline report.

One possible issue with calling Report methods from different places is that the top level of Report.xmlNode, the etree representation of the XML data file, may not be at the same level in the hierarchy and this will make writing xpath selections tricky. The solution may be to either add or remove a top level etree element. Here are two examples assuming we want the top level element to be 'MYWRAPPER'. To remove some redundant parent elements:

  if not self.xmlNode.tag == 'MYWRAPPER':
    myWrapperNode = self.xmlNode.xpath0('MYWRAPPER')
    if myWrapperNode is not None:
      self.xmlNode = myWrapperNode
    else:
      print "ERROR no MYWRAPPER node"

Alternatively to add an extra parent element:

  from lxml import etree
  newNode = etree.Element('NEWNODE')
  newNode.append(self.xmlNode)
  self.xmlNode = newNode

Examples of Python Report Definitions

Currently best example:

Testing A Python Report Definitions

If the following script fragment (with the appropriate name of the report class replacing 'my_task_report') is appended to the python file the script can be run from a Python prompt.

def test(xmlFile=None,jobId=None,reportFile=None):
  import sys,os
  if reportFile is None:
    if xmlFile is not None:
      reportFile = os.path.join(os.path.split(xmlFile)[0],'report.html')
    else:
      reportFile = os.path.join(os.getcwd(),'report.html')
  r = my_task_report(xmlFile=xmlFile,jobId=jobId)
  r.as_html_file(reportFile)
  if len(r.errorReport())>0: print 'ERRORS:',r.errorReport()

It is easiest to test if you have previously run the task from the gui so that there is program output xml and a jobId that can be input to the test. The command line is:

> CCCP4I2_TOP/bin/pyi2

>>>  import my_task_module
>>>  my_task_module.test(my_xmlfile,my_jobid)

The xml file should be the output xml from a job run previously and the jobId (not the same as the job number which is displayed in the project viewer) can be found in the meta-data header of files such as params.xml and input_params.xml). This will create report called report.html in the working directory.

Alternatively to test run the report parser in the GUI where you can see the result better: after running an appropriate job once, close the GUI while it is displaying the report for the job, delete the report.html file for that job and reopen the GUI; it should recreate and display the report.

A good current working example is in tasks/buccaneer_build_refine/buccaneer_build_refine_report.py

Appendix A - The Database data accessible in the jobInfo dictionary

filenames / XYZIN
          / HKLIN
          / XYZOUT
          / HKLOUT
          etc
runtime
status
taskname
jobid
jobnumber
projectid
projectname
tasktitle
fileroot - the job directory
descendentjobs - a list of jobIds

Appendix B - The CCP4mg scene file

NB this will only work with recent nightly build versions of CCP4mg (post 16th Oct 2012).

A simple example of scene template:

<scene>
  <data>
    <MolData id='id1'>
       <filename database="filenames/XYZOUT"/>
    </MolData>
  </data>
  <wizard><template>ribbons:colour chains</template>
    <parameters>
      <MolData1>id1</MolData1>
    </parameters>
  </wizard>
</scene>

This template requires only one piece of substituted data - the XYZOUT file from the database is substituted as the filename for the <MolData><filename> element to give the scene file:

  <ccp4i2_header>
    <function>MGSCENE</function>
    <projectName>s12</projectName>
    <userId>lizp</userId>
    <ccp4iVersion>0.0.1</ccp4iVersion>
    <jobId>66b40dd7146111e2875a3c0754185dfb</jobId>
    <projectId>969cdfae138111e2acd83c0754185dfb</projectId>
    <creationTime>14:39 12/Oct/12</creationTime>
    <jobNumber>4</jobNumber>
  </ccp4i2_header>
  <ccp4i2_body>
    <scene>
      <data>
       <MolData id="id1">
         <filename>/Users/lizp/Desktop/test_projects/s12/CCP4_JOBS/job_4/XYZOUT.pdb</filename>
       </MolData>
      </data>
      <wizard>
        <template>ribbons:colour chains</template>
        <parameters>
          <MolData1>id1</MolData1>
        </parameters>
      </wizard>
   </scene>
  </ccp4i2_body>
</ccp4:ccp4i2>

This file has the standard CCP4i2 xml structure with a header containing metadata describing the provenance of the file. The body of the file is one or more <scene> elements. Each <scene> element contains a <data> block that specifies the files to be loaded and a <wizard> block that specifies how the data should be drawn using a CCP4mg wizard template.

The <data> block should list the files to be loaded as <MolData> or <MapData> elements. These elements must have unique id attributes and a <filename> sub-element. Usually the actually filename will need to be substituted in from the filenames in the CCP4i2 database so the database attribute should be used to specify the source of the filename.

The <wizard> element should contain a <template> sub-element that specifies the CCP4mg wizard template in the form folder:template name. A template can be selected from the existing wizard templates which can be viewed in CCP4mg or something appropriate can be created (see ccp4mg/help/picture_wizard.html and talk to Liz or Stuart). Picture wizard templates are partially a Python script which can apply some intelligence in setting up a display but they can also have some options which can be set. The options are in the CHOICES section of the wizard template file. The <wizard> sub-element <parameters> should contain sub-elements specifying values for the wizard 'choices' and particularly should provide values for MolData and MapData parameters which cross-reference to the id attributes of the sub-elements of the <data> section. Note that although the wizard is mostly used in CCP4mg to setup the display of a single coordinate file it can also be used to specify the display of map objects and of multiple data objects.

There is an alternative mechanism to specify a simple scene that does not use the wizard, for example:

<scene>
  <data>
    <MolData id='id1'>
       <filename database="filenames/XYZOUT"/>
       <MolDisp>
        <select>peptide</select>
        <colour>bychain</colour>
        <style>CYLINDERS</style>
       </MolDisp>
    </MolData>
    <MapData id='id2'>
      <filename database="filenames/HKLOUT"/>
      <columns> <F>2FOFCWT </F> <PHI>PH2FOFCWT </PHI> </columns>
      <difColumns> <F>FOFCWT </F> <PHI>PHFOFCWT </PHI> </difColumns>
      <model>id1 </model>
      <gridSize>0.5 </gridSize>
      <contourUnits>sigma </contourUnits>
      <MapDisp>
        <contourLevel>3 </contourLevel>
        <difference>1 </difference>
      </MapDisp>
     </MapData>
  </data>
</scene>

The MolData element can have one or more MolDisp sub-elements. Each MolDisp element should contain select, colour and style elements and can contain much more such as control of labels, visibility, flashing - see the CCP4mg documentation. (????). Note that a select attribute could be added to any of the elements so their values can be taken from the program output.

Any of the aspects of CCP4mg presentation that can be controlled using the CCP4mg XML format can also be incorporated in the scene file. Probably the most useful feature is control of the view using:

   <View>
      <centre_MolData>id2 </centre_MolData>
      <centre_selection>A/1170(DUP) </centre_selection>
      <orientation_auto> <molData>id2 </molData> <selection>A/1170(DUP) </selection> </orientation_auto>
      <scale_auto>1 </scale_auto>
   </View>

Here the MolData object and (optionally) an atom selection that will be the centre of the view are specified. The orientation_auto element then tells CCP4mg to try to find the best orientation to show a specified atom selection. scale_auto is a flag to CCP4mg to scale to zoom of the specified atom selelction rather than show all visible atoms.

Alternatively the scale,orientation and centring can be specified explicitly. In this example variously other display parameters are set - these could also be used the the 'auto' view parameters shown above.

      
   <View>
     <scale>58.0125
     <orientation>
       <q0>0.0134139428983 </q0>
       <q1>0.336496398576 </q1>
       <q2>-0.507403789037 </q2>
       <q3>-0.793178186004/q3>
    </orientation>
    <centre_xyz>
      <x>-23.676</x>
      <y>8.538</y>
      <z>5.262</z>
    </centre_xyz>
    <fog_enabled>1</fog_enabled>
    <fog_near>-1.1</fog_near>
    <fog_far>26.01</fog_far>
    <slab_offset>27.5</slab_offset>
    <slab_enabled>false</slab_enabled>
    <slab_width>55.0</slab_width>
  </View>

Note that it is possible to have more than one scene specified in the file. It is intended that CCP4mg will use these either to create multiple pictures in one program run or, if it is opened in interactive mode, to create a Presentation which gives the user a choice of views.

Appendix C - Style Guide

Firstly the 'running' reports that are updated in real time usually consist of a graph that shows, for example, progress per cycle and possibly a table showing initial, previous and current cycle values.

Final reports should begin with a short text summary of what the task achieved, any issues and warning messages in red. The terminology here (and throughout the report) should be consistent with the terminology used in the gui and documentation. Links to documentation are encouraged.

There is usually an open folder showing a graphical summary of the task followed by closed folders with more detailed information possibly from the separate steps of the task.

Visual presentation (i.e. graphs) is preferred over tables - thought tables presenting the same information may be good. The usual good practice for graphs - label axes, legends etc. apply. A simple clear graph or multiple simple clear graphs are preferred over over-loaded graphs (though beware what users might want to look at simultaneously.)