CCP4i Workshop: Primer for the basic files .tcl, .def, .script, .com

Peter Briggs
March 2001


1. Introduction

The scheme below gives a simple overview of how the different files fit together when a task window is started and a task is run.

Main Window -> Task Interface -> Running Task -> Running Program
modules.def task_name.def &
$CCP4I_TOP/etc/SYSTEM/ $CCP4I_TOP/tasks/ $CCP4I_TOP/scripts/ $CCP4I_TOP/templates/

Each of these files is discussed in a separate section below. I have tried to follow the sequence above and use a standard format for the major sections: each starts with an italicised link to the appropriate part of the CCP4i programmer's documentation, followed by single summary line description in bold type, and ends with a list of comments and hints.

2. Modules File

CCP4i documentation reference: $CCP4I_HELP/programmers/modules.html

  • The modules file defines which items appear in the task menu list on the left hand side of the CCP4i main window.
  • This lives in $CCP4I_TOP/etc/SYSTEM/ where SYSTEM is either UNIX or WINDOWS (stored in $system(OPSYS) inside CCP4i). Here is a description of the main features:

    Modules (see left) are defined by lines of the form:

    MODULE module_name Brief_text_description

    The brief text description is what the user sees in the modules list accessed from the LHS of the main window. For example:

    MODULE datred  Data Reduction
    The module_name (e.g. datred) is used to specify the name of the help file which contains an overview of the tasks in the module ($CCP4I_HELP/modules/module_name.html. By default the Help button on the top right of a task window will link to this help file.
    Underneath each MODULE line is a set of lines which define the task list for that module. Each task in the module task list (see left) is defined by lines of the form:

    task_name Brief_text_description

    The task_name is the basename for the .tcl and .def files (see section below) which define the task interface.

    Brief_text_description is the text the user sees on the button to start the task, and Help_text_description is the text that appears in the message line help on the main window when the cursor is positioned over the task button. For example:

      scalepack2mtz  Import Scaled Denzo Data
      Import data from scalepack

    Some comments...on the modules file

    3. Task Interface

    A task interface requires task_name.def and task_name.tcl files which live in the $CCP4I_TOP/tasks directory.

    3.1 Parameter files: .def file

    CCP4i documentation reference: $CCP4I_HELP/programmers/def_files.html#def_file

  • The .def file for a task defines the type and initial value for the parameters used in that task
  • Here is an example to illustrate the main components of the .def file:

    #CCP4I VERSION CCP4Interface 0.2
    #CCP4I SCRIPT DEF example
    #CCP4I DATE  28/03/01 11:15:30
    #CCP4I USER lizp
    ``Preamble'' - comments used for checking names and dates. This is followed by a list of the parameters.
    TITLE           _title_text         ""
    The generic format is:

    PARAMETER_NAME data_type initial_value

    So for example the parameter TITLE is of type _title_text and has an initial value equal to an empty string.

    Nb: all tasks can have a title (even if the programs don't have a TITLE keyword) so this parameter should always appear.

    REFINE_TYPE     _demo_refine_type   FULL
    INPUT_PHASE     _demo_input_phase   NO
    Here _demo_refine_type and _demo_input_phase are not standard CCP4i data types - they are custom menus which are defined in the task window (.tcl) file (see below).
    INPUT_FILES     _list_of_text       "HKLIN XYZIN"
    OUTPUT_FILES    _list_of_text       "HKLOUT XYZOUT"
    HKLIN           _MTZ_file           ""
    DIR_HKLIN       _default_dirs       ""
    HKLOUT          _MTZ_file           ""
    DIR_HKLOUT      _default_dirs       ""
    XYZIN           _pdb_file           ""
    XYZOUT          _pdb_file           ""
    DIR_XYZIN       _default_dirs       ""
    DIR_XYZOUT      _default_dirs       ""
    All task should have the parameters INPUT_FILES and OUTPUT_FILES, which are lists of logical names for files used in the task. These parameters are used in the View Files From Job option to create the lists of input and output files.

    Also, for each parameter defining a input or output file there should be a corresponding DIR_fileparam parameter which holds the Project alias or path for the file.

    LABIN           _text               "FP SIGFP"
    FP              _mtz_label_f        ""
    SIGFP           _mtz_label_sig      ""
    PHIB            _mtz_label_phi      ""
    FOMB            _mtz_label_w        ""
    An example of MTZ labels the use of MTZ (see below for more detail). Each MTZ column type has a standard CCP4i data type defined in types.def.
    NCYCLES                 _positiveint    5
    EXCLUDE_RESOLUTION      _logical        0
    EXCLUDE_RESOLUTION_MIN  _positivereal   ""
    EXCLUDE_RESOLUTION_MAX  _positivereal   ""
    Some examples of simple parameters using other standard data types.

    Some Comments...on .def files

    3.1.1 Data types and the types.def file

    The data type of a parameter determines what values are allowed for that parameter (to allow for limit input checking, though at the moment only weak type checking is used in CCP4i). It also determines the appearance of the parameter in the task window interface - for example, _logical types appear as a check button, _text as a text entry box, and so on.

    CCP4i's standard data types are defined in the types.def file, in $CCP4I_TOP/etc. A list of the main types can be found here:

    List of CCP4i data types: $CCP4I_HELP/programmers/def_files.html#list

    although this list is incomplete - check the types.def itself to see the complete set.

    It is not so important at this stage to know precisely how the data types are defined, but if you are interested then more information can be found in the CCP4i programmers documentation:

    types.def file: $CCP4I_HELP/programmers/def_file.html#types_def

    Additional non-standard data types can be defined in the task_name_setup procedure in the task window file - most typically these are menus of options which are specific to the task in question. Setting up menus is covered in detail below.

    3.2 Task window file: .tcl file

    CCP4i documentation reference: $CCP4I_HELP/programmers/task_windows.html

  • The task window file defines the appearance and behaviour of the task interface window.
  • It is written in Tcl but all essential functionality is provided by a library of CCP4i commands so in practice a minimal knowledge of Tcl should be required for most interfaces. Documentation for the main commands can be found at: $CCP4I_HELP/programmers/indexframeset.html

    3.2.1 Procedures in the task window file

    CCP4i documentation reference: $CCP4I_HELP/programmers/task_windows.html#procedures_and_arrays

    The task window file task_name.tcl contains one or more Tcl procedures which are called in a specific order when the task window is started or the task is run:

    Procedure name Function Called when
    task_name_setup (Optional) Perform initialisations, including defining local menus Immediately before the main task_name_task_window procedure
    task_name_task_window (Mandatory) This is the core procedure and contains all the commands required to define the graphical interface After task_name_setup
    task_name_run (Optional) Perform any necessary checking and tidying up of parameters prior to actually running the task When the Run option is selected by the user from the task interface window
    task_name_review (Optional) Perform any additional operations after the task has been run by the user, e.g. to update loggraphs. NB this is not widely used. Immediately after the task has been run using the Run option
    task_name_miscellaneous (Optional) Other procedures specific to this task, e.g. for implementing extending frames (see Liz's notes), or to be called as a result of the -command option on a widget. As required from inside task_name_task_window

    The first three types are detailed in the sections below.

    3.2.2 The task_name_setup procedure

    This procedure is optional. Its main use is to define the menus which are specific to the task in question, and although it can also be used for other task-specific initialisations these are not discussed here. Setting up custom menus

    The DefineMenu command is used to set up a new menu. It is documented at $CCP4I_HELP/programmers/DefineMenu.html, here are some examples to illustrate how it is used.

    A simple example:

    proc demo_setup {  typedefVar arrayname } {
        upvar #0  $typedefVar typedef
        upvar #0 $arrayname array
        DefineMenu _demo_refine_type [ list FULL FAST ]
        return 1
    This sets up a new menu called _demo_refine_type, with two options specified in the list. Note that _demo_refine_type is a new data type and not the name of a new variable, so to use it add a line in the .def like:
        REFINE_TYPE     _demo_refine_type       FULL

    A more typical example using an alias list:

        DefineMenu _demo_refine_type [list "full refinement" \
                                            "quick refinement" ] \
                                     [ list FULL \
                                            FAST ]
    The menu is set up as before but now each entry in the first list also has a corresponding alias in the second. Conventionally the alias list will contain the (sub)keywords used in the program while the first list contains more readable versions which are visible to the user in the task window.

    The GetValue command is used to return the alias (e.g. FULL) rather than the menu item (e.g. "full refinement") for a parameter which is a menu. This command is documented at $CCP4I_HELP/programmers/GetValue.html

    Some comments...on the task_name_setup procedure

    3.2.3 The task_name_task_window procedure

    CCP4i documentation reference: $CCP4I_HELP/programmers/task_windows.html#windows_and_widgets

  • This is the core procedure which contains all the commands required to define the graphical task interface.
  • As mentioned previously, CCP4i provides a library of commands which are used to build windows with the conventional format, without the programmer needing to write Tk directly. An introduction to the commands can be found at $CCP4I_HELP/programmers/task_windows.html#CCP4I_library.

    In broad terms the task window can be thought of in terms of a simple hierarchy of elements:

    Task window ... ... arranged into horizontal partitions called folders ... ... made up of lines

    In a similar way there is also hierarchy of commands associated with each of these basic elements:

    Level Task window Folders Lines
    Associated CCP4i commands CreateTaskWindow OpenFolder CreateLine family

    Below I introduce these three sets of commands: CreateTaskWindow, OpenFolder and the CreateLine commands; a table summarising these (and others) can be found in Appendix B.


    CreateTaskWindow creates a window with the conventional CCP4i format (including the Run, Save&Restore, Exit and Help buttons) and initialises the parameters in the array by reading the $CCP4_top/tasks/taskname.def file and (if it exists) the user's project default taskname.def. Therefore this command is always the first to be called.

    The documentation for CreateTaskWindow is at $CCP4I_HELP/programmers/CreateTaskWindow.html, but an example of the usual way that the command is used is given below:

      if { [CreateTaskWindow $arrayname  \
            "Run My Task" "MyTask" \
            [ list "Required Parameters"  \
            "Define Protein Ranges to Refine" \
            "Crystal Parameters" ] ] == 0 } return
    This opens a task interface window with the title Run My Task and icon name MyTask (used when the interface window is minimised). The window will have three folders (in addition to protocol and file folders) - the number is not specified explicitly, instead it is implied by the number folder titles given in the list.


    OpenFolder specifies the start of each folder inside the task window. Documentation for OpenFolder is at $CCP4I_HELP/programmers/OpenFolder.html.

    There are three folder types:

    The visibility (or otherwise) of each folder can also be specified when the OpenFolder command is invoked:

    The default visibility status is open, but this can be changed for a particular folder, e.g.

          OpenFolder 2 closed
    makes a folder with default status closed. Folder visibility can also be dependent on parameter values, e.g.
          OpenFolder 3 SF_ACTION open [ list REDUCE ] hide
    makes a folder with default status hide, but this will change to open if the parameter SF_ACTION takes the value of "REDUCE".

    This command specifies the start of the folder only. The folder contents i.e. the text and widgets are specifed by the commands which immediately follow it. The documentation for OpenFolder is at $CCP4I_HELP/programmers/OpenFolder.html

    CreateLine... Family

    The CreateLine... commands are used to draw the lines of text and widgets in the folder, and appear after the relevant OpenFolder command has been invoked, e.g.

      OpenFolder protocol 
      CreateTitleLine line TITLE
      CreateLine line \
            message "Refinement method (REFI TYPE)" \
            help refi_type \
            label "Do" widget REFINE_TYPE  \
            message "Use input phases? (PHASE or NOPHAS)" \
            help nophase \
            label "using" widget INPUT_PHASE
    will create the following lines in the task window:

    Documentation for the CreateLine... commands can be found via the CCP4i programmer's documentation (see below for a list, and relevant links), but brief descriptions of the most important elements are given below.

    label text
    The specified text appears in the line
    widget PARAMETER
    PARAMETER is one of the parameters defined in the .def file, and this option creates an appropriate Tk ``widget'' to allow the user to manipulate the value of that parameter.
    The type of widget is determined automatically by the data type of the parameter:
         _integer   - entry box
         _real      - entry box
         _logical   - checkbutton
         _text      - entry box
         _file      - entry box
         _menu      - pop-up menu
         _mtz_label - pop-up menu or entry box
    so their is no need for the programmer to specify this.
    NB: Specifying the -oblig option after a widget command will force the widget to be highlighted in gold until valid input is entered. (Note that this is only meaningful for entry box widgets!)
    message_text is a short (i.e. one-line) message which appears in the gold bar at the top of the task window when the cursor moves over the succeeding widgets in the line. message can appear as many times as necessary in a single line.
    help anchor_point
    This is used in conjunction with the SetProgramHelpFile command, which sets the basename of a html program documentation file in $CHTML. A right mouse button click over the succeeding widgets will bring up the help file at the specified internal anchor_point e.g. mytask.html#anchor_point.
    help can appear as many times as necessary in a single line.
    toggle_display PARAMETER status Value_List
    The toggle_display element can be used to control the visibility of the line dependent on the value of PARAMETER - when PARAMETER is set equal to one of the values in the Value_List then the line assumes the specified visibility status (open or hide).

    Another example, this time using the Create...FileLine and CreateLabinLine commands:

      OpenFolder file 
      CreateInputFileLine line \
            "Enter input MTZ file name (HKLIN)" \
          "MTZ in" \
           HKLIN DIR_HKLIN \
           -fileout HKLOUT DIR_HKLOUT "_demo" \
           -setfileparam space_group_name SPACE_GROUP \
      CreateLabinLine line \
           "Observed amplitude (FP) and obligatory sigma (SIGFP)" \
           HKLIN "FP" FP  [list FP F_P] \
           -sigma "Sigma" SIGFP [list SIGFP SIGF_P SIGP]
      CreateOutputFileLine line \
           "Output MTZ File" \
           "MTZ out" \
    This is not the full example, which creates a file folder looking something like:

    Again, full descriptions of these commands can be found in the CCP4i programmer's documentation but the key points are outlined here.

    The CreateInputFile line creates a line for an input file name. The first two arguments are respectively the message line help text and the text appearing immediately before the filename entry box in the window. The next two arguments are the parameter names for the file (e.g. HKLIN) and the project (e.g. DIR_HKLIN), which must be defined in the .def file:

       HKLIN           _MTZ_file           ""
       DIR_HKLIN       _default_dirs       ""

    The fileout option is used to create a default output file name based on the input file name - in this case the default output file name (in parameter HKLOUT in project DIR_HKLOUT) will be assigned as the input file name with _demo appended.

    The setfileparam option is used to set a parameter to an initial value based on the contents of the input file, in this case parameter SPACE_GROUP will be assigned the space group name extracted automatically from the MTZ header. Other quantities can be extracted e.g. cell parameters, resolution limits etc. Several setfileparam options can be specified on a single CreateInputFileLine and the same quantity to be assigned multiple times to more than one parameter.

    The CreateOutputFileLine line is similar to the CreateInputFileLine, but is much simpler. Both types also support a toggle_display option similar to that described for CreateLine.

    Finally, CreateLabinLine is used for MTZ file label selection. It draws a line with a menu for the user to select a labelled column, the names of which are extracted automatically from the specified input file. If the sigma is also specified then a second menu is drawn for the associated sigma.
    The documentation should be consulted from a description of the exact syntax of this command, but it is easy to use.

    Some comments...about CreateLine...

    3.2.4 The task_name_run procedure

    CCP4i documentation reference: $CCP4I_HELP/programmers/task_windows.html#task_name_run

    The task_name_run procedure is not compulsory, but if it exists then it will be executed when the user hits the Run button in the task window. It should be used to do any tidying and checking of the parameters before the programs are run.

    Typically there are three specific parameters which might need tidying up:

    CCP4i expects every task to have parameters INPUT_FILES and OUTPUT_FILES which are necessary for the project database to save the names of input and output files correctly. The input and output files for a task often depend on the mode of running and these parameters might need to be updated after the user has set the run mode and just before the job is actually run.

    A simple example of tidying the file lists:

    proc demo_run { arrayname } {
      upvar #0 $arrayname array
      set array(INPUT_FILES) "HKLIN"
      switch [GetValue $arrayname MAP_OR_COORDS] \
      MAP {
          set array(USE_XYZIN) 0
          append array(INPUT_FILES) " MAPIN"
      } PDB {
          set array(USE_XYZIN) 1
          append array(INPUT_FILES) " XYZIN"
      } LIB {
          set array(USE_XYZIN) 1
      return 1
    LABIN (or similar lists of MTZ column labels)
    In CCP4i the LABIN parameter is usually a list of the MTZ input column label parameters. The LABIN parameter is used by the LABELLINE keyword in the command template files below to generate the LABIN keyword in the program command file. The required MTZ input columns are often dependent on the mode of running the task and so the LABIN parameter might need to be updated after the user has set the run mode and just before the job is actually run.

    A simple example of tidying up the column selections:

    proc demo_run { arrayname } {
      upvar #0 $arrayname array
      if [regexp PHASE [GetValue $arrayname INPUT_PHASE]] {
        set array(LABIN) "FP SIGFP PHIB FOMB"
      } else {
        set array(LABIN) "FP SIGFP"
      return 1

    Some comments procedure

    4. Running the Task

    4.2 Script file - .script file

    CCP4i documentation reference: $CCP4I_HELP/programmers/run_scripts.html

  • The script file contains the functionality which is invoked when the user hits the Run button on the task window.
  • When a task is run, a separate process is started automatically which executes the task_name.script file, which should be in the $CCP4I_TOP/scripts directory. The script file is also written in Tcl but again CCP4i provides a set of commands so most simple scripts should require minimal Tcl knowledge.

    The simplest scripts just run a single program, for example:

      CreateComScript demoprog demo_script
      set status [Execute $cmd $demo_script program_status report]
    This script does three things:

    This is all you should need - the underlying CCP4i libraries take care of passing the parameter values, opening temporary files, assigning a log file name and so on.

    More complicated scripts call several programs using the same basic sequence of commands, with Tcl commands forming the ``glue'' to make decisions, perform loops and so on, but this is beyond the scope of these notes. The best recommendation is to examine some existing task scripts e.g. for fft or dm for examples.

    Some comments on...script files

    4.2 Command template file - .com file

    CCP4i documentation reference: $CCP4I_HELP/programmers/command_template.html

  • Command templates are used to generate the command files for CCP4 (or other) programs.
  • The procedure CreateComScript will substitute parameters from the parameter (.def) file into the template (.com) file to create a command file - a set of keywords which are used as input to a program.

    The existing CCP4i documentation for template files is an excellent reference (see URL above) which gives a clear and comprehensive overview. Here I will simply show a few relatively simple examples by way of illustrating what is possible.

    A simple example

    A typical line in a template file might look like:

    There are two important points: A line will only be written to the command file if the write flag is true (the write flag itself is not written to the command file). So in this example, if the values were set as
    SET_RESOLUTION    _logical      1
    RESO_MIN          _resolution   10.0
    RESO_MAX          _resolution   3.0
    then the line
    resolution 10.0 3.0
    would be written to the command script. If SET_RESOLUTION was set to 0 then no line would be written.

    If the line should always be written to the command script then the write flag can be explicitly set to 1 in the template file, e.g.

    1 resolution $RESO_MIN $RESO_MAX

    A simple example using continuation lines

    Some program keywords have optional arguments or subkeywords, for example

    fiddlefactors <factor_1> <factor_2> [ <factor_3> ]

    where the last argument is optional. In this case we could use a template like:

    1 fiddlefactors $FACTOR_1 $FACTOR_2
    The first line is always written, because it has a write flag of 1. The second line encodes a component of the command which will only be written if the parameter EXTRA_FIDDLE is 1 (true), with the ``-'' character at the beginning of the line indicating that it is a continuation of the preceding line.

    In CCP4 command scripts ``-'' is used as a continuation character, so when the line is written (provided that EXTRA_FIDDLE is 1) then this character will be appended to the end of the preceding line:

    fiddlefactors 0.1 20 -

    A variation for this template is:

    $SET_FIDDLEFACTORS fiddlefactors $FACTOR_1 $FACTOR_2
    Now the first line depends on the value of SET_FIDDLEFACTORS. Note that if this parameter is false (0) then neither line will be written, regardless of the value of EXTRA_FIDDLE.

    An example using double continuations

    Some programs have complex keywords where suboptions have their own suboptions. These can be encoded with the double continuation symbol ``--'', for example:

    1 exclude
    The sub-component keywords sig2 and f2min will only be written out if the preceding components are written and if their own write flag function is true.

    (The IfSet procedure is another CCP4i command, which checks that a parameter is not an empty string.)

    Control Structures for more complicated command scripts

    The command template format supports control structures of the form IF-ELSE-ENDIF, LOOP and CASE (note that the keywords MUST be uppercase and written on separate lines, and that it is permissible to nest LOOPs and IFs). For more details about these I suggest that you consult the relevant part of the documentation, at $CCP4I_HELP/programmers/command_template.html#control_structures.

    One last example - LABELLINE for writing LABIN and LABOUT Commands

    The LABELLINE command is used within command templates to facilitate LABIN and LABOUT lines.

    Here is a simple example. Say that we have defined the labels as follows:

    LABIN           _text               "F1 SIGF1 PHI"
    F1              _mtz_label_f        "FP"
    SIGF1           _mtz_label_sig      "SIGFP"
    PHI             _mtz_label_phi      "PHIC"
    The LABIN parameter holds the list of ``program labels'' (actually a list of parameters which hold the values of the individual ``file labels'' elsewhere in the .def file). Then the template line
    will result in the command line: labin F1 = FP - SIGF1 = SIGFP - PHI = PHIC

    Again, for more details about the LABELLINE command I suggest that you consult the relevant part of the documentation, $CCP4I_HELP/programmers/command_template.html#labelline.


    A. How parameters from the .def file get into the .tcl file

    CCP4i documentation reference: $CCP4I_HELP/programmers/task_windows.html#procedures_and_arrays

    All the parameters for one instance of a task window are stored in a single Tcl array - normally this has the same name as task_name but it's not safe to assume that this is always the case. The names of parameters from the .def file are indices in this array

    When the array name is passed to a procedure then the Tcl upvar command must be used to associate that name with a local variable, so that the parameter values stored in the array can be accessed and changed. (Or in programmer-speak: Tcl variables are passed by value, not by reference.)

    An example:

    proc example_task_window { arrayname } {
       upvar #0 $arrayname array

    Some comments...on array naming conventions

    B. Task Window Command Summary

    Command Description Documentation in $CCP4I_HELP/programmers/
    CreateTaskWindow Open a window of the usual format for task interfaces CreateTaskWindow.html
    OpenFolder begin defining the contents of a folder in a task interface window; set the status of the folder OpenFolder.html
    CreateLine create a line in an interface window CreateLine.html
    CreateTitleLine create an entry line for the title of the run  
    draw a line for the user to select input (output) file, including the Project alias, View file and Browse buttons CreateInputFileLine.html
    CreateLabinLine draw a line for the user to select labelled columns from an input file CreateLabinLine.html
    CreateLineTemplate create a template to define the layout of a line CreateLineTemplate.html
    SetProgramHelpFile specify the CCP4 (or other) program help file which is to be targeted by the on-line help SetProgramHelpFile.html
    open (close) a subsidiary frame within a task window folder OpenSubFrame.html