Missing Number Flags in MTZ files: Developers Guide

Purpose

The missing number flag (MNF) indicates that a certain datum, contained in an MTZ file, has no measured or calculated value. This missing number flag can either be a floating point number or a NaN (Not a Number). If a datum has the same value as the MNF then it is flagged as missing when the record is read from the MTZ file. The missing number flag must be set to a value that is not sensible for any of the data in the file. This makes NaN a good choice for the MNF and it is the default MNF. This choice does lead to some programming difficulties which have been addressed in the in the current release (3.0).

The only "missing datum" test previously used was the occurrence of SIGF = 0.0 to determine whether a measurement was missing; and figure-of-merit (FOM) = 0.0, to determine whether an experimental phase was missing.

N.B. A NaN is a IEEE description which is an entity defined as REAL but which cannot be used in arithmetic operations. On some machines it cannot be printed or used for logical operations. The VMS equivalent to NaN is Rop. Although, NaNs must be treated differently from numbers (see library routines) for simplicity this document will refer to them as numbers.

Programs

The MTZLIB library routines have been extended to include handling the MNF. The header now contains a record giving the assigned MNF. As each record is read the elements are checked, and a logical array (LOGMSS) of the same length as the record is filled; TRUE if the element is flagged as missing, and FALSE otherwise. See documentation on LRREFM.

Small changes have been made to the following routines in MTZLIB.F:

LROPEN        This now reads the value of the MNF for 
              this file.

LRREFL        For each record read, the logical array 
LRREFR        LOGMMS is set.

LWOPEN        This write the MNF to the output header.

LHPRT         This prints the header, now including the 
              MNF value, to the standard output.

New routines added to MTZLIB.F are:

LRREFM        returns logical array (LOGMSS) of the same 
              length as the MTZ record. If the i-th 
              element in the record=MNF, then the 
              corresponding logical element of the array 
              LOGMSS is set true. In all other cases it 
              is false.

SET_MAGIC     either returns the MNF value from MTZ header 
              or sets the MNF in header.

EQUAL_MAGIC   initialises an array so that all elements 
              are set to a MNF.

RESET_MAGIC   resets the MNF for a particular data record. 
              This does not change the MTZ header.

IS_MAGIC      tests invidual datum to see if it is equal 
              to the MNF.

There are two buried routines that are machine dependent;

QNAN          sets a datum to NaN.
QISNAN        tests a datum to see if it is equal to NaN.

QISNAN in diskio.f (UNIX version) and vmsdiskio.f (VMS version) and QNAN is in library.c (UNIX version) and vmsdiskio.f (VMS version).

More details of these library routines are given in libspec.html.

Data

New style MTZ files will have the chosen MNF value stored in the MTZ header records. ( Old style MTZ files which do not use MNFs nor the relevant keyword in the header can still be used, however all files output from programs will have the MNF set. This could cause confusion if an old style MTZ file is not converted to contain MNFs before it is used as input.

An old style MTZ file can be converted to a new style MTZ file with MNFs using MTZMNF, see document on mtzmnf. This is EXTREMELY ADVISABLE if you want to output another file.

The rules used by MTZMNF in conversion and for all standard programs are:


H, K, L       All progs.: they will never be a MNF. Fatal 
              error if otherwise.

F, SIGF       MTZMNF: Both will be flagged as  MNFs or both 
I, SIGI       not.
D, SIGD       Other progs.: if one is flagged and the other 
              is not then a warning will be printed.

FC, PHIC      All progs.: calculated structure factor; 
              should either be both flagged as MNFs or both 
              not. Fatal error if otherwise.

W, PHIB,      All Progs.: experimental phase and weight; 
[HLA, HLB,    either all should be flagged as MNF or
HLC, HLD]     all not. Fatal error if otherwise.

FreeRflag     All progs.: should not normally be a MNF (unless 
              reflection is systematically absent) but not 
              fatal if otherwise.

Multi-record files can contain MNFs for data items such as XDET YDET etc., if the data processing program has not output them. The programs in the suite which use Multi-record files are sortmtz, scala, rotavata, agrovata, rotaprep, absurd and uudupl. At present these programs do not use MNFs effectively. Programmers working with multi-record files will need to consider the appropriate action.

For programs in the suite using standard MTZ files the functionality has been kept the same. That is, that where F=<0.0 or SIGF=<0.0 were checked, now the MNF is checked as well.

Examples

At present there are four different possibilities when considering MNFs. Reading and writing MTZ files and manipulating the MNF in the data in order to sort the data and to print it to the standard output.

1. Reading MTZ

The first case is when you are reading in an MTZ file. All data items must be checked for the MNF. Below is an example of how to read in and check data items. When using MTZ files the ordering of the subroutine calls are the same as before ie. MTZINI, LROPEN, LRREFL etc.. After calling LRREFL or LRREFF it is necessary to call LRREFM to return the logical array LOGMSS. LOGMSS is set to TRUE if a datum is missing, otherwise it is set FALSE. This array is used within the main program to test for data which is missing. Example from MLPHARE;


120 CONTINUE

...

C            **************************
        CALL LRREFF(1,SSQOL4,ADATA,EOF)
C            **************************
C
        IF (EOF) GO TO 130
C
C  LOGMSS(I) returned FALSE if no magic_no set or if ADATA(I) = magic_no
C
C       **************************
        CALL LRREFM(1,LOGMSS)
C       **************************
...
C    Label 120: read the next reflection
C  First check for MNF set
        IF (LOGMSS(4) .OR. LOGMSS(5) ) GO TO 120
C  Now do the old style check, on SIGF = 0.0
        IF (ADATA(4).LT.0.000001) GO TO 120
        FPRO    = ADATA(4)
        SIGFPRO = ADATA(5)
C

Notice that both F and SIGF are tested for MNF. Normally both F and SIGF are flagged as missing, or both are present, BUT it is possible to set FP and SIGF to refer to different data "pairs" in all CCP4 programs.

2. Manipulating data in order to print to standard output.

The second case is where you wish to print data on the default output stream (unit=6). Not all systems will be able to print a NaN without crashing. Therefore, you must reset the missing number flag in the data to a floating point number. To do this use RESET_MAGIC, this routine does not alter the MNF set in the MTZ header. It is not necessary to ensure that the MNF is an invalid value for all data columns because this is merely for display purposes. If you feel that this is necessary then see the fourth example. Example from MTZDUMP;


C
C            ******************************************
C VALML set to somethhing you can print eg -999.0
        CALL LRREWD(MTZIN)
        WRITE (6,'(/,A,/)') ' LIST OF REFLECTIONS'
.....
C
C            *************************
        CALL LRREFL(MTZIN,RESOL,ADATA,EOF)
        CALL RESET_MAGIC(MTZIN,ADATA,BDATA,MCOLS,DUMMY,VALML)
C            *************************
.....
C
          DO 160 JDO160 = 1,3
            IHKL(JDO160) = NINT(ADATA(JDO160))
  160     CONTINUE
C
....
              WRITE (6,6022) IHKL, (BDATA(J),J=4,NCOLS)
C

3. Writing MTZ output files

a) When an input MTZ file has already been opened on the same unit number. The MNF like the rest of the header information ( CELL, SYMMEYRY etc) is copied to the output header.
b) When no input MTZ file has been opened on the same unit number. The default MNF is NaN, and this can only be altered in CAD or by a call to SET_MAGIC.

Writing MTZ files follows the same procedure as before. In the example below BDATA is output to MTZOUT. The input data is contained in ADATA which is transferred to BDATA for all data items that are not equal to MNF. Note that BDATA is initialised to the MNF using EQUAL_MAGIC. This procedure need not be followed in all cases, because ADATA could be written out again with some modifications, and then the MNFs would still be retained. (eg SCALEIT). Example from MLPHARE:


C
C   First fill the BDATA array with missing number flags.
C            *******************************
      DO 10 JJ = 1,MaxMTZCols
        CALL EQUAL_MAGIC(1,BDATA,MaxMTZCols)
   10 CONTINUE
C            *******************************
C
C   Begin filling BDATA(.) copying elements from ADATA(.) if they are not missing
      BDATA(1) = ADATA(1)
      BDATA(2) = ADATA(2)
      BDATA(3) = ADATA(3)
      IF(.NOT. LOGMSS(4)) BDATA(4) = ADATA(4)
      IF(.NOT. LOGMSS(5)) BDATA(5) = ADATA(5)

...

C---- Now transfer phib and fom to BDATA(.) providing they have been calculated
C---- the value -99999.0 is used as an internal check on this.
C
      IF(ABEST.NE.-99999.0) THEN
       BDATA(IPHIB) = RADDEG*ABEST
       BDATA(IPHIB+1) = FOM
      ENDIF
C
.....
C   Any BDATA entry which has not been inserted will have remained as a MNF
C          *******************
      CALL LWREFL(1,BDATA)
C          *******************
C

4. Processing data from several input files, eg in SORTMTZ.

This procedure will have to be adopted for other library functions such as SRTRLS which you may not be able to handle NaNs. In this case there may be different MNF values in the different input files. The output file will contain the MNF flag found in the first input file.

This requires that the MNF in the data is reset to a floating point number. (In SORTMTZ MNF is reset to -999999.0 and this value is checked against all input data. If a datum is found to have a value more negative than this the program terminates). This is an example from SORTMTZ:


C
      DATA        ....  /,VRSET_MAGIC/-999999.0/

...

C
C---- Check to see if VRSET_MAGIC missing value flag is acceptable
C
C     ===========================================
      CALL LRINFO(INFILE,VERSN,NCOL,NREFL,RANGES)
C     ===========================================
C
      DO 20 I=1,NCOL
        IF (RANGES(1,I) .LT. VRSET_MAGIC) THEN
          WRITE(LUNOUT,'(A,I6,A,F10.1,/,A)') ' Data from file ',
     +    INFILE,' is more negative than ',VRSET_MAGIC,
     +    ' You must either scale down the data or change VRSET_MAGIC'
          CALL CCPERR(1,' *** ERROR: Program Terminating')
        ENDIF
  20  CONTINUE

...

       if (first_file) then

C---- Initialise sort, set parameters, etc
........
C
C
C  Find the MNF for first file to save for  output file
C
C       =======================================
        SETVAL = .FALSE.
        CALL SET_MAGIC(INFILE,VAL_MAGIC,SETVAL)
C       =======================================

...

C   Cycle around many hklin files ; 
C   the internal MNF may be different for different files

C---- Open next input file
C
C     =======================================
      CALL LROPEN(INFILE,FILNAM,IPRINT,IFAIL)
C     =======================================
......

C---- Read reflection records
C
C     ====================================
      CALL LRREFL(INFILE,SRESOL,BDATA,EOF)
C     ====================================
C
      IF (EOF) GO TO 60
C
C---- Convert missing data from original MNF for this file to VRSET_MAGIC
C
C     ===========================================================
      CALL RESET_MAGIC(INFILE,BDATA,ADATA,NCOL,DUMMY,VRSET_MAGIC)
C     ===========================================================
C
C---- Pass keys and record to sort
C
      ISTAT = SRTRLS(ADATA)

...

C  When all files read and records passsed to sort routines.

C set MNF to VAL_MAGIC, if there is more than one input file then MNF is 
C converted to that of the first one.
C         ===================================================
          SETVAL = .TRUE.
          CALL SET_MAGIC(OUTFIL,VAL_MAGIC,SETVAL)
C         ===================================================
.....
C---- Now pick up records and output
C
        ISTAT = SRTRET(ADATA)
        IF (ISTAT.EQ.ENDSRT) THEN
          GO TO 90
        ELSE IF (ISTAT.EQ.0) THEN
C
C
......
C  Basically you are seting BDATA = ADATA unless ADATA = VRSET_MAGIC
C  In which case BDATA = VAL_MAGIC
C
C         ==========================================================
          CALL RESET_MAGIC(0,ADATA,BDATA,NCOL,VRSET_MAGIC,VAL_MAGIC)
C         ==========================================================
C
C---- Write out reflection record
C
C         =========================
          CALL LWREFL(OUTFIL,BDATA)
C         =========================


.....
C       ==========================
        CALL LWCLOS(OUTFIL,IPRINT)
C       ==========================

One final point, changing the MNF in an output MTZ file is a two stage process. The MNF value in the header is changed by using SET_MAGIC. Then the MNFs in the data itself need to be converted with RESET_MAGIC.

PREV User Guide on MNFs CCP4 Home Page NEXT MNF and Library Routines