The only "missing datum" test previously used was the occurrence of SIGF = 0.0 to determine whether a measurement was missing; and figure-of-merit (FOM) = 0.0, to determine whether an experimental phase was missing.
N.B. A NaN is a IEEE description which is an entity defined as REAL but which cannot be used in arithmetic operations. On some machines it cannot be printed or used for logical operations. The VMS equivalent to NaN is Rop. Although, NaNs must be treated differently from numbers (see library routines) for simplicity this document will refer to them as numbers.
Small changes have been made to the following routines in MTZLIB.F:
LROPEN This now reads the value of the MNF for this file. LRREFL For each record read, the logical array LRREFR LOGMMS is set. LWOPEN This write the MNF to the output header. LHPRT This prints the header, now including the MNF value, to the standard output.
New routines added to MTZLIB.F are:
LRREFM returns logical array (LOGMSS) of the same length as the MTZ record. If the i-th element in the record=MNF, then the corresponding logical element of the array LOGMSS is set true. In all other cases it is false. SET_MAGIC either returns the MNF value from MTZ header or sets the MNF in header. EQUAL_MAGIC initialises an array so that all elements are set to a MNF. RESET_MAGIC resets the MNF for a particular data record. This does not change the MTZ header. IS_MAGIC tests invidual datum to see if it is equal to the MNF.
There are two buried routines that are machine dependent;
QNAN sets a datum to NaN. QISNAN tests a datum to see if it is equal to NaN.QISNAN in diskio.f (UNIX version) and vmsdiskio.f (VMS version) and QNAN is in library.c (UNIX version) and vmsdiskio.f (VMS version).
More details of these library routines are given in libspec.html.
An old style MTZ file can be converted to a new style MTZ file with MNFs using MTZMNF, see document on mtzmnf. This is EXTREMELY ADVISABLE if you want to output another file.
The rules used by MTZMNF in conversion and for all standard programs are:
H, K, L All progs.: they will never be a MNF. Fatal error if otherwise. F, SIGF MTZMNF: Both will be flagged as MNFs or both I, SIGI not. D, SIGD Other progs.: if one is flagged and the other is not then a warning will be printed. FC, PHIC All progs.: calculated structure factor; should either be both flagged as MNFs or both not. Fatal error if otherwise. W, PHIB, All Progs.: experimental phase and weight; [HLA, HLB, either all should be flagged as MNF or HLC, HLD] all not. Fatal error if otherwise. FreeRflag All progs.: should not normally be a MNF (unless reflection is systematically absent) but not fatal if otherwise.
Multi-record files can contain MNFs for data items such as XDET YDET etc., if the data processing program has not output them. The programs in the suite which use Multi-record files are sortmtz, scala, rotavata, agrovata, rotaprep, absurd and uudupl. At present these programs do not use MNFs effectively. Programmers working with multi-record files will need to consider the appropriate action.
For programs in the suite using standard MTZ files the functionality has been kept the same. That is, that where F=<0.0 or SIGF=<0.0 were checked, now the MNF is checked as well.
1. Reading MTZ
The first case is when you are reading in an MTZ file. All data items must be checked for the MNF. Below is an example of how to read in and check data items. When using MTZ files the ordering of the subroutine calls are the same as before ie. MTZINI, LROPEN, LRREFL etc.. After calling LRREFL or LRREFF it is necessary to call LRREFM to return the logical array LOGMSS. LOGMSS is set to TRUE if a datum is missing, otherwise it is set FALSE. This array is used within the main program to test for data which is missing. Example from MLPHARE;
120 CONTINUE ... C ************************** CALL LRREFF(1,SSQOL4,ADATA,EOF) C ************************** C IF (EOF) GO TO 130 C C LOGMSS(I) returned FALSE if no magic_no set or if ADATA(I) = magic_no C C ************************** CALL LRREFM(1,LOGMSS) C ************************** ... C Label 120: read the next reflection C First check for MNF set IF (LOGMSS(4) .OR. LOGMSS(5) ) GO TO 120 C Now do the old style check, on SIGF = 0.0 IF (ADATA(4).LT.0.000001) GO TO 120 FPRO = ADATA(4) SIGFPRO = ADATA(5) CNotice that both F and SIGF are tested for MNF. Normally both F and SIGF are flagged as missing, or both are present, BUT it is possible to set FP and SIGF to refer to different data "pairs" in all CCP4 programs.
2. Manipulating data in order to print to standard output.
The second case is where you wish to print data on the default output stream (unit=6). Not all systems will be able to print a NaN without crashing. Therefore, you must reset the missing number flag in the data to a floating point number. To do this use RESET_MAGIC, this routine does not alter the MNF set in the MTZ header. It is not necessary to ensure that the MNF is an invalid value for all data columns because this is merely for display purposes. If you feel that this is necessary then see the fourth example. Example from MTZDUMP;
C C ****************************************** C VALML set to somethhing you can print eg -999.0 CALL LRREWD(MTZIN) WRITE (6,'(/,A,/)') ' LIST OF REFLECTIONS' ..... C C ************************* CALL LRREFL(MTZIN,RESOL,ADATA,EOF) CALL RESET_MAGIC(MTZIN,ADATA,BDATA,MCOLS,DUMMY,VALML) C ************************* ..... C DO 160 JDO160 = 1,3 IHKL(JDO160) = NINT(ADATA(JDO160)) 160 CONTINUE C .... WRITE (6,6022) IHKL, (BDATA(J),J=4,NCOLS) C
3. Writing MTZ output files
Writing MTZ files follows the same procedure as before. In the example below BDATA is output to MTZOUT. The input data is contained in ADATA which is transferred to BDATA for all data items that are not equal to MNF. Note that BDATA is initialised to the MNF using EQUAL_MAGIC. This procedure need not be followed in all cases, because ADATA could be written out again with some modifications, and then the MNFs would still be retained. (eg SCALEIT). Example from MLPHARE:
C C First fill the BDATA array with missing number flags. C ******************************* DO 10 JJ = 1,MaxMTZCols CALL EQUAL_MAGIC(1,BDATA,MaxMTZCols) 10 CONTINUE C ******************************* C C Begin filling BDATA(.) copying elements from ADATA(.) if they are not missing BDATA(1) = ADATA(1) BDATA(2) = ADATA(2) BDATA(3) = ADATA(3) IF(.NOT. LOGMSS(4)) BDATA(4) = ADATA(4) IF(.NOT. LOGMSS(5)) BDATA(5) = ADATA(5) ... C---- Now transfer phib and fom to BDATA(.) providing they have been calculated C---- the value -99999.0 is used as an internal check on this. C IF(ABEST.NE.-99999.0) THEN BDATA(IPHIB) = RADDEG*ABEST BDATA(IPHIB+1) = FOM ENDIF C ..... C Any BDATA entry which has not been inserted will have remained as a MNF C ******************* CALL LWREFL(1,BDATA) C ******************* C
4. Processing data from several input files, eg in SORTMTZ.
This procedure will have to be adopted for other library functions such as SRTRLS which you may not be able to handle NaNs. In this case there may be different MNF values in the different input files. The output file will contain the MNF flag found in the first input file.
This requires that the MNF in the data is reset to a floating point number. (In SORTMTZ MNF is reset to -999999.0 and this value is checked against all input data. If a datum is found to have a value more negative than this the program terminates). This is an example from SORTMTZ:
C DATA .... /,VRSET_MAGIC/-999999.0/ ... C C---- Check to see if VRSET_MAGIC missing value flag is acceptable C C =========================================== CALL LRINFO(INFILE,VERSN,NCOL,NREFL,RANGES) C =========================================== C DO 20 I=1,NCOL IF (RANGES(1,I) .LT. VRSET_MAGIC) THEN WRITE(LUNOUT,'(A,I6,A,F10.1,/,A)') ' Data from file ', + INFILE,' is more negative than ',VRSET_MAGIC, + ' You must either scale down the data or change VRSET_MAGIC' CALL CCPERR(1,' *** ERROR: Program Terminating') ENDIF 20 CONTINUE ... if (first_file) then C---- Initialise sort, set parameters, etc ........ C C C Find the MNF for first file to save for output file C C ======================================= SETVAL = .FALSE. CALL SET_MAGIC(INFILE,VAL_MAGIC,SETVAL) C ======================================= ... C Cycle around many hklin files ; C the internal MNF may be different for different files C---- Open next input file C C ======================================= CALL LROPEN(INFILE,FILNAM,IPRINT,IFAIL) C ======================================= ...... C---- Read reflection records C C ==================================== CALL LRREFL(INFILE,SRESOL,BDATA,EOF) C ==================================== C IF (EOF) GO TO 60 C C---- Convert missing data from original MNF for this file to VRSET_MAGIC C C =========================================================== CALL RESET_MAGIC(INFILE,BDATA,ADATA,NCOL,DUMMY,VRSET_MAGIC) C =========================================================== C C---- Pass keys and record to sort C ISTAT = SRTRLS(ADATA) ... C When all files read and records passsed to sort routines. C set MNF to VAL_MAGIC, if there is more than one input file then MNF is C converted to that of the first one. C =================================================== SETVAL = .TRUE. CALL SET_MAGIC(OUTFIL,VAL_MAGIC,SETVAL) C =================================================== ..... C---- Now pick up records and output C ISTAT = SRTRET(ADATA) IF (ISTAT.EQ.ENDSRT) THEN GO TO 90 ELSE IF (ISTAT.EQ.0) THEN C C ...... C Basically you are seting BDATA = ADATA unless ADATA = VRSET_MAGIC C In which case BDATA = VAL_MAGIC C C ========================================================== CALL RESET_MAGIC(0,ADATA,BDATA,NCOL,VRSET_MAGIC,VAL_MAGIC) C ========================================================== C C---- Write out reflection record C C ========================= CALL LWREFL(OUTFIL,BDATA) C ========================= ..... C ========================== CALL LWCLOS(OUTFIL,IPRINT) C ==========================
One final point, changing the MNF in an output MTZ file is a two stage process. The MNF value in the header is changed by using SET_MAGIC. Then the MNFs in the data itself need to be converted with RESET_MAGIC.