WISE Software Interface Specification; QAoutput_icl04; (WSDC D-I105) Written by F. Masci, version 1.9, 05/28/2009 Interface Name: Level-1a frame QA metadata Type of Interface: QA metadata written to ASCII file in IPAC table format Read By: WISE QA Application Framework ___________ H. Brandenburg D. Kirkpatrick C. Gelino Dynamic (tempcal) calibration PL ________ H. Brandenburg Written By: ICAL ____________________________________ F. Masci DESCRIPTION: The Instrumental Calibration Pipeline (ICAL) shall generate a table storing metrics and diagnostics to support Quality Assurance (QA) activities downstream. The table shall be in the IPAC standard format. ICAL shall generate one of these tables per frameset. EXAMPLE TABLE: Below is an example of an output QA metadata table file from ICAL. The metrics are not necessarily realistic. Specific algorithms are described in the ICAL Subsystem Design Specification (document WSDC D-D018: http://web.ipac.caltech.edu/staff/fmasci/home/wise/sds-wsdc-D018-ical.pdf) See below for some notes and associated graphics files. \ WISE QA metadata for instrumentally calibrated frame \ Generated by instruframecal, v.3.7 on 2009-02-24 at 10:04:55 \ Definitions: \ msk: pixels in frame-processing/bad-pixel mask \ int: pixels in 'active' region of intensity frame \ unc: pixels in uncertainty frame (active region) \ ref: pixels in reference region of intensity frame \ Metric units (excluding enumerates): native intensity image units \ FITS file products: \ 1: /wise/fmasci/testdata/instrucal/NedSim_Nov08//outputs/01301d001-w*-int-1a.fits \ 2: /wise/fmasci/testdata/instrucal/NedSim_Nov08//outputs/01301d001-w*-unc-1a.fits \ 3: /wise/fmasci/testdata/instrucal/NedSim_Nov08//outputs/01301d001-w*-msk-1a.fits \ |name |band |hdrname |type |value |comment | |c |i |c |c |c |c | ical:refNum 1 - i 16320 ref pixels: Number of reference pixels ical:refMin 1 - r 0 ref pixels: Minimum pixel value ical:refMax 1 - r 501 ref pixels: Maximum pixel value ical:refMean 1 - r 129.278686523438 ref pixels: Mean pixel value ical:refMedian 1 - r 130 ref pixels: Median pixel value ical:refSigMADMED 1 - r 2.9652042180623 ref pixels: Sigma scaled from Median Absolute Deviation from Median ical:refSigLTMADMED 1 - r 2.9652042180623 ref pixels: Sigma from Lower-Tail Median Absolute Deviation from Median ical:refMed16ptile 1 - r 3 ref pixels: Median - 16th percentile ical:refMed84ptile 1 - r 2 ref pixels: 84th percentile - Median ical:mskNumGood 1 - i 1022009 msk pixels: Number of 'good' (unmasked) frame pixels in active region ical:mskNumTotBad 1 - i 10149 msk pixels: Number of 'bad' pixels (with mask value > 0) ical:mskNumStaticBad 1 - i 9952 msk pixels: Number of static 'bad' pixels (from ground cal and updated long-term) ical:mskNumDynaBad 1 - i 189 msk pixels: Number of dynamically masked pixels in ical processing ical:mskNumSat 1 - i 8 msk pixels: Number of saturated pixels (at any SUR sample) ical:intNumNaN 1 - i 0 int pixels: Number of NaNs and |frame values| > machine maximum ical:intNSpikeLo 1 - i 97 int pixels: Number of low-tail outlier spikes ical:intNSpikeHi 1 - i 96 int pixels: Number of high-tail outlier spikes ical:intMin 1 - r -154.509796142578 int pixels: Minimum pixel value ical:intMax 1 - r 38933.1328125 int pixels: Maximum pixel value ical:intMean 1 - r 4.05855798721313 int pixels: Mean pixel value ical:intMedian 1 - r 0.0263664722442627 int pixels: Median pixel value ical:intFuzMode 1 - r -0.233849287033081 int pixels: (Fuzzy) Mode pixel value ical:intStdDev 1 - r 202.098526000977 int pixels: Standard (RMS) Deviation from mean (unbiased population estimate) ical:intSigMADMED 1 - r 2.84016707024665 int pixels: Sigma scaled from Median Absolute Deviation from Median ical:intSigLTMADMED 1 - r 2.40194108580032 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median ical:intSigLTMADFM 1 - r 2.64950231616642 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Fuzzy Mode ical:intI16_84Range 1 - r 2.86092338562012 int pixels: Sigma from (84% - 16%)/2 range ical:intMed16ptile 1 - r 2.56534240722656 int pixels: Sigma from median - 16th percentile ical:intMed84ptile 1 - r 3.15650436401367 int pixels: Sigma from 84th percentile - Median ical:intMod16ptile 1 - r 2.30512664794922 int pixels: Mode - 16th percentile ical:intMod84ptile 1 - r 3.41672012329101 int pixels: 84th percentile - Mode ical:intRatMRange 1 - r 1.48222663875355 int pixels: Ratio: (84th percentile - Mode) / (Mode - 16th percentile) ical:intRatMedRange 1 - r 1.23044173562243 int pixels: Ratio: (84th percentile - Median) / (Median - 16th percentile) ical:intMeanCh1 1 - r 6.15206384658813 int pixels: Mean pixel value for amplifier channel 1 ical:intMedianCh1 1 - r 0.029117226600647 int pixels: Median pixel value for amplifier channel 1 ical:intStdDevCh1 1 - r 288.41748046875 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 1 ical:intSigLTMADMEDch1 1 - r 2.3522043446501 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 1 ical:intMeanCh2 1 - r 2.40662002563477 int pixels: Mean pixel value for amplifier channel 2 ical:intMedianCh2 1 - r 0.0713993310928345 int pixels: Median pixel value for amplifier channel 2 ical:intStdDevCh2 1 - r 41.6669273376465 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 2 ical:intSigLTMADMEDch2 1 - r 2.44848395942064 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 2 ical:intMeanCh3 1 - r 1.5121682882309 int pixels: Mean pixel value for amplifier channel 3 ical:intMedianCh3 1 - r -0.0114278793334961 int pixels: Median pixel value for amplifier channel 3 ical:intStdDevCh3 1 - r 23.5343971252441 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 3 ical:intSigLTMADMEDch3 1 - r 2.44867819661921 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 3 ical:intMeanCh4 1 - r 1.45700991153717 int pixels: Mean pixel value for amplifier channel 4 ical:intMedianCh4 1 - r 0.0220947265625 int pixels: Median pixel value for amplifier channel 4 ical:intStdDevCh4 1 - r 22.4599533081055 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 4 ical:intSigLTMADMEDch4 1 - r 2.29306256411432 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 4 ical:intMeanCh5 1 - r 4.67818689346313 int pixels: Mean pixel value for amplifier channel 5 ical:intMedianCh5 1 - r 0.030318021774292 int pixels: Median pixel value for amplifier channel 5 ical:intStdDevCh5 1 - r 203.332443237305 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 5 ical:intSigLTMADMEDch5 1 - r 2.28808680446796 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 5 ical:intMeanCh6 1 - r 1.08549439907074 int pixels: Mean pixel value for amplifier channel 6 ical:intMedianCh6 1 - r 0.00359678268432617 int pixels: Median pixel value for amplifier channel 6 ical:intStdDevCh6 1 - r 13.5652570724487 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 6 ical:intSigLTMADMEDch6 1 - r 2.47498399315972 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 6 ical:intMeanCh7 1 - r 6.49912357330322 int pixels: Mean pixel value for amplifier channel 7 ical:intMedianCh7 1 - r 0.0582516193389893 int pixels: Median pixel value for amplifier channel 7 ical:intStdDevCh7 1 - r 274.643615722656 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 7 ical:intSigLTMADMEDch7 1 - r 2.390568577357 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 7 ical:intMeanCh8 1 - r 5.82884645462036 int pixels: Mean pixel value for amplifier channel 8 ical:intMedianCh8 1 - r 0.0182328224182129 int pixels: Median pixel value for amplifier channel 8 ical:intStdDevCh8 1 - r 239.736221313477 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 8 ical:intSigLTMADMEDch8 1 - r 2.40688344159725 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 8 ical:intMeanCh9 1 - r 4.05000734329224 int pixels: Mean pixel value for amplifier channel 9 ical:intMedianCh9 1 - r 0.0318514108657837 int pixels: Median pixel value for amplifier channel 9 ical:intStdDevCh9 1 - r 156.231155395508 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 9 ical:intSigLTMADMEDch9 1 - r 2.35190211934568 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 9 ical:intMeanCh10 1 - r 1.18710827827454 int pixels: Mean pixel value for amplifier channel 10 ical:intMedianCh10 1 - r 0.00787723064422607 int pixels: Median pixel value for amplifier channel 10 ical:intStdDevCh10 1 - r 18.1743507385254 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 10 ical:intSigLTMADMEDch10 1 - r 2.45833491368517 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 10 ical:intMeanCh11 1 - r 1.65603005886078 int pixels: Mean pixel value for amplifier channel 11 ical:intMedianCh11 1 - r 0.0106725692749023 int pixels: Median pixel value for amplifier channel 11 ical:intStdDevCh11 1 - r 26.4674415588379 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 11 ical:intSigLTMADMEDch11 1 - r 2.3929681755781 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 11 ical:intMeanCh12 1 - r 4.61555957794189 int pixels: Mean pixel value for amplifier channel 12 ical:intMedianCh12 1 - r -0.0140678882598877 int pixels: Median pixel value for amplifier channel 12 ical:intStdDevCh12 1 - r 234.666732788086 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 12 ical:intSigLTMADMEDch12 1 - r 2.57753222026774 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 12 ical:intMeanCh13 1 - r 3.65248847007751 int pixels: Mean pixel value for amplifier channel 13 ical:intMedianCh13 1 - r 0.0989964008331299 int pixels: Median pixel value for amplifier channel 13 ical:intStdDevCh13 1 - r 89.9735336303711 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 13 ical:intSigLTMADMEDch13 1 - r 2.63147289772842 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 13 ical:intMeanCh14 1 - r 17.3562774658203 int pixels: Mean pixel value for amplifier channel 14 ical:intMedianCh14 1 - r 0.106661319732666 int pixels: Median pixel value for amplifier channel 14 ical:intStdDevCh14 1 - r 550.907653808594 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 14 ical:intSigLTMADMEDch14 1 - r 2.52798406063788 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 14 ical:intMeanCh15 1 - r 1.26541984081268 int pixels: Mean pixel value for amplifier channel 15 ical:intMedianCh15 1 - r -0.00885748863220215 int pixels: Median pixel value for amplifier channel 15 ical:intStdDevCh15 1 - r 18.5819358825684 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 15 ical:intSigLTMADMEDch15 1 - r 2.308693091291 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 15 ical:intMeanCh16 1 - r 1.50358474254608 int pixels: Mean pixel value for amplifier channel 16 ical:intMedianCh16 1 - r 0.0456792116165161 int pixels: Median pixel value for amplifier channel 16 ical:intStdDevCh16 1 - r 22.6797122955322 int pixels: Standard (RMS) Deviation from mean (unbiased) for amplifier channel 16 ical:intSigLTMADMEDch16 1 - r 2.32261595712775 int pixels: Sigma from Lower-Tail Median Absolute Deviation from Median for amplifier channel 16 ical:uncMin 1 - r 2.07140517234802 unc pixels: Minimum pixel value ical:uncMax 1 - r 42.9028778076172 unc pixels: Maximum pixel value ical:uncMean 1 - r 2.57031202316284 unc pixels: Mean pixel value ical:uncMedian 1 - r 2.52590227127075 unc pixels: Median pixel value ical:uncI16_84Range 1 - r 0.292872076034546 unc pixels: Inter-84th - 16th percentile range ical:uncRatLTMADMED_Med 1 - r 0.950923997780775 unc pixels: Ratio: intSigLTMADMED / uncMedian ical:uncRat16ptile_Med 1 - r 1.01561427629422 unc pixels: Ratio: intMed16ptile / uncMedian ical:uncLocScal1 1 - r 1.10157218493447 unc pixels: local ratio "robust RMS / unc estimate" for unc rescaling using quantile estimators ical:uncLocScal2 1 - r 0.984184001901196 unc pixels: local ratio "robust RMS / unc estimate" for unc rescaling using mode estimators ical:intFdynaflg 1 FDYNAFLG i 1 Usability flag (on 01301d001-w1-lincor-1a.fits) for dynamic flatcal ical:intTdynaflg 1 TDYNAFLG i 1 Usability flag (on 01301d001-w1-flatcor-1a.fits) for dynamic tempcal ~ ~ repeated for bands 2, 3 and 4 ~ ~ NOTES: * All frame statistics (unless specified) exclude prior-masked pixels, NaNs, and values exceeding machine minimum/maximum limits. * The various sigma (data-scale) measures are not redundant. It is prudent to measure some of the important metrics in several different ways since a specific algorithm may not be robust against unforeseen glitches and irregular behavior in the data. Each sample metric is also subject to uncertainty and bias (relative to the 'true' value of the underlying population), and it's important not to be mislead by any one of them. * We don't expect all metrics to be propagated to the final WISE QA Application Framework. Some of these will serve as internal health/sanity checks. * The number of channel statistics are different for bands 1,2 (HgCdTe arrays) and bands 3,4 (Si:As arrays). There are 16 amplifier channels for the HgCdTe arrays and 4 "quadrant" channels for Si:As. * The dynamic calibration usability flags: intFdynaflg and/or intTdynaflg are only written when ICAL is run in a mode that enables it to save intermediate FITS products for the downstream "DynaCal" pipeline. FILE NAME: Since ICAL generates one QA metadata table per frameset (i.e., all bands corresponding to a single frameId), the filename shall be: -meta-ical.tbl ANCILLARY QA PRODUCTS: The following plots and graphics are currently being generated. From ICAL pipeline: * Histograms of reference pixels and 'active' pixels * Row/column median cuts versus column/row pixel number * Plot of 1st and 2nd moment statistics versus amplifier channel number for bands 1,2 * Plot of 1st and 2nd moment statistics versus amplifier quadrant for bands 3,4 From QA Application Framework: * JPEGs of intensity frame pixels * JPEGs of frame-processing masks