Provisional Flats & Darks, and ground comparisons

Summary

o Below we compare the latest darks and flats derived from flight data versus
  those from MIC2 testing. Performance using comparisons of coadds made from
  re-calibrated frames and statistics are also given. The MIC2 test products 
  are from the second test (Mar/Apr 2009) and use the engineering model
  electronics.
  
o All flight flats appear to be remarkably stable. For early mission analyses,
  see: Flight Responsivity Maps

o The W1,W2 darks appear remarkably stable too. However some time-variability
  is seen in the W3,W4 darks when these are computed from different datasets.
  It is not known yet whether this is tied to the pedigree of the input
  data (e.g., astrophysical background variations), or is really instrumental
  in nature. In the end, we have a good set of working darks. Further 
  residuals are expected to be removed by dynamic calibration using
  "sky-offset" products. These products are created from a running block
  median along every scan, re-baselining to zero, and then subtracting these
  from the respective input frames. This functionality has not yet been
  validated.

o In a nutshell, the w3,w4 darks were derived from flight data using the
  following method. A simple detector observation model for pixel i can be
  written:

  O_i ~ G_i*S_i + D_i, where
  G_i = the gain (= relative flat here),
  S_i = true sky,
  D_i = dark+bias.

  A proxy for the sky S_i is taken to be the median raw-frame (L0) signal
  over all array pixels minus some (unknown) absolute dark level DL, i.e.,  
  S_i ~ L0 - DL.  
  The observation model then becomes:  
  O_i ~ G_i*L0 + C_i, where
  C_i = D_i - (G_i*DL).
    
  The flat (slope) G_i and intercept C_i are estimated from fitting to L0 data
  that follows the change in zodiacal background. The dark signal per pixel
  is then: D_i ~ C_i + (G_i*DL). For now, we assumed values of the absolute
  dark level DL from MIC2 testing. One can do a better job and determine DL
  using a global minimization fit. This is planned for later. Regardless of
  the absolute dark level, what's important here is to capture the dark
  variation over each array, e.g., the banding structure. Note that this
  model ignores the linearity correction. This will be folded in later.
  For now, the values of D_i from this method vastly improve the instrumental
  calibration for W3 and W4.

o Actual FITS files of products are in:
  phoebe:/Users/fmasci/01.20.10_provisional/

Flats

W1, left, right = flight, MIC2

W2, left, right = flight, MIC2

W3, left, right = flight, MIC2

W4, left, right = flight, MIC2

Ratio: flight/mic2: lefttop,righttop,leftbot,rightbot=w1,w2,w3,w4 [see color bar for scale]

Darks

W1, left, right = flight, MIC2

W2, left, right = flight, MIC2

W3, left, right = flight, MIC2

W4, left, right = flight, MIC2

Difference: flight-mic2: lefttop,righttop,leftbot,rightbot=w1,w2,w3,w4 [see color bar for scale]

Performance

The images below are W3 coadds made from L1 frames processed with ground calibrations [left] versus those processed using the latest flight calibrations [right] - mainly darks, flats, and a correction for droop. The galaxy at the center is NGC1097. The purpose here is to show the reduced instrumental residuals when the new calibrations are used. Further below is the W4 coadd comparison. Each coadd is of the same dimension of a typical Atlas Image (~1.56 degrees on a side).

For additional testing of new vs. old flats on individual frames, see the animations at the end of: Flight Responsivity Maps.

Some Statistics

The most useful comparisons here are the "MEDIAN 1-SIGMA UNC" values
(data derived uncertainties, not from a model). To get a feel for the
achieved accuracy, the relative error is given by:
100 * (MEDIAN 1-SIGMA UNC/MEDIAN)%.


DARKS:

FLIGHT:
w1:
  TRIMMED MEAN = 126.790045764026
  MEDIAN = 126.854270935059
  TRIMMED STDDEV = 0.495638944760485
  MEDIAN 1-SIGMA UNC = 0.0182627849280834

w2:
  TRIMMED MEAN = 126.831773534848
  MEDIAN = 126.888732910156
  TRIMMED STDDEV = 0.464302403097224
  MEDIAN 1-SIGMA UNC = 0.0139244571328163

w3:
  TRIMMED MEAN = 267.210005242453
  MEDIAN = 267.955871582031
  TRIMMED STDDEV = 12.2105976976511
  MEDIAN 1-SIGMA UNC = 2.18255567550659

w4:
  TRIMMED MEAN = 283.375599575752
  MEDIAN = 282.940551757812
  TRIMMED STDDEV = 5.94605463475657
  MEDIAN 1-SIGMA UNC = 1.08535397052765

--

MIC2:
w1
  TRIMMED MEAN = 126.731373656845
  MEDIAN = 126.79020690918
  TRIMMED STDDEV = 0.482345103471963
  MEDIAN 1-SIGMA UNC = 0.0847787484526634

w2:
  TRIMMED MEAN = 126.822081763272
  MEDIAN = 126.875679016113
  TRIMMED STDDEV = 0.462724204947326
  MEDIAN 1-SIGMA UNC = 0.0796581357717514

w3:
  TRIMMED MEAN = 265.42783784427
  MEDIAN = 265.169158935547
  TRIMMED STDDEV = 4.85252146790588
  MEDIAN 1-SIGMA UNC = 0.57173627614975

w4:
  TRIMMED MEAN = 282.206094879428
  MEDIAN = 281.670013427734
  TRIMMED STDDEV = 5.84730234722142
  MEDIAN 1-SIGMA UNC = 0.306508123874664

FLATS:

FLIGHT:
w1:
  TRIMMED MEAN = 0.990915229715353
  MEDIAN = 1.00114178657532
  TRIMMED STDDEV = 0.0450786573033271
  MEDIAN 1-SIGMA UNC = 0.00547813484445214

w2:
  TRIMMED MEAN = 0.98984747479498
  MEDIAN = 0.999574363231659
  TRIMMED STDDEV = 0.0729365545581228
  MEDIAN 1-SIGMA UNC = 0.00246186321601272

w3:
  TRIMMED MEAN = 0.999937674691483
  MEDIAN = 1.0005464553833
  TRIMMED STDDEV = 0.0157995175459051
  MEDIAN 1-SIGMA UNC = 0.00106819916982204

w4:
  TRIMMED MEAN = 0.999982556490866
  MEDIAN = 1.0022965669632
  TRIMMED STDDEV = 0.0174808210201523
  MEDIAN 1-SIGMA UNC = 0.00114717206452042

--

MIC2:
w1:
  TRIMMED MEAN = 0.994243404738707
  MEDIAN = 0.999867618083954
  TRIMMED STDDEV = 0.0296867655476971
  MEDIAN 1-SIGMA UNC = 0.000689558743033558

w2:
  TRIMMED MEAN = 0.993606854545473
  MEDIAN = 0.99871826171875
  TRIMMED STDDEV = 0.0518478689763131
  MEDIAN 1-SIGMA UNC = 0.000801462447270751

w3:
  TRIMMED MEAN = 1.00144394199958
  MEDIAN = 1.00002634525299
  TRIMMED STDDEV = 0.0135510509124029
  MEDIAN 1-SIGMA UNC = 0.000152157081174664

w4:
  TRIMMED MEAN = 0.997730377076971
  MEDIAN = 1.0010598897934
  TRIMMED STDDEV = 0.0160948322558013
  MEDIAN 1-SIGMA UNC = 0.0000961642144829966



Last update - 20 January 2010
F. Masci - IPAC/Caltech