|
New Darks & Flats for 1st pass processing | |
o Below we summarize new w3,w4 darks and w1-w4 flats to support first pass
processing of survey data with version 3.5 of the WSDS. Comparisons to
provisional flight darks and flats deployed on Jan 10, 2010 are also made.
o Due to updated software and calibrations on ops as of ~Feb 4, flats
and darks were made using scans observed from ~Feb 4 to Feb 24. 28117
frames were used for w1,w2 and 3920 "cleanish" frames for w3,w4.
By "cleanish", we mean relatively clean of long term latents, moon
artifacts, and at >3000 sec from anneals.
o The flats (and darks made using a self-calibration method therefrom)
used frames corrected for non-linearity (all bands) and "global" quadrant
droop (for w3,w4). The ground non-linearity calibration was used. The old
(provisional) flight flats were made using exclusively raw L0 frames.
o The classic stacking method was used to make w1,w2 flats and the "slope"
or change-in-zodi method was used for w3,w4.
o The new flats for all bands differ by no more than ~1.5% at the pixel
level from the old flats. The w3 flat contains a new "mottled" structure
not seen in the earlier provisional flat (see figs below). To ensure this
mottling is genuine (i.e., not due to the flat generation method), we
recreated the w3 flat using the stacking method. We also fiddled with the
outlier rejection parameters. The structure was still visible. We then split
the input frame list into 4 lots of ~1000 frames, made subflats, and the
mottling structure was present in each. It appears that the relative
responsivity structure for w3 changed from late Jan/early Feb onwards.
It is not known whether this was gradual or abrupt. More analysis is needed.
o The mottled structure in the new w3 flat gives a residual of < 1.3% in the
pixel RMS when applied to early survey data (selected at <~ Jan 24).
This flat removes the mottled appeareance in later survey data and therefore
is definitely needed. A plan is underway to create time-dependent flats at
least for w3 (and possibly w4 if warranted) to support future reprocessing.
o In a nutshell, the w3,w4 darks were derived from flight data by applying
a delta-correction to the earlier provisional flight darks. First, the
provisional flight darks used the following method. A simple detector
observation model for pixel i was assumed:
O_i ~ G_i*S_i + D_i, where
G_i = the gain (= relative flat here),
S_i = true sky,
D_i = dark+bias.
A proxy for the sky S_i is taken to be the median raw-frame (L0) signal
over all array pixels minus some (unknown) absolute dark level DL, i.e.,
S_i ~ L0 - DL.
The observation model then becomes:
O_i ~ G_i*L0 + C_i, where
C_i = D_i - (G_i*DL).
The flat (slope) G_i and intercept C_i are estimated from fitting to L0 data
that follows the change in zodiacal background. The dark signal per pixel
is then: D_i ~ C_i + (G_i*DL). We assumed values of the absolute dark level DL
from MIC2 testing. Regardless of the absolute level, what's important here
is that we capture the dark variation, e.g., the banding structure.
For the new darks, the change-in-zodi method was initially used for flats,
but the input frames were first dark-subtracted (using the provisionals)
and then linearized. So instead of fitting to L0 data, we fitted to
linearized pixels L_i versus their median per frame L=< L_i >. The latter
is a proxy for the change in overall sky signal S_i ~ L:
L_i ~ G_i*L + deltaD_i;
Given that the input data was dark subtracted, the intercept deltaD_i
from this new fit is the correction needed for the old (provisional)
dark to derive the new dark:
D_i(new) ~ D_i(old) + deltaD_i;
This method removed some of the ugly latent signatures in the old
provisional dark (see figs below). This is because the input data
for this second round of calibrations was cleaner.
The new w3,w4 darks appear to contain more hot pixels. To check that these
are genuine and not glitches introduced by the above procedure, a median
stack image of all the corresponding L0 frames was made. Bad pixels should
show up at the same location as in the darks. Indeed, there was an
excellent correlation.
o The w3,w4 darks derived in this manner vastly reduce residuals when applied
to frames compared to frames calibrated using ground or the earlier
(provisional) flight darks. The new w3,w4 darks differ from the provisional
set by ~10-16%. This may seem large, but it's all coupled to the new flats.
We should mention that there is a final delta-dark correction in the
pipeline to capture and remove any residual bias structure. This is
achieved through the dynamic sky-offset calibration.
o FITS files of all calibration products are in:
phoebe:/Users/fmasci/03.02.10_v3.5/
 |
W3 is at bottom left. The ~0.5% residual mottling is from the new flat.
 |
Each subflat used ~1000 frames each - extracted in observation time-order from the master list of ~4000 frames.
 |
High-zodi background frames were purposefully picked to exacerbate flat-fielding residuals. Below are some zoom-ins to illustrate the impact of the W3 mottling in the new flat.
LEFT: processed "w3 late frame(02134a)/old flat";
RIGHT: processed "same frame/new flat".
 |
Figures below:
LEFT: processed "earlier w3 frames(from 00969a,00953a, 00926a)/old flat";
RIGHT: processed "same frames/new flat";
 |
Note the higher incidence of hot pixels in the new W4 dark. These are consistent with the input frames.
 |
The latent artifacts in the W3 difference image are from the old dark (see fig above). These are at a different spatial frequency than the mottling seen in the new W3 flat.
 |
The following were computed for an early survey w3 frame processed using the old (provisional) dark and flat. See further below for stats computed off the same frame using new calibrations. A zoomed-in "mottled" region equivalent to that shown in the figure under Section 2.3 was used.
FITS FILENAME = outputsW3_ical/00926a158-w3-int-1b.fits NUM PIX = 15872 NUM PIX AFTER TRIMM = 15869 TRIM THRESHOLD = 4.5 MIN = 2572.11059570312 MAX = 3820.29907226562 MEAN = 2676.50756835938 TRIMMED MEAN = 2676.42076397477 MEDIAN = 2676.37109375 MODE = 2679.93359375 UNBIASED STDDEV = 29.6681213378906 TRIMMED STDDEV = 28.1960008286655 MAD FROM MEDIAN = 27.9263816447817 MAD FROM MEDIAN OF LOWER TAIL = 27.7517343016158 MEDIAN - 16%-tile = 27.8077148437501 84%-tile - MEDIAN = 27.7557421874999 [84%-tile - 16%-tile]/2 = 27.781728515625The following were computed off the same frame and region but processed with the new dark and flat. The pixel RMS (using the TRIMMED STDDEV metric) increased by ~1.1%.
FITS FILENAME = outputsW3_icalFeb26cals/00926a158-w3-int-1b.fits NUM PIX = 15872 NUM PIX AFTER TRIMM = 15868 TRIM THRESHOLD = 4.5 MIN = 2556.89624023438 MAX = 27376872325120 MEAN = 3434736384 TRIMMED MEAN = 2673.14835566335 MEDIAN = 2673.23046875 MODE = 2676.68994140625 UNBIASED STDDEV = 305957535744 TRIMMED STDDEV = 28.5017989674174 MAD FROM MEDIAN = 28.5024464046751 MAD FROM MEDIAN OF LOWER TAIL = 28.7701183430609 MEDIAN - 16%-tile = 28.2336523437498 84%-tile - MEDIAN = 28.0895507812502 [84%-tile - 16%-tile]/2 = 28.1616015625