IV. NEOWISE Data Processing

IV.2. Scan/Frame Pipeline Updates

IV.2.a. Instrumental Calibration

Contents

IV.2.a.i. Overview

The instrumental calibration (ICal) of single-exposure frames from the 4-band (All-Sky) cryogenic phase of the mission was described in detail in section IV.4.a. of the All-Sky Explanatory Supplement. Incremental updates were then made to support the 3-band Cryo (VII.3.b.) and 2-band Post-Cryo (VIII.3.b.) Data Releases. Here we describe updates made to the pixel calibrations and the methods used to remove instrumental signatures from the raw W1,W2 band single-exposure images acquired by the reactivated NEOWISE mission (Dec. 13, 2013 onwards). The ICal processing flow is the same as that shown for W1,W2 in Figure 1 of section IV.4.a. in the All-Sky Explanatory Supplement. Cautionary notes for the NEOWISE single-exposure images are outlined in section III.2.

IV.2.a.i.1. Detector (Level-0 Single-Exposure) Trending

Before outlining our calibration procedures, we summarize some trending metrics that attempt to track general detector performance over the course of the reactivated NEOWISE mission, with comparisons to the original post-cryo phase (Sept. 29, 2010–Feb. 1, 2011). These metrics are derived directly from the raw (Level-0) data with no instrumental calibrations and no pre-processing applied. They are shown as a function of MJD (or date) and beam-splitter assembly (BSA) mount temperature in Figures 1 to 4. The scattering of values at high levels in the background and pixel-RMS levels in Figures 1 and 2 are due to moon-light contamination. The prominent, periodic increases in the W2 background and pixel-RMS levels are caused by temperature increases (see Figure 3a and Figure 4a).

The "hot" pixel trends (Figures 3a, 3b) were made using the dark (relative-bias) calibration products (section IV.2.a.v.) by counting the number of pixels therein that exceed a fixed threshold of 20 DN. This threshold was determined from examining the high-tails of the dark-pixel histograms. The noisy-pixel fractions (Figures 4a, 4b) were determined from thresholding stacked standard-deviation maps made from Level-0 frames (section IV.2.a.ii.). A fixed threshold of 30 DN was used on these stack-noise maps as inferred from histograms. Note that these thresholds are not those used to create the image-masks for production (i.e., that flag unusable pixels; see below). A large fraction of the hot (high-dark) pixels was found to subtract-out nicely using both the static dark-bias maps (section IV.2.a.v.) and the dynamic sky-offset calibrations (section IV.2.a.ix.). The noisy pixel fractions are closer to what one would mask and declare unusable. The hot pixel trends (Figures 3a, 3b) are normalized to the number of hot pixels at the end of the original post-cryo phase (Feb. 1, 2011). For reference, these numbers (using the same methodology and thresholds) are 132,930 for W1 and 174,976 for W2.

Overall, both the W1 and W2 band detectors performed just as well in terms of sensitivity as in the original post-cryo phase when all calibrations were applied (see below). The only exception is that more bad pixels were accumulated following reactivation of the payload from its warm state at the end of 2013. The largest discrepancies were seen with W2 where the number of hot and noisy pixels (Figures 3a, 4a) increased by a factor of ~ 1.8 on average, compared to ~1.4 for W1. This increase in the number of bad (primarily hot) pixels affected the overall Level-0 pixel RMS, particularly in W2 (Figure 2), but not necessarily our ability to remove them using the dynamic calibrations (section IV.2.a.ix.). The seasonal variations in the number of hot pixels is closely correlated with the changes in the BSA mount temperature (Figures 3a, 3b). The number of noisy pixels is also correlated with temperature, but it also appears to be affected by moon avoidance spacecraft maneuvers, the impact of which cause extraordinary excursions in noisy pixel counts, especially and more intensely as the BSA mount temperature rises (Figures 4a and 4b).

Figure 1 - Background level in Level-0 single-exposure frames as a function of MJD (Dec. 2014 - Dec. 2022). Backgrounds are pixel mode estimates in a relatively clean region on each array, i.e., with low numbers of bad pixels. Red dots are binned medians and horizontal black lines are median estimates from the last week of original post-cryo (Jan. 25–Feb. 1, 2011). Left: W1, right: W2. Click to enlarge.

Figure 2 - Pixel RMS values in Level-0 single-exposure frames as a function of MJD (Dec. 2014 - Dec. 2022). RMS values are robust estimates based on percentiles in a relatively clean region on each array, i.e., with low numbers of bad pixels. Red dots are binned medians and horizontal black lines are median estimates from the last week of original post-cryo (Jan. 25 - Feb. 1, 2011). Left: W1, right: W2. Click to enlarge.

Figure 3a - Number of hot (or "high dark") pixels relative to those at the end of the original post-cryo (Feb. 1, 2011) as a function of date where points are separated by time windows spanning 3 to 4 days. The top axis shows the corresponding average BSA-mount temperature for each time window. See text for details. Click to enlarge.	Figure 3b - Same as Figure 3a but as a function of BSA-mount temperature on the abscissa. The top axis shows the corresponding date for the middle of each time window. See text for details. Click to enlarge.

Figure 4a - Percentage of noisy pixels over each array as a function of date where points are separated by time windows spanning 3 to 4 days. The first time-point corresponds to the end of original post-cryo (Feb. 1, 2011). The top axis shows the corresponding average BSA-mount temperature for each time window. See text for details. Click to enlarge.	Figure 4b - Same as Figure 4a but as a function of BSA-mount temperature on the abscissa. The top axis shows the corresponding date for the middle of each time window. See text for details. Click to enlarge.

IV.2.a.i.2. Calibration Summary and Intervals

The detector (pixel) calibrations and parameters used in processing the NEOWISE image data are summarized in Table 1. All W1 and W2 calibrations (except for the non-linearity correction) were derived and tuned using the NEOWISE survey data. Also shown in Table 1 is whether the calibration is a static product (derived once and applied throughout the survey with no updates), period-dependent (i.e., progressively updated in response to changes in detector behavior), or dynamic (derived/applied during pipeline processing on a per-frame or per-scan basis).

Table 2 summarizes the calibration period intervals where a fixed set of calibrations were applied. This includes the specific products that were updated in each period since not all products were rederived. Other relevant tables that track calibration updates and/or detector performance, in general, are Table 3 (bad pixel history) and Table 5 (detector electronic gains and read-noise). There are more calibration intervals early in the NEOWISE survey (18 intervals prior to Feb. 7, 2014) due to the warmer thermal environment and gradual cool-down of the detectors following reactivation (see Figure 10 in section I.2.c.iii).

IV.2.a.i.3. Calibration Procedure, Tracking and Tuning

The detector (pixel) calibrations were periodically updated in response to temperature variations throughout the NEOWISE survey, both during the initial cool-down period following reactivation and then due to subsequent "seasonal" variations. For the latter, systematic trends were observed in the numbers of bad pixels (with lo/hi response, dark signal, and/or with a noisy output in general; see Figures 3 - 4). The decision on whether to update calibrations for a specific period was based on monitoring and analyzing the following four diagnostics:

1. A significant number of residual hot and noisy pixels started to appear in the single-exposure intensity images that were not corrected/masked using static calibrations from the existing/previous interval as well as the ongoing dynamically-derived calibrations.
2. Significant residuals started to develop in the relative residual low-spatial frequency relative responsivity maps as determined from spatially-binned differences in source photometry between NEOWISE and the earlier AllWISE Release Catalog.
3. Related to (i): the pixel RMS in single-exposure products (medianed over all exposures per scan) started to deviate significantly.
4. Robust estimates of the ratio of "image-derived pixel RMS" to "noise-model-derived pixel uncertainty" started to deviate significantly from unity. This indicated that the electronic gain and/or read-noise calibrations needed updating (section IV.2.a.iv).

Our (re)calibration procedure is as follows. Further details are given in the sections below.

1. Generate high-spatial frequency (pixel-to-pixel) relative responsivity maps (flat-fields) using raw Level-0 exposures via the "background-gradient" method (section IV.2.a.viii). These maps are then regularized by resetting abnormally low/high pixel values < 0.1 / > 1.8 to unity and tagging them in the frame masks with bit #3 or #4 (section IV.2.a.iii).
2. Generate dark (offset) relative bias-structure maps also using the raw Level-0 exposures (section IV.2.a.viii). These are then regularized by resetting high pixel values > 32,000 DN to the static bias value of 128 DN and tagging them in the frame masks with bit #4 (section IV.2.a.iii).
3. Generate RMS-stack maps from the temporal pixel-dispersion in stacked raw Level-0 exposures (section IV.2.a.iv). These are produced in tandem with the dark/bias calibrations in step 2. Several RMS-stack maps across several consecutive calibration intervals are then combined to produce a single effective RMS-stack map by retaining the maximum RMS value per pixel from all intervals. This gives a "conservative" RMS-stack map that stores information on the noisiest pixels over time, including those that may have been noisy (unstable) for at least one calibration interval in the past. These effective stack-RMS maps are used to generate or update two other calibration products: (i) static calibration masks for tagging noisy pixels (with pixel stack-RMS > 25 DN and bit #1; see section IV.2.a.iii) and (ii) the read-noise sigma maps (step 6 below).
4. Generate the "static" calibration image masks (for the calibration interval of interest) by combining all "bad" pixels from thresholding the products generated in steps 1, 2, and 3.
5. If warranted, compute and validate new electronic gains and read-noise sigmas using pre-processed single-exposure intensity products (see section IV.2.a.iv). The read-noise sigmas are used in step 6.
6. Generate the "static" read-noise maps (for the calibration interval of interest) using the outputs from steps 3 and 5 (see section IV.2.a.iv).
7. Using all new calibrations, process of order 20 recent scans (~ 5000 frames) through the scan-frame pipeline. This includes dynamic calibration. This test provides a first check of the calibration performance. In particular, we collapse all test scans to create residual low-spatial frequency relative responsivity maps determined from spatially-binned differences in source photometry between NEOWISE and the AllWISE Release Catalog. These maps are examined for excess (spatially extended) residuals.
8. If significant residuals are found in the residual low-spatial frequency relative responsivity maps from step 7, we augment the high-spatial frequency responsivity maps from step 1 by multiplying them with the low-frequency responsivity maps and renormalizing to create new flat-field calibrations. The goal here is to remove (or minimize) any low-spatial frequency residual patterns seen in the spatially binned photometric differences.
9. Reprocess the test scans from step 7 and regenerate the residual low-spatial frequency relative responsivity maps to validate the flat-fielding performance. If all is sufficiently flat (to 1% or better), we proceed to step 10, otherwise, we iterate steps 8 and 9 until any significant systematic residuals disappear. Note: this augmentation of the high-frequency responsivity maps using photometrically-derived responsivity maps was performed towards the end of year 1 of NEOWISE and continued throughout year 2. For most of year 1, any low-frequency responsivity residuals (typically of order 3 to 5%) were not corrected. See section IV.2.a.viii.1 for details.
10. Compute and update the pixel-uncertainty scaling factors (section IV.2.a.iv) from analyzing the pixel-RMSs and noise-model uncertainty estimates in the single-exposure products. If the gain and read-noise values were estimated correctly, these scaling factors should be close to unity for the calibration interval of interest. At times, however, these values drifted, and instead of re-deriving the electronic gain and read-noise (which are computationally expensive), we updated the pixel-uncertainty scaling factors instead.

IV.2.a.ii. Bad-pixel Masking and Saturation Tagging

Similar to other (NEO)WISE processing phases, the frame masks are initialized in the ICal pipeline by first copying all information from the "static" calibration mask for the specific band (procedure in section IV.2.a.i.3) into the first 8 bits of a 32-bit buffer (see Table 4). The dynamic bits are then assigned on a per-frame basis according to various conditions encountered as each processing step was applied, e.g., saturated and broken pixels, bad/unreliable linearity corrections, temporally-transient bad pixels, and spatial pixel-outlier spikes.

The number and percentages of bad (fatal) pixels tagged in the frame masks are summarized in Table 3. These pixels (along with the dynamically-set saturated pixels) are set to NaN in the single-exposure intensity and uncertainty frames during processing (see section IV.2.a.iii). These NaN'd pixels are recognized and omitted from the single-exposure photometric measurements downstream. These fatal pixels are predominately those tagged as generally "noisy" (or unstable) for the specific calibration interval (Figure 4 and section IV.2.a.i.3), but they also include permanently bad pixels tagged with a very low or high responsivity, dark current, or an abnormal non-linear response. For comparison, the number of bad (fatal) pixels towards the end of the original post-cryo phase (Jan. 25 - Feb. 1, 2011), assuming the same thresholds, was a factor of ~1.8 lower for both bands than reported in Table 3.

Figures 5a and 5b show the static calibration image masks for W1 and W2 respectively, for three intervals during year 1 of NEOWISE. The largest temporal changes were for W2, owing to its greater sensitivity to the ambient thermal (dark) background. This background was highest soon after the reactivation of the spacecraft in December 2013.

Figure 5a - W1 masks showing bad pixel locations that are set to NaN in instrumentally-calibrated single-exposure products for three different periods during NEOWISE: Dec. 13, 2013 (left); June 4, 2014 (middle); Dec. 9, 2014 (right). Click to enlarge.

Figure 5b - W2 masks showing bad pixel locations that are set to NaN in instrumentally-calibrated single-exposure products for three different periods during NEOWISE: Dec. 13, 2013 (left); June 4, 2014 (middle); Dec. 9, 2014 (right). Click to enlarge.

Bits 9 to 19 in Table 4 refer to the dynamically tagged saturation bits according to values encoded in the raw Level-0 frames set on board the payload (the DEB processing unit). This range reflects the readout sample number in the ramp where the signal first saturates, i.e., hits the maximum of 16,383 DN after A/D conversion in the FEB processing unit. The lowest bits, therefore, refer to more severe cases of saturation, as induced by brighter sources. Bits 9 to 18 refer to "soft" saturation while bit 19 refers to "hard" saturation, i.e., the source was so bright that all reads were saturated. Here, the Level-0 pixel values are fixed at the 128 DN bias level. Figure 6 shows the number of soft and hard saturated pixels during the course of the NEOWISE survey. The gaps (or breaks) between MJD ~ 56749.23 - 56767.46 are due to a spacecraft anomaly (see section I.2.c.iv). The horizontal lines show the corresponding median levels from the end of the original post-cryo (Jan. 25–Feb.1, 2011). These levels are more or less consistent with the NEOWISE trends.

Despite the unreliability of the on-board saturated-pixel tagging in the earlier phases of the mission (most notably during the original post-cryo phase), it worked remarkably well for NEOWISE following reactivation. This was related to having the FEB thresholds correctly set. We are yet to encounter instances of mis-tagged saturated pixels in the NEOWISE single-exposure images. For details on the approximate magnitude limits at which saturation sets in, see section II.1.c.iv.

Figure 6 - Number of hard and soft-saturated pixels per array versus MJD as inferred from the on-board (DEB) encoding in Level-0 frame data. Horizontal blue and red lines represent the median number of soft and hard saturated pixels respectively at the end of original post-cryo (Jan. 25 - Feb. 1, 2011). Left: W1, right: W2. Click to enlarge.

IV.2.a.iii. Bit-Mask Definitions

We made every effort to replicate the bit-mask definitions as used in earlier processing phases (4-band cryo; 3-band cryo; and 2-band post-cryo), however due to changes in detector behavior and advances in calibration methodology, there are small differences. The bit-mask definitions for the NEOWISE W1 and W2 single-exposure image masks are summarized in Table 4. As a detail, the pixels that are set to "NaN" (generally unusable) in the NEOWISE frame intensity and uncertainty products are those initially tagged with any of the following bits: 1, 3, 4, 6, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 (decimal bitstring = 1048154). Furthermore, the pixels that are excluded from the profile-fit photometry measurements in the NEOWISE Source Database are those tagged with any of the following bits: 0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 23, 24, 27, 28, 30 (decimal bitstring = 1504706175). Note that a majority of these bits are never set in the frame masks (= "reserved" in Table 4) and are therefore meaningless. This bitstring is replicated from earlier data releases.

IV.2.a.iv. Electronic Gain, Read-Noise, and Pixel Uncertainty Estimation

The pixel noise model is given by Eq. 2 in section IV.4.a.ii of the All-Sky Explanatory Supplement. The electronic gain and read-noise were derived in the same manner as for earlier phases of the mission (e.g., section IV.4.a.ii.1. of the All-Sky Explanatory Supplement). This entails performing a linear regression to the lower envelope of the distribution of robust frame pixel variance versus background; then estimating the electronic gain and read-noise from the fitted slope and intercept. Figure 7 shows an example of this regression using approximately 118,000 framesets towards the end of year 1 of NEOWISE.

Given the large number of frames needed to robustly estimate the electronic gain and read-noise (typically two weeks' worth of data), we estimated these over-broad intervals during NEOWISE that were usually not coincident with the calibration intervals in Table 2. The trends are shown in Figures 8 and 9 and tabulated with the corresponding calibration intervals in Table 5. The trends in Figures 8 and 9 are qualitatively similar to the temperature trends in Figure 10 of section I.2.c.iii when one matches up the times at which the electronic gain and read-noise were estimated.

Figure 7 - Robust single-exposure pixel variance versus mode using all frames in scan-range 54790a - 55289b (acquired Nov. 26 - Dec. 13, 2014). Red lines are robust linear regression fits to the lower envelope of the distributions and are used to estimate the electronic gain and read-noise. See text for details. Left: W1, right: W2. Click to enlarge.

Figure 8 - Electronic gain estimates as a function of date since the start of NEOWISE. Note that there have been no changes since April 2016. See text for details. Click to enlarge.

Figure 9 - Read-noise sigma estimates as a function of date since the start of NEOWISE. Note that there have been no changes since April 2016. See text for details. Click to enlarge.

The read-noise sigma estimates tabulated above are single values pertaining to the entire arrays (W1 or W2). To obtain a more accurate (or representative) calibrations as a function of pixel position on each array, we used the RMS-stack maps from the temporal pixel-dispersions (step 3 in section IV.2.a.i.3) to construct effective read-noise maps. A pixel value in this read-noise map is computed using the following equation:

(Eq. 1)

where the quantity inside the inner brackets (parenthesis) with subscript "≥0" denotes the difference between the stack-variance value for the pixel and the mode over all pixels. Differences < 0 are reset to zero and stack-variance values above the threshold used for masking noisy pixels (section IV.2.a.i.3) are reset to the threshold value itself. The last variance term in Eq. 1 is the actual read-noise variance estimate from Table 5 - a constant for each array. The read-noise map defined by Eq. 1 is used in the pixel-error model equation (Eq. 2 in section IV.4.a.ii of the All-Sky Explanatory Supplement).

The pixel uncertainty estimates were validated against robust measurements of the RMS fluctuation about the background in the corresponding intensity frames. We used an automated algorithm to select background regions that were approximately spatially uniform and had a moderately low source density. A trimmed lower-tail standard deviation from the mode was used as a proxy for the RMS. This measure was divided by the mode of pixel uncertainties over the same regions to compute a pseudo reduced-χ metric per pixel. This measure is shown as a function of MJD in Figure 10. Apart from a few excursions (processing glitches), this metric is mostly within 1±0.04 throughout the NEOWISE survey, implying the uncertainty model estimates tracked the noise fluctuations in pixel signals reasonably well. This is consistent with global analyses of the PSF-fit photometry to point sources where the reduced χ² values are on average ≈ 1.

The "RMS/UNC" ratios shown in Figure 10 were sometimes used to rescale the pixel-error model equation so that pixel uncertainty estimates therefrom were more-or-less consistent with local RMS values in the intensity frames. This was used for ad-hoc temporary adjustments to the error model to avoid re-deriving electronic gains, read noises, and accompanying read-noise maps since these were computationally expensive. These "RMS/UNC" ratios are what define the "uncertainty model scaling parameters" in Tables 1 and 2.

Figure 10a - Medianed ratio of robust pixel RMS within specially selected frame region to mode of pixel uncertainties from noise model in the same region as a function of time over year 1 of NEOWISE. The vertical red lines mark the boundaries of the calibration intervals defined in Table 2. See text for details. Click to enlarge.	Figure 10b - Same as Figure 10a but for year 2 of NEOWISE. Click to enlarge.
Figure 10c - Same as Figure 10a but for year 3 of NEOWISE. Click to enlarge.	Figure 10d - Same as Figure 10a but for year 4 of NEOWISE. Click to enlarge.
Figure 10e - Same as Figure 10a but for year 5 of NEOWISE. Click to enlarge.	Figure 10f - Same as Figure 10a but for year 6 of NEOWISE. Click to enlarge.
Figure 10g - Same as Figure 10a but for year 7 of NEOWISE. Click to enlarge.	Figure 10h - Same as Figure 10a but for year 8 of NEOWISE. Click to enlarge.
Figure 10i - Same as Figure 10a but for year 9 of NEOWISE. Click to enlarge.

IV.2.a.v. Dark and Relative Bias Calibrations

The dark (offset) calibrations are designed to capture relative spatial variations and structure in the detector bias. They are not intended to subtract an absolute level of the dark signal. The latter is difficult since the detectors are always exposed to the sky. The dark (relative bias) maps are generated from the raw Level-0 (L0) frames by first subtracting their median-pixel signals and then stacking (collapsing) the frames using trimmed-averaging on each individual pixel. Of order 7000-9000 filtered L0 frames (per band) spanning typically four consecutive days are used when creating the darks for a specific calibration interval. By filtered, we mean omitting frames that are within 35 degrees of the moon; less than 15 degrees of the galactic plane, and within 5 degrees of the SAA boundary. Anomalously high pixel values (> 32,000 DN) are tagged in the static bad-pixel calibration masks (sections IV.2.a.i.3 and IV.2.a.ii).

Figures 11a and 11b show the dark (relative-bias) maps for W1 and W2 respectively, for three intervals during year 1 of NEOWISE. It is difficult to discern any difference by eye since the changes in the hot (hi-dark) pixel distribution over the course of the survey are only several percent (e.g., see Figure 3a). It's important to note that these dark (offset) calibrations are treated as "static" for the duration of a calibration interval and perform reasonably well at removing the cold/hot (lo/hi-dark) pixels early on. As time progresses, they become less accurate. However, this is where the dynamic sky offsets pick up the slack, i.e., help mitigate the residual lo/hi dark pixels not removed by the static darks.

Figure 11a - W1 dark (relative-bias) maps for three different periods during NEOWISE: Dec. 15, 2013 (left); June 14, 2014 (middle); Dec. 10, 2014 (right). Click to enlarge.

Figure 11b - W2 dark (relative-bias) maps for three different periods during NEOWISE: Dec. 15, 2013 (left); June 14, 2014 (middle); Dec. 10, 2014 (right). Click to enlarge.

IV.2.a.vi. Channel-noise Corrections

An overview of the "basics" of the channel-noise correction algorithm as used to process frames for the All-Sky and 3-band Cryo Data Releases is given in section IV.4.a. of the All-Sky Explanatory Supplement. There we made extensive use of the "reference" pixels at the ends of each amplifier channel, i.e., which mostly contained the respective bias levels that correlated well with signals in the active regions of each channel. However, as we progressed into the post-cryo phase (including reactivation in late 2013), the reference pixels did not perform as well at predicting the underlying bias levels. We resorted to correct for the "bulk channel fluctuations" using the active pixels for each channel. This entailed computing the pixel mode in each channel and checking for significant offsets relative to the adjacent channels. If significant, the levels were then equalized across channels. We further improved the robustness of the algorithm following reactivation. This involved first regularizing an input frame by estimating and subtracting a robust Slowly Varying Background (SVB) and resetting high signals greater than some threshold to 0. This created an internal working image from which the channel corrections could be robustly estimated. These corrections were then applied to the original input frame. All parameters (20 in total per band) were re-tuned for NEOWISE. Figure 12 shows a blink animation of a W2 NEOWISE frame processed with and without the new channel-noise correction algorithm.

Figure 12 - Click for a blink-animation of a W2 frame processed with and without the new channel-noise correction algorithm optimized for NEOWISE (internal WSDS software delivery version 7.5).

The channel-noise correction algorithm performed extraordinarily well for NEOWISE, or at least much better than it did for the earlier processing phases. However, one rare (small) caveat remains for NEOWISE. Occasionally, when the detectors observed a large background gradient perpendicular to the detector read-out (amplifier) channels, the channel-noise correction algorithm could not reliably correct for the intrinsic channel-to-channel variations, resulting in very large residual gradients. This effect was most common in W1 due to its greater sensitivity to background gradients relative to the underlying ambient and zodiacal backgrounds, particularly when a bright off-axis source perpendicular to the scan direction was present since the W1 detector channels are aligned with the scan direction. There was a small update to the algorithm (a threshold parameter) on 5/20/2014 to mitigate this effect.

IV.2.a.vii. Linearity Calibration

We retained the same non-linearity calibration as was derived from ground test data and used to correct the 4-band, 3-band cryo, and 2-band post-cryo data in earlier processing phases. The formalism was described in detail in section IV.4.a.vi of the All-Sky Explanatory Supplement. The relevant correction formulae (per pixel) are given by Equations (18) and (20) therein.

It was very difficult to validate the accuracy of the non-linearity calibration for the W1, W2 NEOWISE frames. Our best diagnostic was from examining color-magnitude diagrams and looking for systematic deviations in the trends expected for known stellar populations. One complicating issue is the presence of a bias in the PSF-fit photometry of saturated sources in W1 and W2. Figure 13 shows examples of color-magnitude diagrams involving the 2MASS K_s band. Instead of classically narrow (and vertical) color distributions, these turn "wild" at magnitudes brighter than W1 ~ 8 and W2 ~ 7 (the approximate saturation limits). At first, we thought an erroneous non-linearity correction was to blame, but this does not appear to be the case. Turning off the non-linearity correction makes little difference, in fact, it leads to an increased scatter (red points) in the photometry. Furthermore, the behavior for saturated sources is such that their estimated fluxes become relatively brighter the brighter they are. If due to non-linearity, it would require a correction that reduces flux for bright sources. At the time of writing, this does not conform with the traditional behavior of MgCdTe detectors as characterized in the laboratory and on other facilities.

Figure 13 - Color-magnitude diagrams involving the 2MASS Ks band to explore the impact of the linearity corrections (on versus off). Detector non-linearity cannot explain the growing bias for fluxes brighter than the W1 and W2 saturation limits (~ 8 and ~ 7 mag. respectively). All NEOWISE photometry is PSF-fit photometry from the single-exposure source database. See text for details. Left: W1, right: W2. Click to enlarge.

IV.2.a.viii. Relative Responsivity (Flat-field) Calibrations

The high-spatial frequency (pixel-to-pixel) relative responsivity maps (or flat fields) were generated from the raw Level-0 (L0) frames using the "change-in-background" method. Here, we measure the change in intensity in a pixel (i.e., slope) in response to the variation in overall intensity on a frame as the astrophysical (e.g., zodiacal) background varies. For details see section IV.4.a.v of the All-Sky Explanatory Supplement. This "change-in-background" method helped mitigate incomplete knowledge of the absolute instrumental dark/bias level.

Of order 7000-9000 filtered L0 frames (per band) spanning typically four consecutive days are used when creating flat-fields for a specific calibration interval. This number was sufficient to fill the detector wells and obtain good signal-to-noise per pixel (typically 50 or more). By filtered, we mean omitting frames that are within 35 degrees of the moon; less than 15 degrees of the galactic plane, and within 5 degrees of the SAA boundary. Anomalously low/high relative responsivity values (< 0.1 / > 1.8) are reset to unity and tagged in the static bad-pixel calibration masks (sections IV.2.a.i.3 and IV.2.a.ii).

Due to the rapid cooling following reactivation, more (narrower) calibration time intervals were used early in the NEOWISE survey. Histograms of the derived relative-responsivity for the W1 and W2 arrays during this period are shown in Figure 14. Most of the variation during this period occurred for pixels in the high responsivity tail. The overall responsivity pattern on each array then stabilized around mid-February 2014. All the W1,W2 responsivity maps made during the first year of the NEOWISE survey are shown in Figures 15 and 16. Most of these were made for spot-checks and regular instrumental trending, but some (where pertinent) were used to construct new calibration intervals (e.g., see update history in Table 2).

Figure 14 - Histograms of the relative pixel-to-pixel responsivity as a function of date during the early cool-down period following reactivation of the detectors. See text for details. Left: W1, right: W2. Click to enlarge.

Figure 15 - Click to view a PDF file of all the W1 relative-responsivity calibration maps made during year 1 of NEOWISE. Image stretch is 0.97 (darkest) to 1.03 (brightest) in all maps.	Figure 16 - Click to view a PDF file of all the W2 relative-responsivity calibration maps made during year 1 of NEOWISE. Image stretch is 0.97 (darkest) to 1.03 (brightest) in all maps.

IV.2.a.viii.1. Performance of Responsivity Calibrations

To validate the performance of our relative spatial responsivity corrections in general, we periodically analyzed the source photometry from ~200 scans, or ~45,000 frames after filtering to avoid contamination from the moon, galactic plane, and/or SAA. The NEOWISE-extracted sources were then spatially binned and their photometry differenced with the photometry of matching sources in the AllWISE Data Release Catalog. This catalog was generated from co-adding (averaging) many dithered exposures and is thus expected to be immune to residual patterns in the relative responsivity on the scale of individual exposures. It, therefore, provides a good reference catalog. The NEOWISE - AllWISE photometric difference maps are then normalized and smoothed using a constant Gaussian kernel to remove the binning pattern and yield the low-spatial frequency relative responsivity maps.

If our relative responsivity corrections were perfect (to their achievable accuracy), the low-spatial frequency residuals should be close to unity everywhere. Figure 17 shows some typical low-spatial frequency residual responsivity maps we obtained during year 1 of NEOWISE. The bulk of the residuals are within 1±0.016 (or ±1.6%) for W1 and slightly less for W2. At times, they may have been as high as ±4%. Therefore, repeated single-exposure photometry across multiple exposures covering the same sky region is likely to show the same scatter.

Figure 17 - Residual low spatial-frequency responsivity maps created from stack-averaging spatially binned "NEOWISE - AllWISE" photometry differences using sources extracted from > 45,000 single-exposure frames acquired over one period in year 1 of NEOWISE. The color bars denote the relative size of the residuals. See text for details. Left: W1, right: W2. Click to enlarge.

The cause of these residuals could not be resolved for most of year 1 of NEOWISE. It appeared that the high-spatial frequency (pixel-to-pixel) relative responsivity maps described in section IV.2.a.viii could not capture throughput variations at low-spatial frequencies, perhaps induced by non-uniformities in the optical train. Towards the end of year 1 however, we decided to adopt a two-step calibration process whereby we augmented the high-spatial frequency maps by multiplying them with the low-frequency maps (using spatially-binned differenced photometry from a first processing pass) and renormalizing to create new effective flat-field calibrations. Figure 18 shows an example of the residuals following this augmentation for data processed early in year 2 of NEOWISE. The improvement is dramatic. The low-spatial frequency residuals in the single-exposure products are suppressed to better than ±0.6%. This method was used for the remainder of the NEOWISE survey.

Figure 18 - Residual low spatial-frequency responsivity maps created from stack-averaging spatially binned "NEOWISE - AllWISE" photometry differences using sources extracted from > 45,000 single-exposure frames acquired over one period immediately after year 1 of NEOWISE (i.e., into year 2). To be compared with Figure 17. The color bars denote the relative size of the residuals. See text for details. Left: W1, right: W2. Click to enlarge.

IV.2.a.viii.2. Responsivity Trending

Responsivity maps such as those in Figure 18 are now generated on a weekly basis to keep track of the positions and magnitudes of variations and to determine when calibration updates may become necessary. Figure 18a shows both the RMS and the minimum and maximum values of the relative responsivity during years 3 through 9 of NEOWISE. All quantities (min, max, RMS) are highly correlated between W1 and W2, indicating that the variations are due to optical effects and are generated upstream of the W1/W2 dichroic.

Figure 18a - RMS, minimum, and maximum responsivities measured over years 3 through 9 of NEOWISE from weekly responsivity maps such as Figure 18. RMS + 1.0 for W1 and W2 are shown by the solid lines, while the maximum and minimum responsivities are shown as dotted lines. The last recalibration from Table 2 is indicated by the vertical red line. Click to enlarge.

The magnitudes of the variations in responsivity depend strongly on the position in the focal plane. Figure 18b shows the RMS in responsivity over time in blocks of 128 x 128 pixels. Both W1 and W2 show a similar pattern, with relatively stable regions to the left of array center (where sigma ≈ 0.0005), and more strongly-varying regions at the corners of the arrays. The upper-right and lower-right corners are particularly volatile, with 1-sigma variations more than eight times larger. The responsivity variations in these regions over time are compared in Figure 18c. The relative responsivities show a clear annual periodicity, though the sense of the variations may be out of phase from one part of the array to another. Moreover, in the least stable parts of the arrays the relative responsivities can vary by more than 1% over a span of less than two months. It is neither desirable nor practical to recalibrate the photometry pipeline on such short time scales, and the responsivity variations will obviously contribute to the photometric noise floor. On longer time scales, averaging over the relative responsitivities (Figure 18d) shows there is very little bias from one part of the array to another, with sigma < 0.1% and the worst outliers at ≈ 0.2%.

Figure 18b - RMS in the responsivity variations over time in 128 x 128 pixel blocks in W1 (left) and W2 (right). The stretch extends linearly from 0 to 0.004, and pixels in the lower-right corners have sigma = 0.004.

Figure 18c - Variations in responsivity over years 3 through 9 of NEOWISE of pixels with 129 < x < 256, 384 < y < 512 (solid lines) and 896 < x < 1024, 1 < y < 128 (dotted lines). The last recalibration from Table 2) is indicated by the vertical red line. Click to enlarge.

Figure 18d - Mean relative responsivities, averaged over years 3 through 9, in 128 x 128 pixel blocks in W1 (left) and W2 (right). The stretch extends linearly from 0.998 to 1.002.

IV.2.a.ix. Dynamic Calibrations

The purpose of dynamic calibration (dynacal) is to capture instrumental variations and transients as close as possible to the frame observation time and then correct them. NEOWISE processing used the same dynacal methodology as in 4-band cryo for W1 and W2, as outlined in section IV.4.a.viii of the All-Sky Explanatory Supplement. Two dynamic calibrations are performed for NEOWISE: (i) sky offsets and (ii) the detection and tagging of temporal bad-pixel transients.

The sky offsets are computed using a single frame-step moving median along a scan of pre-calibrated frames that are dark subtracted, linearized, and flattened using "static" calibrations (see intervals in Table 2). The frame median is subtracted from each frame prior to median combining and frames contaminated by an excess of cosmic rays or with a high source density (e.g., close to the galactic plane) are omitted. Note that the actual frame where the sky offset is applied (subtracted) is omitted from the window on which the moving median is computed. This is to avoid "incestuously calibrating" a frame using information about itself (e.g., see the "histogram-spike anomaly" reported in early processing phases). The moving median is 37 frames long with a minimum of 29. Below this minimum, dynacal assigns the best calibration product closest in time to the frame in question. Frames at the start or end of a scan boundary (or within half a window length from a boundary) are assigned an "inner" sky-offset product that's closest in time.

The sky-offset products are intended to capture and correct spatial variations in the relative dark/bias level over frames, including cold and hot pixels not removed by the static dark (relative bias) maps (section IV.2.a.v). I.e., as a static dark loses its accuracy over time, the sky offset can still correct the spatial variations. This can be seen in Figures 19 and 20. These show the trend in the number of low/high-tail pixels in the sky-offset histograms (below/above thresholds of -20/+20 DN). Close to the start of a calibration interval (red vertical lines), the number of low/high-tail pixels is generally low since the static dark subtracts them reasonably well, however, late in an interval, the numbers can be high since the static dark is out of synch with the true state of the detector. The variation within an interval is also influenced by seasonal temperature changes (e.g., Figure 3a).

Figure 19 - Number of pixels in low tails (< -20 DN) of sky-offset pixel distributions (medianed over all frames per scan) as a function of time over year 1 of NEOWISE. The vertical red lines mark the boundaries of the calibration intervals defined in Table 2. See text for details. Click to enlarge.	Figure 20 - Number of pixels in high tails (> +20 DN) of sky-offset pixel distributions (medianed over all frames per scan) as a function of time over year 1 of NEOWISE. The vertical red lines mark the boundaries of the calibration intervals defined in Table 2. See text for details. Click to enlarge.

The second type of dynamic processing in ICal is the detection and tagging of temporal bad-pixel transients (bit #21 in Table 4). The intent here is to tag spatial outliers (low/high pixel spikes) that persist for N consecutive frames for the same pixel along a scan. The method is described in section IV.4.a.viii.3 of the All-Sky Explanatory Supplement. For NEOWISE, we assumed N = 5. This operation, therefore, combines both the spatial and temporal dimensions and excludes pixels that were already tagged by the static mask bits (bits 0...7 in Table 4). It is performed before the final detection and replacement of spatial outliers (glitches) described in section IV.2.a.x. The tagging of persistent transients is therefore somewhat redundant but conservative, i.e., it provides additional protection against outliers that escape removal by the dynamic sky offsets.

IV.2.a.x. Spatial-outlier (Glitch) Replacement and Masking

The detection of spatial outliers (low/high pixel spikes) relative to the local background level is accomplished using the same median-filtering approach as used in earlier processing phases, e.g., see section IV.4.a.ix of the All-Sky Explanatory Supplement. This is performed at the end of processing in the ICal pipeline but prior to setting "fatally-bad" pixels to NaNs. If an outlier spike (either low or high) is found, mask bit #28 is set in the accompanying frame mask (section IV.2.a.iii). These pixels are not declared unusable (i.e., "fatal") and hence are not NaN'd.

An additional detail for NEOWISE single-exposure processing is that every spatial outlier that's tagged in the mask is replaced in the intensity frames with a median of its neighboring pixels. Note that there's logic to avoid replacing spikes near/on real source signal. This is to avoid biasing source flux by inadvertently clipping the cores of bright sources. Therefore, given this replacement, a pixel tagged as a spatial outlier (bit #28) in the frame mask will not mean that you will see an outlier at this position in the intensity frame, however, you have the option of omitting this pixel from your analysis if the actual replacement was not optimal.

xi. Generic Pixel-Calibration Performance

To explore our overall instrumental calibration performance, we computed a robust measure of the pixel RMS per frame based on percentile differences and then computed a global median over all filtered frames per scan. The filtering avoided frames contaminated by the moon, galactic plane, and/or SAA. This scan-medianed RMS is shown as a function of MJD in Figure 21. Its intent is to infer the relative calibration performance throughout the NEOWISE survey. Also, it is worth noting that the pixel RMS in the NEOWISE single-exposure products is lower in general than the median values from the last week of the original post-cryo phase (Jan. 25–Feb. 1, 2011), i.e., the horizontal lines in Figure 21.

Figure 21a - Robust pixel RMS medianed over all calibrated frames per scan as a function of time over year 1 of NEOWISE. The vertical red lines mark the boundaries of the calibration intervals defined in Table 2. The horizontal blue and green lines (W1 and W2 respectively) are the median RMS values from the last week of original post-cryo (Jan. 25–Feb. 1, 2011). Click to enlarge.	Figure 21b - Same as Figure 21a but for year 2 of NEOWISE. Click to enlarge.
Figure 21c - Same as Figure 21a but for year 3 of NEOWISE. Click to enlarge.	Figure 21d - Same as Figure 21a but for year 4 of NEOWISE. Click to enlarge.
Figure 21e - Same as Figure 21a but for year 5 of NEOWISE. Note the change in scale from years 1-4. Click to enlarge.	Figure 21f - Same as Figure 21a but for year 6 of NEOWISE. Note the change in scale from years 1-4. Click to enlarge.
Figure 21g - Same as Figure 21a but for year 7 of NEOWISE. Note the change in scale from years 1-4. Click to enlarge.	Figure 21h - Same as Figure 21h but for year 8 of NEOWISE. Note the change in scale from years 1-4. Click to enlarge.
Figure 21i - Same as Figure 21a but for year 9 of NEOWISE. Note the change in scale from years 1-4. Click to enlarge.

Last update: 14 March 2023