Droop and its Manifestions

1. Overview

This document reviews the droop correction as implemented in version 6.8 of the Instrumental Calibration (ICAL) module and included in WSDS delivery version 3.5. Examples of testing on flight data are also presented. Droop is only seen in bands 3 and 4. The red boxes in Figure 1 show the location of the three droop correction steps in the ICAL pipeline.

Figure 1 - ICAL pipeline flow. Droop correction steps for W3,W4 are shown in red. Click to enlarge.

2. Background and Methodology

Droop manifests itself in two forms: (i) intra-quadrant splitting, usually bisecting a saturated source or region, and (ii) global quadrant-to-quadrant (Q-to-Q) effects leading to a depression of one or more quadrants followed by an increase (or rebound) in the quadrant signals in subsequent frames. During the rebound phase, the quadrants also become unstable at the location of bad-pixel clusters and the banding/split structure seen in the dark becomes apparent, lasting for 10-20 frames. The residual bands (in excess of those seen in the static dark calibration) are also referred to as stationary splits and can be corrected using the same algorithm as for intra-quadrant droop-splits. Both forms of droop have been confirmed to be additive in nature and it is assumed that the banding-split residuals are additive too so all these can be corrected using offset-matching. There is also a final (optional) refinement step for cases where a global Q-to-Q correction could not be computed. We describe the algorithms (in order of execution) below.

2.1 Intra-quadrant droop-split and residual banding-split corrections

Before outlining the steps, we note that this version of the algorithm corrects splits using exclusively the active pixels. An initial version attempted to use the reference pixels at the top and bottom of each quadrant and demarcated by the split locations. This approach was aborted since (i) splits are often associated with heavily drooped cases where a large fraction of the reference pixel values are truncated at zero (represented as values 32767 in the downlink, which means the DEB slopes are really < 0). This implies that any metric that uses the reference pixels (e.g., median) will be biased high. There are indeed some cases with a good fraction of usable reference pixels, but their occurrence is infrequent to warrant mixing algorithms. The number of usable reference pixels becomes even lower when multiple splits are present, since one must consider only those pixels within regions separated by splits. The situation is worst for W4 where there are only two rows of pixels to work with; (ii) Furthermore, as will become apparent below, optimal correction for splits (whether droop or residual-banding related), requires that metrics be computed off a pre-calibrated frame, e.g., one that has been dark and responsivity corrected. This is not possible for the reference pixels because there is no attempt to calibrate them in processing. The current calibrations do not preserve reference pixel information that can be used in any meaningful manner downstream.

1. We start with a raw L0 frame and create an internal regularized spit-detection image. This image is made by subtracting the static dark, dividing by the static responsivity map, and replacing bad pixels (using the static bad-pixel mask) with a median of the respective quadrants. The latter is needed since the bad pixels are usually represented as saturated/high values in the input frame and these interfere when trying to search for droop-related splits across real saturated sources downstream. It's important to note that this is a temporary working frame stored in memory (and can be optionally written to a FITS file) which serves only for detecting candidate splits and computing corrections for further dissemination. Actual corrections are applied to the input L0 frame for propagation downstream, as described below. The dark and responsivity corrections are needed because first, the strong edge vignetting in a L0 frame makes the computation of offset corrections close to a frame edge unreliable, and second, the banding-related split corrections are intended to be residual corrections on top of the static dark that will be subtracted further downstream. If we did not subtract the dark, we run the risk of removing the entire "static" banding structure using our split corrections and this will need to be re-factored back (in some cumbersome way) before the final dark is subtracted in the pipeline.
2. The active regions of each quadrant are extracted from the internal working frame. Next, we search for candidate splits in each quadrant. We first collapse the columns in a quadrant (or take a cut across them) by computing the quantiles corresponding to some fraction falow in the low-tail of the data distribution in each column. falow is a parameter specified in the wise-meta.tbl file and at the time of writing, falow=0.1. Note that falow=0.5 corresponds to the median. We choose to work with quantiles in the low-tail of the histogram because they are less biased by the presence of bright extended structure and sources. I.e., such measures will be closer to the underlying background. W3 and W4 are background dominated for the most part, but occasionally, bright extended structure can occur. A median is not as robust against such structure than some low-tail quantile and hence is more likely to over-estimate the droop correction. Therefore, low-tail quantiles are more representative of the background values and relative offsets derived there from are expected to better track horizontal shifts in the pixel distribution due to droop. An example comparing median and quantile differences across a split is shown in Figure 2. Given the noise, the real offset appears to be closer to the 10%-tile (0.1 quantile) difference of ~7 DN, and not the much larger median difference of ~54 DN.
3. Given the 1-D vector of column-quantile values from step 2 for a quadrant, we compute its first derivative (just the first difference array of pairwise values in the increasing x-direction). This vector is thresholded using the input parameter splithres to find the locations of all the candidate splits. Droop-related splits are retained by finding those splits which intercept at least one saturated pixel (with pixel values in the range 32753 <= value <= 32761 in the L0 frame). Splits associated with the residual banding are also retained by comparing the candidate split list with the known stationary banding-split locations. The idea here is to be complete in the split detections by setting splithres to be relatively low (e.g., 4 DN).
4. For a candidate split, the lower-quantile values in strips of width 7 pixels (at the time of writing) on either side of the split at a distance of +/- 2 pixels from the split location are computed. Note that these are still computed from the internal temporary working frame. These strips contain more pixels than the single columns used for the initial thresholding in 3 and therefore provide a more reliable measure to filter out spurious splits. For a given split, the quantile difference of its adjacent strips is then thresholded against a second (but higher) threshold parameter (postsplithres, e.g., ~6 - 10 DN) to determine whether the split jump-size is significant and worth correcting. Note that for splits falling within 7 pixels of a quadrant edge, only those available columns falling between the edge and the buffer distance of 2 pixels from the split location are used to compute the lower-quantile. For splits at or closer than the 2 pixel buffer, single column-by-column corrections are computed, similar to that used for the split transition region in step 6 below.
5. If a split survives after the post-thresholding step in 4, the quantile difference in adjacent strips is then used as the offset correction to apply to all active and reference pixels in the quadrant on one side of the split in the original L0 frame. It doesn't matter which side is corrected. I happened to pick the left side throughout. Reference pixels are also corrected for self-consistency since these are used to compute global Q-to-Q corrections below (see Section 2.2). Note that only pixel columns at distances > 2 pixels from the split are corrected since the split transition region is not necessarily a sharp step. The transition region needs special treatment (see next step).
6. Finally, we correct split transition columns which span a width of 5 pixels (+/- 2 pixels including the location of the split). For these columns, we use the original column quantile vector from step 2. For each column, offset corrections are computed using the low-tail single column quantile and the quantile in the strip on the uncorrected side of the split. The goal is to equalize the level in the split transition region with other regions in the quadrant.
7. If there is more than one split in the quadrant (e.g., other splits related to droop, banding-residuals, or both), we iterate on steps 4, 5, and 6 until all splits and transitions are corrected. To ensure this, the internal working temporary frame is self-consistently updated following each correction to ensure corrections for other splits do not become biased. Again, each split correction is applied to the input L0 active and reference pixels, and the process is repeated for each quadrant.

Figure 2 - Example of a droop-related split in the top left quadrant of frame 00630x242-w3. This compares the median and lower 10%-tile estimates in the histogram of pixel values in narrow strips on either side of the split. The nebulosity appears to have more of an adverse effect on the median. Click to enlarge.

2.2 Global inter-quadrant (Q-to-Q) droop corrections

Following the split corrections to equalize levels within quadrants, Q-to-Q relative offsets are then computed and applied to equalize levels over the whole frame. This step exclusively uses the reference pixels where possible (which may have been updated by the split correction step above). The correction is computed for each separate quadrant by first computing the median of all good reference pixels (above or below it) where "good" means those pixels with value < 32767 (where 32767 is equivalent to a DEB instrinsic slope value < 0). This median is only computed from the good pixels if their fraction in a reference region is > some threshold gfrac (where currently gfrac = 0.5). The 32767 values cannot be reset to zero and included in the median estimate since otherwise the reference signal will be over-estimated. We found that a good reference pixel fraction of > 0.5 minimizes such biases.

The median reference value for each quadrant is then differenced against some long-term median reference signal for that quadrant to compute a relative offset correction. These long-run medians are given by the input refbaseN parameters for W3, W4 (where N = quadrant 1,2,3,4). These baselines are stable over long periods and appear to represent the natural levels that the reference signals return to following a droop and/or rebound event (see Figure 3). The relative corrections are then applied to the active pixels in each respective quadrant. This brings them in line with what one would have observed if no droop or rebound event occurred.

If the fraction of good reference pixels is <= gfrac for any quadrant, no correction is applied to the active pixels. Instead, a flag is set to indicate this and propagated downstream to the optional droop-refinement step (see Section 2.3).

We close this section with a note on the order of application of the split and Q-to-Q corrections. The splits are corrected first since this removes an ambiguity on which side of a split to correct first. The split corrections are applied to both active and reference pixels in a self-consistent manner so that the updated reference pixels can be used in the Q-to-Q step as described above. Furthermore, this order removes a potential bias on a median reference signal if it turns out a split exactly bisects a quadrant straight down the middle, i.e, 50% of the quadrant pixels lie on one side. The median reference signal will lie somewhere across the split transition (which has some slope) and will not be representative of the reference signal in either half of the drooped/undrooped region. Initial equalization of levels across splits is therefore key before attempting the Q-to-Q correction.

Figure 3 - Run of reference pixel medians (top), corresponding active pixel medians before Q-to-Q correction (middle), and after correction (bottom) for all frames in a scan. Click to enlarge.

2.3 Optional final global droop refinement

If a global correction using reference pixels alone (as described in Section 2.2) could not be computed for any quadrant, e.g., because there were an insufficient number of good reference pixels (their fraction is < some threshold gfrac), we attempt to compute an offset using the active pixels in neighboring quadrants where a correction was possible. This step is only performed if the drpflag parameter is set to 1 in wise-meta.tbl. If set, this step is performed after all instrumental calibrations have been applied in order to minimize biases from quadrant-dependent instrumental residuals.

We first compute lower-tail quantile values corresponding to the falow input parameter (currently = 0.1) in strips of width 50 pixels for W3 and 25 pixels for W4 running up and down the center the frame. Figure 4 shows an example of a correction to be computed for Q1. The correction for this quadrant uses the strips colored solid red. If a Q-to-Q droop correction using reference pixels was possible for Q2 (i.e., its good pixels satisfied > gfrac), then the offset correction for Q1 (which is added to all active pixels of Q1) is given by the quantile difference: q2 - q1. If a correction was not possible for Q2, then we look at the bottom quadrant, Q4, and use the active pixels in its strip to compute the offset correction for Q1, i.e., q4 - q1. The same procedure is used for the other quadrants that failed to be corrected in the initial global Q-to-Q step. E.g., for Q3, the correction sought for is first q4 - q3, but if Q3 failed to be corrected in the Q-to-Q step, we would try using q2 - q3.

If no correction is possible after two attempts to use two neighboring "good" quadrants (because their Q-to-Q reference pixel corrections were not possible), we give up and leave it at that. Note that we avoid using strips aligned in the horizontal direction above/below the quadrants since the existence of incorrectly corrected splits (either from droop or residual banding) could bias the results.

Figure 4 - Schematic for final (optional) droop refinement if reference pixel method for global Q-to-Q droop correction for any quadrant could not be performed, e.g., due to an insufficient number of good reference pixels. The solid red strips in Q2 or Q4 are used to correct Q1, more specifically, it's whichever of Q2 or Q4 was successfully corrected upstream using reference pixels.

Below we show a collection of W3 and W4 L0 frames with examples of droop. Further below are the corresponding calibrated/corrected L1b frames. The various flavors of droop shown here is by no means complete. There will be more surprises when v3.5 is deployed and operational.

3. Examples (W3)

L0 input frames:

Figure 5 - A managerie of L0 (w3) input frames with a variety of droop phenomena.

Calibrated/corrected L1b outputs:

Figure 6 - Calibrated frames corresponding to Figure 5. All processed using the same droop-correction parameters.

4. Examples (W4)

L0 input frames:

Figure 7 - A managerie of L0 (w4) input frames with a variety of droop phenomena.

Calibrated/corrected L1b outputs:

Figure 8 - Calibrated frames corresponding to Figure 7. All processed using the same droop-correction parameters.