previous page   back to table of contents   next page

V. AllWISE Data Processing

V.3. Multiframe Pipeline Updates

V.3.b. Source Extraction


V.3.b.iv. Iterative Background Estimation

WPHOT processes MDET detections in approximately descending order of S/N based on the flux in all bands. This order is approximate because WPHOT may overrule MDET in estimating the overall S/N, but in general the idea is to process the brightest sources before nearby fainter sources so that the flux of the former sources can be subtracted from the individual frames before processing the latter.

This is consistent with the passive-deblending design. The primary blend component, when it is not the only source in the fitting region, may be influenced by nearby fainter sources, so that by processing all sources within the fitting region simultaneously, the primary component emerges unscathed. The portion of the blend solution corresponding to nearby fainter blend components, however, may neglect other nearby sources outside of the primary's fitting region, hence the solutions for the passive-deblend components are discarded, and those sources are left to come through the input queue later on their own, when the fitting region will be centered on them, and their own nearby fainter sources can be fit simultaneously.

For this to work, the flux from a primary component must be subtracted from the input measurements so that it is not there to disturb the fainter sources when their turn arrives for primary-component processing. This has been the design in previous versions of WPHOT and remained in the current design, but as mentioned below, it was accidentally subverted in AllWISE. Nevertheless, the design involved flux subtraction after source extraction, and this has implications for the background estimation for sources processed subsequently.

In the AllWISE version of WPHOT, after flux subtraction, the background is re-estimated. Previous failure to do this was identified as the source of flux underestimation, i.e., the background was overestimated in some cases because the flux from brighter sources was still present in the background estimates despite having been subtracted from the frames themselves.

Initial testing showed that this recursive background re-estimation removed the source flux underestimation. Unfortunately, when the dual solution was implemented, an undetected typographical error was introduced in the code that was not caught by the compiler's automated checking, and its influence was subtle enough not to raise critical alarms during operations tests. The effect of this was essentially to short-circuit the flux subtraction. An uninitialized variable caused a failure in the identification of which frames needed to be processed, and as a result, the frames were identified on a random basis with about 90% on average being missed.

This failure to remove flux from previously processed sources was first noticed in the QA analysis as doubly extracted sources, i.e., a given source could have another source at almost the same position with almost the same fluxes. This turned out to be faint nearby sources migrating to the leftover flux in order to minimize the model chi-square, i.e., moving to the remains of a previously processed bright source reduced chi-square better than remaining at the MDET detection position and fitting the fainter flux there.

Because of ordinary photometric noise, the flux subtraction had never been perfect, so the possibility that a faint source might abandon its MDET location in order to migrate to leftover flux had always existed, but it should have been very rare, and the flux in the model fit should have been drastically reduced. Neither of these expectations were met.

Greatly expanded debugging capabilities were added in order to discover the reason for this anomaly. Eventually the generation of FITS-file postage stamps of the before-and-after flux subtraction revealed that in the majority of frames, the flux was not being subtracted at all. Unfortunately, the time needed to unearth the source of the anomaly was considerably longer than schedule pressure allowed for production runs of the pipeline, and the typographical error in the code was found almost at the end of the production processing.

This is the source of the "SSST" phenomenon ("Small-Separation Same-Tile"), and since flux subtraction occurs only on a small percentage of the frames, the background re-estimation was almost completely neutralized.


Last update: 2013 October 23


previous page   back to table of contents   next page