IV. NEOWISE Data Processing

IV.3. WISE Moving Object Pipeline Subsystem (WMOPS)

Contents

IV.3.a. Introduction

The WISE Moving Object Pipeline Subsystem (WMOPS) was designed to detect Solar system objects in the WISE single-exposure data as quickly as possible during data processing, with particular emphasis on the detection of Near-Earth objects (NEOs). To facilitate ground-based recovery efforts and thus extend the observing arc of 8–12 detections beyond the span of a few days (on average), the detections and astrometry are reported to the community as soon as possible. In practice, WMOPS has reported the tracklets of candidate object detections within approximately three days of their detection times to the Minor Planet Center (MPC), the IAU-designated clearinghouse for the detections of small bodies in the Solar system.

The NEOWISE spacecraft and instrument functioning parameters presented their own challenges unique to IR observations from spacecraft platforms. Parallax motion is minimized by the terminator-following orbit of NEOWISE, making the distances to moving objects more difficult to determine, and objects with low-projected sky velocities more difficult to detect than from ground-based observatories. Designed to be sensitive to both slow- and fast-moving objects, WMOPS operates on the extracted source lists produced by the Scan/Frame Pipeline processing of each NEOWISE pole-to-pole scan. These source lists are astrometrically and photometrically calibrated, spurious detections of image artifacts are identified, and many of the previously known Solar system objects in the field of view of each NEOWISE exposure are associated with source detections.

WMOPS processing begins by first selecting sources with a flux measurement signal-to-noise ratio (SNR) above a 4.5-σ threshold that were not flagged as artifacts. Profile-fit measurements resulting in rchi2>4 were also rejected as these are likely cosmic rays or other artifacts, except in the case of detections with SNR>15-σ, where the rchi2 requirement is ignored. WMOPS then identifies and filters out "stationary" objects that repeated in position between scans. Linkages between pairs of non-stationary detections were then made using the FindTracklets routine (courtesy of the Pan-STARRS project; see Kubica et al. 2007 Icarus 189, 151-168), and then detection-pair linkage is performed using the CollapseTracklets routine (courtesy of the Vera C. Rubin Observatory/LSST project; Myers et al. 2008, DPS 40, 52.06) to generate lists of candidate moving object tracks. These steps are run several times, over adjacent overlapping regions of the sky. A database of detection pairs is kept during the processing, as well as a "legacy" database that is updated post-processing, which serves as the WMOPS "memory" between runs.

After each run, the candidate object tracklets are vetted using a combination of automated and manual quality assessment processes. Confirmed tracklets and visual magnitude estimates are then reported to the MPC. More than a critical and effective repository of reported observations, the MPC provides multiple services to the community for the verification and confirmation of tracklets. These services include vetting of candidate tracklet astrometry for realistic orbital solutions, posting of object ephemerides of high-priority objects for follow-up to the community via the NEO Confirmation page (along with multiple orbital solutions), linking of reported objects with prior discoveries, and continually updating object brightness estimates and object orbital elements based on newly linked observations. Regular reports are also made to the small body observing community in the form of electronic circulars. The above list of services that the MPC provides is not exhaustive, and it should be noted that the MPC plays a critical role in most Solar system object search programs.

Tracklets with fewer than 5 detections were not considered valid detections for the automated WMOPS search. Owing to the success of the NEOWISE processing up to the point of the source-list extraction, WMOPS is able to push to lower signal-to-noise thresholds and lower the minimum required detections in a tracklet, allowing for an increased detection rate. Note that WMOPS is an extremely large and complicated subsystem. This section provides a working overview of the subsystem design.

IV.3.b. NEOWISE-Reactivation Adaptations

The general architecture of WMOPS is nearly identical to what was run during the WISE prime mission, and is described in Section IV.3 of the WISE All-Sky Release Explanatory Supplement with the following important modifications:

• During the WISE prime mission, WMOPS was run at 3- to 4-day intervals, so that sufficiently long tracklets could be generated. However, for the reactivated mission, WMOPS is run instead at 2- to 4-day intervals (3 times per week), in order to decrease the time between the final detection of a rapidly moving object and the reporting of its tracklet to the MPC. This reporting latency was about 4 days (median) during the prime mission. During the reactivated mission, the median value of this interval is measured to be less than 2.5 days.


• During the NEOWISE Reactivation mission, improvements were made to the computation of the W1 and W2 frame-transient flags (w?frtr), so that improved transient pixel identification was propagated into the source extraction database. This improvement enables the usage of this flag in the stationary object rejection (SOR) step to eliminate many spurious noise detections from the input to the tracklet generation routines. For the NEOWISE Reactivation mission, WMOPS rejects detections fainter than a fixed SNR, usually 9, with nonzero values in either of the first two w?frtr columns.


• It was realized during the prime mission that a large χ² residual to the PSF-fit photometry, or rchi2, often indicated a spurious transient or noise-detection, rather than a true source detection. The exception to this occurred for detections with high SNRs, and particularly for those that may be extended, like bright active comets (faint active comets often have rchi2~1 since their central condensations dominate the signal). Hence, a ceiling for a source's SNR was introduced, above which the χ² was not utilized for rejection. This ceiling was lowered to a level of 15, which was more appropriate to the W1 and W2 detectors for the Reactivation mission.


• The number of background confusion sources and noise detections per Single-exposure frame greatly increases for W1 and W2 relative to W3 (Section II.2 of the All-Sky Release Explanatory Supplement). In crowded sky regions, the number of confusing sources overwhelms the WMOPS algorithm, with thousands of detections-per-frame. The surface density drops off quickly with increasing SNR, though. In order to still detect high-SNR objects moving through these fields, WMOPS increases the SNR threshold for consideration of detections with more than 250 detections-per-frame after SOR and initial detection quality filters. The SNR threshold is raised in steps of 0.1 until the number of detections-per-frame drops to a pre-defined level of 100 sources per frame. If the SNR cutoff exceeds 10, all detections from the frame are rejected outright.


• The legacy database keeps track of detections between WMOPS runs, and so facilitates the merging of detections from different runs into candidate tracklets. The legacy database spans an interval between runs. The detections within the interval are re-submitted for linkage in the next run. This interval was lengthened for NEOWISE processing so the chains of detections would extend beyond the 5-detections threshold with the shortened run-intervals.


• Previously processed NEOWISE frames that fell in regions of processing (globs) which also contain new, unprocessed, frames were treated as new input and processed with the new frames to avoid failure to grow tuples in the inter-run region.


• In the reports submitted to the MPC from the prime mission, moving object tracklets could in principle share detections and a given tracklet could even contain two different detections with the same time stamp. The MPC would filter and reject these mismatched detections. For NEOWISE Reactivation, a routine was added to eliminate duplicate detections between tracklets, as well as to eliminate extra detections in the tracklet with the same timestamp. Selections were made in this new routine based on the inclusion and rejections of the detections based on the minimization of preliminary great-circle orbit fit residuals, as well as the maximization of tracklet length.


• After several optimization tests, the minimum SNR threshold was set to 4.5 for NEOWISE Scan/Frame pipeline processing detections.


• WMOPS QA web evaluation products were optimized. Individual detections could be rejected from the tracklet during the web-based eyes-on QA vetting (Figure 1). Tracklet preliminary orbit fit residuals could be optionally re-calculated when detections were eliminated.

IV.3.c. Subsystem Overview

IV.3.c.i. Operational Description

WMOPS fulfilled one of the tasks of the original NEOWISE project, an augmentation of the WISE program funded by the NASA NEOO program to discover and characterize new and existing NEOs in the WISE images. In order to facilitate ground-based follow-up, WMOPS reports tracklets to the MPC within 10 days of the central detection time of each new-object track. These reports were made in the prescribed MPC format for astrometric observations from a satellite-based observing platform.

WMOPS was run at 2- to 4-day intervals, requiring WMOPS to complete processing in under 3 to 4 cluster-CPU hours. It was the goal of WMOPS to detect 80% of the objects that fulfilled the requirements for detections, namely exceeding an SNR threshold of 7 and having a minimum of 6 detections. Before WSDS v3.5 implementation during the prime mission, we achieved 90% detection on the automated processing (pre-eyes-on QA; see below), surpassing our original goal. With v3.5 we increased to 92% completeness. Our desired goal was also to achieve 90% reliability in the tracklets we report to the MPC. As follow-up with any set of tracklets can be spotty, and especially with the thousands of tracklets reported by WMOPS to the MPC each run, our only means of registering tracklet reliability was, and remains, by feedback from the Minor Planet Center. Each complete WMOPS run has fewer than 1% bad tracks on average.

To minimize processing time within the WSDS requirements, WMOPS works with detections, eliminating artifacts and stationary objects from the processing instead of subtracting a static sky from the images and generating a new source list from there. This, the detector characteristics, and the WISE spacecraft's survey cadence make WMOPS a uniquely designed subsystem. Beyond the linear-motion search algorithms themselves, it is these specifics, i.e. the way the surrounding software sifts through and filters input detections and output tracklets, that often determine the efficacy of any search software for a moving object survey. Figure 1 summarizes the basic flow of the WMOPS subsystem. WMOPS runs off of the products of the Scan/Frame Pipeline. In particular, there are dependencies with the Artifact Identification and Instrumental Calibration in addition to Single-exposure detection lists.

A scan list is usually provided for each WMOPS run. Before frame processing can commence, WMOPS needs to assess which frames have complete coverage to determine the stationary objects. HEALPix, a package specifically designed and tested to conduct computations in spherical coordinate systems, is used to find neighboring frames, and corner positions of nearby frames are converted to the pixel coordinates of the central frame. A recursive algorithm is employed to determine if and by what frames complete coverage of the central frames area is obtained. The covered frames (nearly all of the frames over the course of the survey have complete coverage) in each of the scans are then read and the detections filtered according to several criteria. For NEOWISE data, WMOPS only functions in W1 and W2. The broad criteria for rejection of detections on ingest are as follows: too low of signal-to-noise values in both W1 and W2, flagged artifacts of specified nature, specified extremely pathological PSF chi-squared fit variance, and excessively large numbers of non-stationary object detections-per-frame (see above). A critical step in the frame processing was the stationary-object-rejection (SOR), which initiates the spatial processing of the detections on a per-frame basis. The idea behind SOR is simple; detections from separate scans that do not move more than, or are co-located within, a minimum distance are rejected. The operational rejection limit for minimum moving-object motion is about 2 arcseconds, or a little less than the NEOWISE pixel size. To avoid wrap-around problems for the SOR step, WMOPS converts all positions (e.g. on the overlap frames) into the coordinates of the central frame being processed.

After SOR, the remaining detections are paired with each other according to a distance and time separation so that pairs representing hypothetical objects, which would have velocities greater than the fastest object, which could have 5 detections in the NEOWISE survey, are excluded. These pairs are then fed through the CollapseTracklets routine, which connects pairs into tuples of 3 or more detections based on speed, direction of travel and position. These tuples are then stored in a collection of flat files based on the frame that spawned the tuple. For a tuple to be included in the output of the per-frame processing it must have reached a length of at least 3.

Tuples are subsequently grown by repeated runs of CollapseTracklets on tuples from overlapping regions of sky called globs. For a tuple to be included in the output of a glob it must be of length not less than 5 detections. To permit an object to move from one glob to an adjacent, overlapping glob, no glob is ever processed at the same time as an overlapping glob, and output from a glob is used as input to all overlapping globs run after the former glob. At the conclusion of the processing of a glob, meta information about the glob is stored in the WMOPS legacy database, which then informs future WMOPS runs about previously processed globs whose output can be used as inputs to globs in said future run (as appropriately overlapped). Through this database, tuples can span not only spatial regions of processing in a single WMOPS run, but they may also grow to span multiple WMOPS runs.

The output tuples are then filtered to remove subset and duplicate tuples converted into tracklets (tracklet and tuple are sometimes used interchangeably) in the standard MPC reporting format and two files are generated and placed in a results directory: last.mpc and last.mpc.sids. These files contain the MPC-formatted output (see the MPC web site) and the source_id list for each track. Orbit fits then are generated using the MPC Initial Orbit Determination (MPCIOD) and Digest2 (which performs great-circle orbit fits) software provided by the Minor Planet Center in executable form for use by WMOPS, and the orbital elements and fit residuals of those tracklets that converge to physical orbits are placed in the last.els file. This completes the activities of the WMOPS routine, which as its last act, kicks off the wmopsqa routine.

Figure 1 - Summary flow of the WMOPS subsystem. The ingesting, glob processing, and formatting are done primarily by the WMOPS Perl-based software. The quality assurance is done in a partly automated fashion by the wmopsQA routine and by eyes-on vetting of tracklets by the QA team. In the next-to-last QA filtering step, final vetting on individual tracks is done on the QA web pages for tracks with poor residuals from the Digest2 routine provided by the MPC. Final products are generated using the wmopsMPCGen routine and the duplicate-rejection routine is run on the final MPC-formatted list before sending the astrometry to the MPC.

The wmopsQA routine does the final automated filtering and generates the CGI-script files, lists, plots and thumbnail products for the eyes-on QA steps. There is a run-summary page as well as individual tracklet pages (Figure 2) generated for the QA. These pages are generated using the first.cgi and track.cgi scripts. Every tracklet that is not automatically rejected by the automated filters has the full set of eyes-on QA data products generated in its own separate QA sub-directory. Plots are generated using gnu-plot scripts and output in JPEG format. Thumbnails are generated with a scaling of +7σ, -3σ scaling.

The list of rejected and accepted tracklets are kept in the qa_state.txt file, which is modified by the submit.cgi script, which, in turn, is initiated by a button on the tracklet-level QA page. The qa_state.txt file is essentially a modification of a .sids (as in last.mpc.sids) file with the vetting status label ("Accept," "Artifact Reject," "Review," or "Reject") of the tracklet pre-pending the .sids line entry. After the eyes-on tracklet review is completed, in a QA filtering step, products are generated using the wmopsMPCGen routine and a final vetting on individual tracks is done via the web products for tracks with poor residuals from the Digest2 routine provided by the MPC. The wmopsMPCGen routine creates the QA-vetted list of MPC-formatted tracks, and makes crude estimates of R-band magnitude for the objects to aid follow-up by the general community. A routine to eliminate duplicate usage of detections in different tracklets is then run on the final report and a single list of tracklets in MPC format is submitted to the MPC.

Figure 2 - Discovery of 2014 HQ124, a PHA which came within 3 lunar distances of the Earth, in the WMOPS QA page raw image arrays. Sky-plane motion charts and text-based data are not shown here, but were generated as wmopsQA products. Note the stacked image in the final column of the top two rows, and the individual detection vetting buttons. Thumbnail images were critical to the successful identification of new objects.

IV.3.c.ii. Caveats

The 90% completeness value from the prime mission referred strictly to those objects with 5 or more detections that:

• had flux measurement signal-to-noise values > 7 in one band,


• were not co-located with a background stationary object (e.g. a star) or a detector or image artifact,


• were not saturated or otherwise too bright for proper photometry.

The minimum tracklet length validation requirement and the minimum sky-plane projected velocity imposed by the SOR defined the range of motion to which WMOPS was sensitive. For current operations, the parallax motions below 5 arcseconds over 90 minutes (approximate time between WISE coverages near the ecliptic) corresponded to a distance of roughly 28 AU. The limit of five detections imposed an upper limit in that objects may have outpaced the survey cadence before the minimum number of detections could be obtained. Motions greater than the upper limits perpendicular to the scan direction were roughly 3.3°/day, considering both an irregular cadence pattern as well as an evenly progressing survey. The maximum speed of a confirmed object detected by WMOPS was 4.3°/day (Figure 3), but the component perpendicular to the scan direction was 3.22°/day. The issue of completeness for the particular population of Near Earth Objects is discussed in Mainzer et al. 2011. ApJ 743, 156.

Figure 3 - Histograms of the sky-plane velocities of the objects reported to the MPC during years 1 through 10 of the Reactivation mission. The velocities are median values across the detected tracklet and the right plot is shown in log-space to highlight the fastest-moving objects. The object with the highest sky-plane velocity was the NEO 2020 TK3, which in October of 2020 had a sky-plane velocity of 70.5°/day, and had a fortuitous geometry that allowed two detections to be recovered manually by Masiero et al. (2023, PSJ, 4,225) as it was passing through the overlap region between consecutive survey images. The fastest object detected by WMOPS was 2018 QT1, with an average projected sky motion of 4.3°/day, modestly exceeding the WMOPS theoretical "speed limit" of 3.3°/day.

Finally, the estimated visual wavelength magnitudes reported to the MPC, which may remain in their MPCAT-OBS data (see below), are estimated from IR fluxes and colors and are likely not accurate to better than 1 magnitude. The standard deviation of these values relative to the true magnitude is expected to be in excess of 0.5 magnitudes.

IV.3.c.iii. Finding WMOPS Objects

The fastest means of finding WMOPS observed objects, beyond the available IRSA search engines, is to search through the MPCAT-OBS archive to extract particular objects using the WISE observatory code, C51. These will return the track that was reported to the MPC. A description of the tracks and how to obtain them is provided in section II.4.e of WISE All-Sky Release Explanatory Supplement.

Note that in addition to objects identified by WMOPS, the MPCAT-OBS archive also contains detections of objects found in the NEOWISE single-exposure images via manual recovery. The process of identifying these objects and the list of objects found via this method are described in Masiero et al. 2018, AJ, 156, 60.

Newly discovered objects of interest, like NEOs, comets, and Centaurs, are issued Minor Planet Electronic Circulars (MPECs). These often report observations from follow-up observers, in addition to the spacecraft observations.

Last update: 18 March 2024