Science data processing for WISE is carried out using the WISE Science Data System (WSDS) developed and operated at the Infrared Processing and Analysis Center, California Institute of Technology (IPAC). The WSDS is a hardware, software, operations and analysis system that converts raw WISE image data packets, engineering telemetry and ancillary information into the photometrically and astrometrically calibrated Image Atlas, Source Catalog, and ancillary data and metadata products.
The functional block diagram showing the elements and major interfaces of the WSDS is shown in Figure 1. The WSDS is comprised of six subsystems: INGEST, PIPELINES, WMOPS, FPG, QA and ARCHIVE. The high-level function of each subsystem is described below.
 |
| Figure 1 - Functional block diagram for the WISE Science Data System (WSDS) |
The WSDS INGEST subsystem receives science data packets, engineering telemetry and ancillary information from the WISE MOS, and assembles from them the Level 0 FITS-format images and metadata. Following their generation, INGEST then stages the Level 0 images and metadata for pipeline processing. INGEST operations were initiated automatically following the receipt of raw image, housekeeping and ancillary data that were sent following each WISE downlink pass. The INGEST subsystem is described in IV.2.
ii. PipelinesThe WSDS PIPELINES subsystem convert Level 0 imaging data and metadata into photometrically and astrometrically calibrated images, extracted source lists and produce corresponding metadata. The PIPELINES subsystem is comprised of two main components: the Scan/Frame Pipeline and the Multiframe Pipeline.
An abbreviated version of the Scan/Frame pipeline was run on a small amount of data following each WISE downlink pass for Quicklook quality assessment of science data.
The processing flow and elements of the Scan/Frame pipeline are described in IV.3.a.
The WMOPS subsystem is part of the NEOWISE enhancement to the primary WSDS that enables the discovery of previously unknown solar system objects, and provides a database for science users to pre-cover objects that are found after the end of the WISE mission. WMOPS utilizes the output from the Single/Frame pipeline to link non-inertial detections from the single-exposure WISE images made near the same point on the sky to form candidate moving object tracklets that are reported to the Minor Planet Center (MPC). The objective of WMOPS during WISE on-orbit operations was to report candidate tracklets within 10 days of the midpoint of the WISE observations. The MPC computed initial orbits and posted objects on their websites for follow-up and confirmation by ground-based observers. The operational flow and elements of the WMOPS subsystem are described in IV.3.b.
iii. Final Product Generation (FPG)
The WSDS FPG subsystem constructs the WISE Image Atlas and Source Catalog from the results of the Scan/Frame pipeline. FPG formally includes the Multiframe pipeline step, along with database and human analysis-intensive operations. The FPG system selects the subset of rows in the source Working Database that satisfy the WISE Level 1 Requirements for Catalog reliability and completeness. Source extractions that do not satisfy quality requirements are put into the Reject Table. For the Image Atlas, Source Catalog and Reject Table, FPG assembles final formats and generates any new derived header or catalog parameters that are not already in the Level 3 image and extracted source archives. The FPG system includes characterization, validation, and documentation of the WISE release data products. FPG for the WISE All-Sky Data Release is described in V.1.
The QA subsystem examines the output and collects metadata from the WSDS PIPELINES and generates concise reports summarizing science data quality at each stage of data processing. These reports are in the tabulated in web-based compilations that provide capability to drill-down to detailed image, graphical and tabular data. Elements and operations of QA subsystem are described in IV.6.
The ARCHIVE subsystem is a multi-tier system that stores and serves raw and processed mission data and metadata. A component of the ARCHIVE subsystem is integrated into the WSDS processing system and a component is integrated into the Infrared Science Archive (IRSA) at IPAC. The elements of the ARCHIVE subsystem in the WSDS are described in IV.7.
orbit - An orbit refers to the observations or block of survey data acquired between the times WISE observed the north ecliptic pole (NEP). The WISE orbital period is approximately 95.3 minutes. Orbit number 1 was defined to start with the fictitious NEP passage immediately prior to WISE launch. Thus, WISE was launched into orbit #1. The WISE All-Sky Data Release includes data taken from the start of full cryogenic survey observations in orbit 356 to the point at which the solid hydrogen was exhausted in the WISE outer cryogen tank in orbit 3550.
scan - A scan refers to the observations or block of survey imaging data acquired between two flight system maneuvers, such as reorientation of the instrument boresight near the ecliptic poles, or slews to acquire TDRSS for data downlink contacts. Because maneuvers often took place near the ecliptic poles, a scan is associated with the image data from approximately a half-orbit. However, scans can cover less or more than half of the arc between ecliptic poles. Scans are identified in the image headers, Single-exposure source database and metadata tables by the scan_id.
frame - A frame is the image data acquired in a single band from a WISE single-exposure. A single-exposure was acquired approximately every 11 seconds during WISE scans. The frame exposure times are 7.7 sec in W1 and W2 and 8.8 sec in W3 and W4. Because scans have different lengths, the number of frame exposures acquired per scan varies. A histogram showing the distribution of the number of frames per scan for all WISE survey scans is shown in Figure 2.
frameset - The set of four images (frames), one in each band (W1, W2, W3, W4), that are acquired simultaneously during each WISE single-exposure. The same region of sky is imaged in all four bands. The Scan/Frame pipeline processes framesets acquired during one scan.
coadd - Image data in a single band produced by combining multiple single exposure frames that cover the same region of the sky. Coadds are produced in the Multiframe pipeline by resampling, reprojecting and coadding single-exposure frames onto predefined spatial footprints.
coaddset - The set of four corresponding coadds, one in each WISE band (W1, W2, W3, W4), that are generated by combining multiple framesets. Coaddsets are generated by the Multiframe pipeline.
Atlas Tiles - A predefined grid of 18,240 regions on the sky onto which coaddsets are generated by the Multiframe pipeline for WISE data release products. Atlas Tiles cover 1.564°x1.564° (4095x4095 pix @1.375"/pix) and are built on an equatorial projection. The WISE All-Sky Data Release includes the Atlas Images and Survey Catalog produced for all 18,240 Atlas Tiles that cover the sky.
Atlas Image - An Atlas Image is the particular case of a coadd image built on the footprint of an Atlas Tile. The All-Sky Data Release intensity images are Atlas Images.
 |
| Figure 2 - Histogram showing the distribution of the number of frames acquired per scan during the WISE survey. |
The terminology used to describe WISE science data during the different
steps of downlink and processing is summarized in Table 1.
Many of the levels are transient, being generated as input for particular
processing operations. Level 0 images,
Level 1 images and extracted source lists
and the Release Image Atlas, Source Catalog and Reject
Table are archived.
| Data Level | Image Data | Extracted Source Data | Notes |
|---|---|---|---|
| Downlink Telemetry | CCSDS encoded downlink transfer frames. | N/A | Generated by spacecraft MUB, downlinked via TDRSS to White Sands Center. Input for the MOS High-Rate Data Processor. |
| Source Data Packets | Packetized, compressed science data and engineering telemetry, ancillary metadata and navigation files. | N/A | Produced by the WISE High-Rate Data Processor at White Sands Center and NAV system at JPL. Transferred to IPAC/WSDC following each downlink. |
| Raw Framesets | Decompressed raw pixel data in FITS format. 1024x1024 pix (W1, W2, W3), 512x512 pix (W4) at 16-bit/pix integer. Minimal header information. | N/A | Transient product. Generated during INGEST processing as step to Level 0. |
| Level 0 (L0) | Decompressed raw pixel data in FITS format. 1024x1024 pix (W1, W2, W3, W4) at 32-bit/pix floating point. Engineering telemetry, metadata and navigation information mapped to headers. | N/A | Produced by INGEST subsystem. Long-term archive product. |
| Level 1 (L1) | Instrumental pixel-calibrated frame-sets (dark-subtracted, reference pixel corrected, flat-fielded, illumination-corrected, linearized, droop-corrected, etc.). FITS format, 1016x1016 pix at 32-bit/pix floating point (reference pixels trimmed). Headers updated to reflect instrumental pixel calibration, reconstructed WCS values (including distortion corrections) and photometric zeropoint keyword values. | Calibrated single-exposure, band-merged source lists. Reconstructed equatorial coordinates, calibrated photometry, artifact flagging. | Generated in the Scan/Frame pipeline. Long-term archive in the Single-Exposure Image Archive and Source Working Database. Included in Release Ancillary Products. Level 1 images and source data may also be referred to as Level 1b (L1b). |
| Level 2 (L2) | Level 1 image that has been 2x upsampled and reprojected (rubber-sheeted) for 4-band registration and distortion correction. Reference pixels trimmed, 2032x2032 pix at 32-bit/pix floating point. | N/A | Transient products - not stored. Generated for source detection and QA. |
| Level 3 (L3) | FITS format images. 4095x4095 pix at 32-bit/pix floating point. Background level-matched, throughput scaled, 4-band registered, upsampled pixels. Updated WCS and photometric zero-point information in headers. | Calibrated, bandmerge coadd source lists. Reconstructed equatorial coordinates, calibrated photometry and uncertainties, artifact flagging, confirmation statistics. Generated by the Multiframe pipeline. | Generated on pre-defined Atlas Tile footprints in the Multiframe pipeline Archived in the Level 3 Image Archive and Source Working Database. These are inputs to the FPG subsystem to produce the All-Sky Release Image Atlas and Source Catalog. Level 3 images and source data may also be referred to as Level 3a (L3a) |
| Release Products | Image Atlas. For the All-Sky Data Release, these are identical to the Level 3 coadded images. | Source Catalog. Subset of Level 3 Source Working Database entries augmented with value-added column updates. | Produced during Final Product Generation. WISE Release Products are served via IRSA interfaces. |
Science data collected by WISE are processed in multiple stages depending on the phase of the mission. This multi-stage approach is designed to balance the desire to release data as quickly as possible to the community with the need to generate well-vetted products that fully exploit the scientific capabilities of WISE.
Each stage of processing is conducted using a different version of the WSDS. The first two version of the WSDS were used only to support pre-launch testing. The first "flight" version of the WSDS was v3.0.
Provisional processing of WISE science data was performed during IOC and the first several months of survey operations, to support testing and to monitor survey performance. This processing stage used the version 3.0 (v3.0) of the WSDS along with calibration products based on the best available ground test results. Provisional processing was discontinued on 20 April 2010. All of the early survey data that underwent provisional processing were subsequently included in first-pass processing. Data products from provisional were superseded by the later processing and are not released.
Following WSDS testing and tuning to actual on-orbit performance during IOC and the first several months of on-orbit operations, first-pass processing of the WISE science data was initiated using WSDS v3.5. This processing phase started on 20 April 2010, and used preliminary instrumental, photometric and astrometric calibrations that are derived from a combination of ground test, IOC and early survey data. All of the provisionally processed early survey data were rerun during first-pass processing to generate a uniform archive of images and extracted source lists. The WISE Preliminary Release Image Atlas and Source Catalog, and Single-exposure images and source database were generated from first-pass processing results. The NEOWISE Preliminary Post-Cryo Release Single-exposure Images and Source Database were also generated from first-pass processing.
Following the end of WISE on-orbit data acquisition and the Preliminary Data Release, all WISE cryogenic and post-cryogenic mission data underwent complete reprocessing through the Scan/Frame and Multiframe pipelines using an updated version of the WSDS (v4). This planned second-pass processing provided the opportunity to correct known deficiencies in the first-pass processing, add functionality that was out of the scope of first-pass processing, and improve calibrations, processing parameters and algorithms that were derived from analysis of payload, spacecraft and data performance over the life of the mission. The Moving Object Pipeline was not rerun during second-pass processing of the Full-Cryo and 3-Band Cryo data. However, WMOPS was rerun on all WISE/NEOWISE mission data in 2013, as described in VIII.3.
The WISE All-Sky Release Image Atlas, Source Catalog and Ancillary products are derived from the output of the second-pass data processing. Section IV.1.e describes some of the major WSDS differences and improvements between second-pass and first-pass processing that are reflected in the quality of All-Sky Data Release products.
The WISE 3-Band Cryo Release Image Atlas,
Source Working Database and Ancillary products are derived from
second-pass processing using WSDS v4. The pipeline modifications
made to handle the 3-Band Cryo data are described in
VII.3.
The NEOWISE Post-Cryo Release Single-exposure images and Source Database are derived from second-pass processing using WSDS v5. Modifications to the pipeline that optimized it for the Post-Cryo data are described in VIII.3.
Last update: 2013 March 1