IV. WISE Data Processing

1. Processing Overview - The WISE Science Data System (WSDS)


Contents

a. Basic Terminology
b. Data Processing Levels
c. Processing Stages
i. Provisional Processing
ii. First-Pass Processing
iii. Second-Pass Processing

Science data processing for WISE is carried out using the WISE Science Data System (WSDS) developed and operated at the Infrared Processing and Analysis Center (IPAC). The WSDS is a hardware, software, operations and analysis system that converts raw WISE image data packets, engineering telemetry and ancillary information into the photometrically and astrometrically calibrated Image Atlas and Source Catalog, the calibrated Single-exposure Images and Extracted Source Working Database, and all metadata that accompany these products.

The functional block diagram showing the elements and major interfaces of the WSDS is shown in Figure 1. The WSDS is comprised of six subsystems: INGEST, PIPELINES, WMOPS, FPG, QA and ARCHIVE. The high-level function of each subsystem is described below. Detailed descriptions of the design, data flow and processing algorithms are provided in Sections IV.2-7.

Figure 1 - Functional block diagram for the WISE Science Data System (WSDS)

INGEST

The WSDS INGEST subsystem receives science data packets, engineering telemetry and ancillary information from the WISE MOS, and assembles from them the Level 0 FITS-format images and metadata. Following their generation, INGEST then stages the Level 0 images and metadata for pipeline processing. INGEST operations were initiated automatically following the receipt of raw image, housekeeeping and ancillary data that were sent following each WISE downlink pass. The INGEST subsystem is described in IV.2.

PIPELINES

The WSDS PIPELINES subsystem convert Level 0 imaging data and metadata into photometrically and astrometrically calibrated images, extracted source lists and produce corresponding metadata. The PIPELINES subsystem is comprised of two main components: the Scan/Frame Pipeline and the Multiframe Pipeline.

WISE Moving Object Pipeline System (WMOPS)

The WMOPS subsystem is part of the NEOWISE enhancement to the primary WSDS that enables the discovery of previously unknown solar system objects, and provides a database for science users to pre-cover objects that are found after the end of the WISE mission. WMOPS utilizes the output from the Single/Frame and Multiscan pipelines to link non-inertial detections from the single-exposure WISE images made near the same point on the sky to form candidate moving object tracklets that are reported to the Minor Planet Center (MPC). The objective of WMOPS during WISE on-orbit operations was to report candidate tracklets within 10 days of the midpoint of the WISE observations. The MPC computed initial orbits and posted objects on their websites for follow-up and confirmation by ground-based observers. The operational flow and elements of the WMOPS subsystem are described in IV.4.

Final Product Generation (FPG)

The WSDS FPG subsystem constructs the WISE Image Atlases and Source Catalog from the results of the Scan/Frame pipeline. FPG includes the Multiframe pipeline step, along with database and human analysis-intensive operations. The FPG system selects the subset of rows in the source Working Databases that satisfy the WISE Level 1 Requirements for Catalog reliability and completeness . For both the Image Atlas and Source Catalog, FPG assembles final formats and generates any new derived header or catalog parameters that are not already in the Level 3 image and extracted source archives. The FPG system includes characterization, validation, and documentation of the WISE release data products. FPG for the WISE Preliminary Data Release is described in V.1.

Quality Assurance (QA)

The QA subsystem examines the output and collects metadata from the WSDS PIPELINES and generates concise reports summarizing science data quality at each stage of data processing. These reports are in the tabulated in web-based compilations that provide capability to drill-down to detailed image, graphical and tabular data. Elements and operations of QA subsystem are described in IV.7.

ARCHIVE

The ARCHIVE subsystem is a multi-tier system that stores and serves raw and processed mission data and metadata. A component of the ARCHIVE subsystem is integrated into the WSDS processing system and a component is integrated into the Infrared Science Archive (IRSA) at IPAC. The elements of the ARCHIVE subsystem in the WSDS are described in IV.7.


a. Basic Terminology

orbit - An orbit refers to the observations or block of survey data acquired between the times WISE observed the north ecliptic pole (NEP). The WISE orbital period is approximately 95.3 minutes. Orbit number 1 was defined to start with the fictitious NEP passage immediately prior to WISE launch. Thus, WISE was launched into orbit #1. The WISE Preliminary data release include data taken from the nominal start of survey observations in orbit 468 through orbit 2062.

scan - A scan refers to the observations or block of survey imaging data acquired between two flight system maneuvers, such as reorientation of the instrument boresight near the ecliptic poles, or slews to acquire TDRSS for data downlink contacts. Because maneuvers often took place near the ecliptic poles, a scan is associated with the image data from approximately a half-orbit. However, scans can cover less or more than half of the arc between ecliptic poles. Scans are identified in the image headers, Single-exposure source database and metadata tables by the scan_id.

frame - A frame is the image data acquired in a single band from a WISE single-exposure. A single-exposure was acquired approximately every 11 seconds during WISE scans. The frame exposure times are 7.7 sec in W1 and W2 and 8.8 sec in W3 and W4. Because scans have different lengths, the number of frame exposures acquired per scan varies. A histogram showing the distribution of the number of frames per scan for all WISE survey scans is shown in Figure 2.

frameset - The set of four images (frames), one in each band (W1, W2, W3, W4), that are acquired simultaneously during each WISE single-exposure. The same region of sky is imaged in all four bands. The Scan/Frame pipeline processes framesets acquired during one scan.

coadd - Image data in a single band produced by combining multiple single exposure frames that cover the same region of the sky. Coadds are produced in the Multiframe pipeline by resampling, reprojecting and coadding single-exposure frames onto predefined spatial footprints.

coaddset - The set of four corresponding coadds, one in each WISE band (W1, W2, W3, W4), that are generated by combining multiple framesets. Coaddsets are generated by the Multiframe pipeline.

Atlas Tiles - A predefined grid of 18,240 regions on the sky onto which coaddsets are generated by the Multiframe pipeline for WISE data release products. Atlas Tiles cover 1.564°x1.564° (4095x4095 pix @1.375"/pix) and are built on an equatorial projection. The WISE Preliminary Data Release include the Atlas Images and Survey Catalog drawn from 10,464 Atlas Tiles.

Atlas Image - An Atlas Image is the particular case of a coadd image built on the footprint of an Atlas Tile. The Preliminary Data Release intensity images are Atlas Images.


Figure 2 - Histogram showing the distribution of the number of frames acquired per scan during the WISE survey.

b. Data Processing Levels

The terminology used to describe WISE science data during the different steps of downlink and processing is summarized in Table 1. Many of the levels are transient, being generated as input for particular processing operations. Level 0 images, Level 1 images and extracted source lists and the Release Image Atlas and Source Catalog are archived.

Table 1 - WISE Data Levels
Data LevelImage DataExtracted Source DataNotes
Downlink Telemetry CCSDS encoded downlink transfer frames. N/A Generated by spacecraft MUB, donwlinked via TDRSS to White Sands Center. Input for the MOS High-Rate Data Processor.
Source Data Packets Packetized, compressed science data and engineering telemetry, ancillary metadata and navigation files. N/A Produced by the WISE High-Rate Data Processor at White Sands Center and NAV system at JPL. Transferred to IPAC/WSDC following each downlink.
Raw Framesets Decompressed raw pixel data in FITS format. 1024x1024 pix (W1, W2, W3), 512x512 pix (W4) at 16-bit/pix integer. Minimal header information. N/A Transient product. Generated during INGEST processing as step to Level 0.
Level 0 (L0)Decompressed raw pixel data in FITS format. 1024x1024 pix (W1, W2, W3, W4) at 32-bit/pix floating point. Engineering telemetry, metadata and navigation information mapped to headers. N/A Produced by INGEST subsystem. Long-term archive product.
Level 1 (L1) Instrumental pixel-calibrated frame-sets (dark-subtracted, reference pixel corrected, flat-fielded, illumination-corrected, linearized, droop-corrected, etc.). FITS format, 1016x1016 pix at 32-bit/pix floating point (reference pixels trimmed). Headers updated to reflect instrumental pixel calibration, reconstructed WCS values (including distortion corrections) and photometric zeropoint keyword values. Calibrated single-exposure, band-merged source lists. Reconstructed equatorial coordinates, calibrated photometry, artifact flagging. Generated in the Scan/Frame pipeline. Long-term archive in the Single-Exposure Image Archive and Source Working Database. Included in Release Ancillary Products. Level 1 images and source data may also be referred to as Level 1b (L1b).
Level 2 (L2)Level 1 image that has been 2x upsampled and reprojected (rubber-sheeted) for 4-band registration and distortion correction. Reference pixels trimmed, 2032x2032 pix at 32-bit/pix floating point. N/A Transient products - not stored. Generated for source detection and QA.
Level 3 (L3) FITS format images. 4095x4095 pix at 32-bit/pix floating point. Background level-matched, throughput scaled, 4-band registered, upsampled pixels. Updated WCS and photometric zero-point information in headers. Calibrated, bandmerge coadd source lists. Reconstructed equatorial coordinates, calibrated photometry and uncertainties, artifact flagging, confirmation statistics. Generated by the Multiframe pipeline. Generated on pre-defined Atlas Tile footprints in the Multiframe pipeline Archived in the Level 3 Image Archive and Source Working Database. These are inputs to the FPG subsystem to produce the Preliminary Release Image Atlas and Source Catalog. Level 3 images and source data may also be referred to as Level 3a (L3a)
Release ProductsImage Atlas. For the Preliminary Data Release, these are identical to the Level 3 coadded images. Source Catalog. Subset of Level 3 Source Working Database entries augmented with value-added column updates. Produced during Final Product Generation. WISE Preliminary and Final Release formats served via IRSA interfaces.

c. Processing Stages

Science data collected by WISE are processed in multiple stages depending on the phase of the mission. This multi-stage approach is designed to balance the desire to release data as quickly as possible to the community with the need to generate well-vetted products that fully exploit the scientific capabilities of WISE.

Each stage of processing is conducted using a different version of the WSDS. The first two version of the WSDS were used only to support pre-launch testing. The first "flight" version of the WSDS was v3.0.

i. Provisional Processing

Provisional processing of WISE science data was performed during IOC and the first several months of survey operations, to support testing and to monitor survey performance. This processing stage used the version 3.0 (v3.0) of the WSDS along with calibration products based on the best available ground test results. Provisional processing was discontinued on 20 April 2010. All of the early survey data that underwent provisional processing were subsequently included in first-pass processing. Data products from provisional were superseded by the later processing and are not released.

ii. First-Pass Processing

Following WSDS testing and tuning to actual on-orbit performance during IOC and the first several months of on-orbit operations, first-pass processing of the WISE science data was initiated using WSDS v3.5. This processing phase started on 20 April 2010, and used preliminary instrumental, photometric and astrometric calibrations that are derived from a combination of ground test, IOC and early survey data. All of the provisionally processed early survey data were rerun during first-pass processing to generate a uniform archive of images and extracted source lists. The WISE Preliminary Release Image Atlas and Source Catalog, and Single-exposure images and source database are generated from first-pass processing results.

iii. Second-Pass Processing

Following the end of WISE on-orbit data acquisition and the Preliminary Data Release, all WISE science data will undergo a complete reprocessing using an updated version of the WSDS (v4) that incorporates the best available calibration and processing parameter and algorithms derived from analysis of payload, spacecraft over the life of the mission, and from analysis of the first-pass processing results. The WISE Final Release Image Atlas and Source Catalog in Spring 2012 will be derived from the output of the second-pass data processing.

Last update: 2011 April 26



Previous page    Next page
Return to Explanatory Supplement TOC