Wisedecom User's Guide ---------------------- 1 - Introduction 1.1 - Purpose and Capabilities of Wisedecom Wisedecom is a Linux command line application written in Perl and C which, with other associated utilities (see chapter 2), can validate output from the prototype HRDP and help debug problems. It is run from within a terminal window as a shell command line application. Here is the simplest command line to use to execute it: wisedecom -v file1 file2 file3 ... where file1-3 (etc.) are input data files (see section 4). Wisedecom's primary purpose is to read science telemetry data in source packet form as written by the HRDP. It can ingest science telemetry source packets of ApID 254 and packet IDs 26-29 and 42-45. It reads the packet primary and secondary headers, extracts the VTC, grouping flags, packet size and other header data, validates the packets for internal consistency, and assembles compressed or decompressed frame data for any band (depending on the packet ID). Compressed frame data can then be decompressed using a version of the usds program (see sections 2.1 and 2.2) modified to act as a stateless, shared-object library which has been linked to the wisedecom program. Output frames can be written as either standard FITS file images, or raw compressed or decompressed frame files. Both FITS images and raw decompressed files can be viewed and compared with the ds9 utility. Unpacketized raw frame data, either compressed or uncompressed, can also be used as input to wisedecom. This may be useful for debugging various potential data problems encountered during HRDP testing. Also useful for debugging is the ability to suppress decompression of compressed frames to allow direct comparison with data input to the HRDP. Wisedecom compares data it produces against a "truth table," a plain text file which is the output of the md5sum utility run on all raw frame files, compressed and uncompressed, which are to be used as input to the HRDP during testing. This comparison is very fast and will confirm or deny the correctness of data from the HRDP. 1.2 - Relevant Contacts and References Tim Conrow (author) Work: 626-395-8435 Mobile: 626-483-8308 Email: tim@ipac.caltech.edu Jason la Pointe (WISE Project Software System Engineer) Work: 818-393-2569 Mobile: 818-263-9692 Email: lapointe@jpl.nasa.gov FITS Documentation: http://fits.gsfc.nasa.gov/fits_documentation.html ds9 User's Guide: http://hea-www.harvard.edu/RD/ds9/ref/ The section on the GUI will probably be the most useful. Linux Guide: http://www.linuxhq.com/guides/LUG/guide.html The section on using the shell command line interface might be useful for those mostly familiar with Windows or Mac GUIs: http://www.linuxhq.com/guides/LUG/node29.html#SECTION00600000000000000000 2 - Supporting Utilities A few applications in addition to wisedecom will prove useful if it is necessary to track down problems which may arise during testing. 2.1 - Uses and Usds Modified versions of the uses (Rice compression) and usds (Rice decompression) utilities are available. These are lightly modified versions of those distributed under license to the WISE project. They may be useful for double checking that supplied compressed and uncompressed versions of the data were properly compressed and are consistent if it is suspected an error was made constructing the simulation images. A modified version of the usds code was used to produce the decompression library used by wisedecom. 2.2 - md5sum (Linux Utility) Md5sum is a Linux utility that performs a cryptographic hashing operation on input files and writes the hash and the filename of each input file on separate lines. These lines can be captured via redirection of stdout to create a 'truth table' which is used by wisedecom to validate frame data it has produced. Like so: md5sum file1 file2 file3 ... > truth.tbl It can take up to several minutes to run for many large files. If all truth images are placed in a the same directory, the truth table can be assembled with md5sum /path/to/truth/images/* > truth.tbl Here is a sample truth table generated by md5sum: 7b8133ee12e8e71da9a2cae545cacc8a LetterF5.raw 023d11cd634cb69b8b520996f5e78956 LetterF5.raw.E 495505e3d9893ebb1aa00d127f261a6a bars.raw 9e9c4a2f7ff81f2d7a0c0ab7d3406632 barsHugePackets.bin e2022127684e76e8b61b61d335be8986 barsPackets.bin 7b8133ee12e8e71da9a2cae545cacc8a letterF.raw 023d11cd634cb69b8b520996f5e78956 letterF.raw.E 8daaae88262f3fcd9c7d3ec7bb216cdb letterFPackets.bin 687d3dcbf9d529cd0708159866f8b83e ramps.raw 6020ae176dd73eb6e92d03d9962aaf3f ramps.raw.E f6767007881c9cd4fdc85f4167d2a631 rampsPackets.bin Note that both compressed and uncompressed files, as well as simulated telemetry files are included. This is fine; the MD5 hash generated by wisedecom from the input data will correctly find the proper originating file with only a crypographically miniscule chance of a mismatch. An MD5 hash (or, inaccurately, a "checksum") is a good way to validate the contents of a file without having to perform a byte-by-byte comparison. When wisedecom produces the data for a frame, compressed or uncompressed, it can quickly compute its MD5 hash and compare it to the hash read from the truth table. If the hashes agree, the wisedecom-produced data and the truth file associated with the hash from the truth table are identical and wisedecom reports the confirmation and the name of the truth file. If they differ, wisedecom has failed to regenerate the truth data and an error message is produced. 2.3 cmp (Linux Utility) Cmp is a Linux command line utility that compares two files and reports the byte offset of any differences. It can be used to gather additional information on the location of differing pixels as part of a debugging process. cmp normally prints a single line message if two files differ and is silent if they are the same. To get more verbosity, add the '-l' flag which will print out byte offsets for all differences. Since this list could be very large, it is recommended the output be piped through 'more', like so: cmp -l file1 file2 | more 2.4 - ds9 (Image Viewer/Comparator) Ds9 is a GUI-based image viewer which emphasizes viewing and intercomparing astronomical images. It is primarily used to view FITS-file-format images, but can be used to view raw data arrays as well. It has the ability to display image data with various color tables and at various stretches (offset and contrast settings) and display multiple images and blink them (i.e. rapidly and repeatedly switch which image is being viewed). In combination, these abilities should make locating and characterizing differences among images--e.g. a truth image and a bad test image--relatively simple. Start ds9 from a command line with 'ds9 &'. Be aware that ds9 displays lower-numbered pixels at the bottom of the image. If this bothers you, you can select Zoom->Y to flip the Y axis. Opening raw frame arrays is done through the File->Open other->Open array dialog. Once you select the raw file to open (click the 'all' button at the bottom of the file dialog to see all the files), you'll need to give the image size (1024x1024 for bands 1-3, 512x512 for band 4), pixel size (UShort), and endianness (big-endian). When viewing WISE simulated sky images, we recommend setting log scaling and 99/5% stretch from the Scale menu. For ramps and other test patterns, linear scaling and minmax stretch is probably better. For a simpler (but less feature-rich) way to display images, see section 2.7 below on the ImageMagick utilities. 2.5 - etee Etee is a purely internal application used to duplicate output to both stdout and/or stderr and send it to the log file. The user will not interact directly with etee. It's listed here so it's presence on the system when wisedecom is running will be expected. 2.6 - dumpfits Dumpfits simply dumps the meta-data in FITS file headers for all FITS files listed on the command line. FITS headers can also be read with ds9, but it may occasionally prove useful to be able to easily get at header data from the command line. Note that it's possible to do a pretty good job of dumping FITS headers using nothing but standard UNIX utilities. Here's one way: fold -w80 xxx.fits | sed -ne '1,/^END *$/p' 2.7 - ImageMagick ImageMagick is a collection of utilities for displaying images ina variety of formats and converting between various formats. It is too large and complex a collection of capabilities to go in to, but we will mention two uses; displaying and converting raw and FITS frames and FITS files. Documentation for ImageMagick's command line tools resides at http://www.imagemagick.org/script/command-line-tools.php To display a FITS-format frame image, use this command line: display xxx.fits To display a raw frame file, use display -size 1024x1024 -depth 16 -endian MSB gray:xxx.raw for band 1-3 frame. Replace the 1024x1024 with 512x512 for band 4 frames. 'xxx.raw' is the raw frame file. Wisedecom output will use the '.frm' suffix instead of '.raw'. To convert a FITS file to a jpeg, use the convert utility like so: convert xxx.fits xxx.jpeg To convert a raw frame band 1-3, use: convert -size 1024x1024 -depth 16 -endian MSB gray:xxx.raw xxx.jpeg Use '-size 512x512' for a band 4 frame. 3 - Execution Environment Wisedecom requires a properly configured user environment. 3.1 - Default User The provided computer has (at least) one user account pre-configured to be capable of executing all necessary applications. The User ID and password for this user account will be provided by other channels. 3.2 - Data Directories Data, including truth images. FITS files, and simulated telemetry, may be placed anywhere on the filesystem. It will prove convenient for it to reside nearby (e.g. a parent, child, or sibling directory), but any directory will do. 3.3 - User Environment Here is a list of the environment variables and values necessary to run wisedecom and other utilities. The default user account will be pre-configured to provide these environment settings. This environment is set up in the user accounts .cshrc file, so the default user will use the C-shell. 3.3.1 - PATH The PATH environment variable is used to locate executable files must include the standard Linux application directories and have additional directories added thusly: setenv PATH XXX/wise/base/deliv/ops/bin:XXX/wise/base/static/bin:$PATH 3.3.2 - LD_LIBRARY_PATH The LD_LIBRARY_PATH variable is used to locate run-time loadable shared libraries required by an application. It must be set thusly: setenv LD_LIBRARY_PATH XXX/wise/base/deliv/ops/lib:XXX/wise/base/static/lib 3.3.3 - PERL5LIB The PERL5LIB is used by Perl to locate library modules to be loaded at run time. It must be set thusly: setenv PERL5LIB XXX//wise/base/deliv/ops/lib/perl:XXX/wise/base/static/lib/perl 3.4 - Executing Wisedecom Once the environment is set correctly (as should already be the case for the default user), wisedecom can be executed anywhere. Try this far a harmless test: wisedecom -help2 This should print out a verbose description of program and parameter usage and end with a zero status code. 4 - Input and Output This section describes the files and formats required for input, and what wisedecom produces as output in response to various parameters. 4.1 - Input Frame Data File Types Wisedecom accepts 3 basic frame file formats: Uncompressed, raw frame files; Compressed raw frame files; source packet telemetry files. Wisedecom can be given as many files for a given run as desired, but raw frame files must match in their compression state (all compressed or all uncompressed) and band, both of which must be specified on the command line. 4.1.1 - Uncompressed Raw Frame Files (URF) URF files contain a raw list of two-byte (16 bit), big-endian pixel values with no intervening meta-data, line endings, etc. Thus URF files for bands 1-3 will contain exactly 1024*1024*2=2097152 bytes, and band 4 will contain exactly 512*512*2=524288 bytes. Any number of URF files may be specified in any one run of wisedecom, but all must represent the same band, which must be specified on the command line. In addition, the "-nopkt" and "-nodecom" switches must be specified since wisedecom generally expects to be decompressing telemetry data. Mainly, URF files are allowed as input for completeness' sake, but it might be useful also as a way to convert an URF file to a FITS file with appropriate headers. If an URF file is used as input and the '-nofits' flag is set, wisedecom is a no-op, writing out exactly what went in. 4.1.2 - Compressed Raw Frame Files (CRF) CRF files is the exact byte stream as output by the USES chip or the USES software simulator (WISE Patch 1) when run with the command line options specified in the WISE Compression/Decompression Software Simulation DFM Rev. 2 (JPL D-34993). The size of USES-compressed files is unpredictable. CRF file names typically have a ".E" (for "encoded") suffix. Any number of CRF files may be specified in any one run of wisedecom, but all must represent the same band, which must be specified on the command line. The "-nopkt" switch must also be specified. 4.1.3 - Source Packet Telemetry Files (SPT) SPT files are either actual or simulated HRDP output; CCSDS compliant source packets containing instrument image data for ApID 254 output by the MUB (or simulated to come from the MUB). The packet contents can be frame data for any band and be either compressed or uncompressed, as indicated by the packet's Packet ID. Packet ID, grouping flags, VTC values, and other header data are assumed to correspond to take these values: Header item | Values ---------------|------------- Packet ID | 26-29, 42-45 Grouping flags | 0,1,2 Version | 0 Type | 0 Spare | 0 Any other value is regarded as corruption and an error is raised. SPT files are the default input type. Any number of SPT files may be specified for a given wisedecom execution. No indication of band number or compression state is required since the packet IDs encapsulate this information. 4.2 - Truth Table File (TT) The truth table file is a pairing of all truth frame file names used for an experiment and the MD5 hash associated with the raw frame data (compressed or uncompressed) file name. A "truth frame file" is the raw frame data used as input to the C&DH Payload Simulator. Wisedecom will check that the uncompressed image data matches one of the files in the truth table. Note that this comparison is done prior to output as a FITS file so there is NO NEED to run wisedecom to output raw files (with the -nofits flag) in order for the comparison to take place. The format of a truth table is the full MD5 sum as a 32 digit (128 bit) hex number, followed by spaces, followed by the truth frame data file name. This is the output format of the md5sum utility. Thus generating a truth table is as simple as listing all truth file names on the md5sum command line and capturing the output in a file. E.g. md5sum /path/to/truth/files/* > truth.tbl FITS files cannot be used as truth files since the meta data in the FITS header will alter the MD5 hash. Similarly, SPT files also cannot be used as truth frame data since the packet headers and any padding will alter the MD hash. Only raw compressed or uncompressed frame data (i.e. a CRF or URF files) will be matched by wisedecom. Note that it doesn't really matter if FITS files and/or SPT files are included in the truth table; they simply won't be matched by wisedecom, but it is important that all URF and CRF files which might be used in an experiment are included so they can be matched. It is an error to include a truth table as input to wisedecom but to fail to match any output raw frame data. Also note that wisedecom will attempt to match compressed data if compressed data is the ultimate output (i.e. the -nod flag was specified; see section 5.4). 4.3 - Output Frame Data File Types Under the control of command line parameters, wisedecom can write URF and CRF file (as described in section 4.1) or FITS image frame (FIF) files. FIF files are the default and are described below. URF and CRF files can be selected as output by command line parameters; this is mostly useful for learning to use wisedecom and for debugging data problems since it's easy to directly compare URF and CRF input and output files using the 'cmp' utility. CRF file output is only available if the input was compressed, since wisedecom cannot compress frame data. 4.3.1 - FITS Image Frame Files (FIF) The default format of wisedecom output frame data is FITS image files. FITS image files can be viewed or manipulated using many standard astronomical utility programs, most importantly, the ds9 image viewer. FITS files are platform independent and allow meta-data to accompany the image data. FIF file names will have the suffix ".fits". FIF file header info can be read using ds9 (from the File menu) or the dumpfits command line utility. Here is a table describing the meta data available in the header which is specific to wisedecom output. Header Key | Description ------------|------------------------------------------------------------------ BAND | WISE instrument band number; 1 to 4 RUNID | Identifier string optionally given by user on the command line to | uniquely identify a given test run DATIME | Date and time the run was made VTC | Vehicle Time Code for the image if input was a SPT file PKTIME | VTC/100 (seconds) PKID | Packet ID from an SPT file source packet header BER | Bit Error Rate requested by the user; usually zero INFILE | Input file which generated this image FRAMENO | Sequential frame number within a given input SPT file TOTFRAME | Cumulative sequential frame number across all input SPT files INBYTES | Size in bytes of raw, depacketized frame data TRUTH | Truth file name which is the source for the input frame data, as | matched from the truth table (c.f. 4.3) using the MD5 hash of the | resulting raw frame data. If no match was found (but a trutgh | table was specified) value will be 'no_match' RICEOPTS | RICE decompression option overrides as specified by the user | (usually just indicates defaults were used) TRUTHTBL | Name of truth table specified (if any) OBASE | Output file base name pattern specified by the user RUNDIR | Directory in which the run occurred INDIR | Directory from which data files were read, as specified by the | '-indir' parameter EXEC | Full path to the wisedecom executable EXECVSN | Wisedecom version number EXECTIME | Modification time stamp of the wisedecom executable file 4.3.2 - Uncompressed Raw Frame Files (URF) Files of this format are output when the '-nofits' flag is used and the input is uncompressed or the '-nodecomp' flag is NOT set (i.e. decompression IS done). See section 4.1 for the format of URF files. 4.3.3 - Compressed Raw Frame File (CRF) Files of this format are output when the '-nofits' flag is used, the input is compressed, and the '-nodecomp' flag is set (i.e. decompression is NOT done). See section 4.1 for the format of CRF files. 4.4 - Informatory Output (stdout) By default wisedecom runs silently unless an error occurs (but see section 4.6 on log file output). If run with the '-v' command line flag, wisedecom will print messages to stdout showing the execution process including: Program name and version; parameter values; input file names as they are read; raw frame data size, output file names; decompressed data sizes. If the '-hdrs' flag is set, packet header information is also output. 4.5 - Warning and Error Output Messages (stderr) If an error occurs which is serious enough to interfere with continued processing of a given frame or input file, an error message will unconditionally be printed to stderr with a prefix "*** WISEdecom". Such an error will usually mean no further processing on that file or frame will be done. If the maximum error count has been exceeded (see the -nerrs parameter) wisedecom execution will cease with a non-zero exit status. In addition to errors, there are a few other conditions which can occur which are not fatal to further processing but which may be important for the user to know. All warning conditions trapped so far involve interpretation of command line input parameters. All wisedecom error and warning messages are listed and briefly explained in appendix A.1. 4.6 - Log File Output Unless suppressed with the '-log' parameter, all wisedecom runs append both stderr and stdout messages to a file called WISEdecom.log in the execution directory. Stdout and stderr are duplicated with the help of the etee helper utility so that a given message can go both to stdout/stderr and the log file. If the '-v' flag is not given to wisedecom, no stdout is generated and so there's nothing to capture in the log file. If the user wishes to capture stdout messages in the log file but not see them in the terminal window, stdout can be redirected to the NULL device by appending '> /dev/null' to the command line. Stderr (error and warnings) will always appear in the log file (unless log file creation has been suppressed). In addition to stdout and stderr, some information is always written to the log file, including: Start and end run tags with context information and run times; termination status code; parameter values. All runs in a given execution directory will be appended to the same log file. See the description of the '-log' parameter to see how to change this behavior. 5 - Wisedecom Parameters Fine-grained control of wisedecom's operation is possible through the available command line parameters. Command line parameters follow the executable name ("wisedecom") and always come before the list of input file names (see section 5.2.3 below) and serve to refine wisedecom's behavior to suit the user's needs. Wisedecom's parameters have defaults that are meant to make it necessary to specify only a very few parameters on the average command line (usually just "-v" and "-tt truth.tbl"). This section presents parameter syntax and descriptions. Parameters are processed in the order they are read. Parameter files are processed first, then the command line. For repeated parameters, those processed later supercede those processed earlier, except for key=value mappings (see section 5.3.6), where new keys accumulate across repeated specifications, but repeated keys have earlier values replaced by later repeats. 5.1 - Parameter Help Parameter descriptions of increasing detail are available from wisedecom through the -help, -help1, and -help2 parameters. To wit: -help: parameter aliases, types, and default value. -help1: same, plus a brief, one-line description. -help2: same, plus additional descriptive text. 5.2 Parameter Syntax Command line parameters come in two broad types; named and unnamed. Named always come before unnamed parameters. Unnamed parameters are taken to be input file names and follow after the last named parameter and its value (if any). 5.2.1 - Named Parameter Syntax Named parameters are introduced with a '-' (dash) followed with no space by the parameter name or one of its aliases (see the parameter help for all aliases), then the value (if any) preceded by a space. Parameter names are case-insensitive. Parameters can take values of various types, or be switches (aka flags) that require no value. Each type is described below. A typical wisedecom run would look like wisedecom -v -tt truth.tbl file1 file2 file3 ... where the "-v" switch (full name: -verbose") has been set to increase informatory output, and the file "truth.tbl" has been given as the value to the string parameter "-tt" (full name: "truth_table_file"). File1-3 (etc.) are unnamed parameters specifying input file names which will be processed in turn. 5.2.2 - Unnamed Parameter Syntax Unnamed parameters are just white-space separated strings that follow the named parameters. Unnamed parameters are untyped and their interpretation is entirely dependent on how they are used. In wisedecom, unnamed parameters are treated as input file names, as though they were given to the "-input_file" parameter (except the named parameters require comma-separated values instead of space-separated). It is an error to specify input files with both the named "-input_file" parameter and using unnamed parameters. 5.3 - Named Parameter Types Named parameters have types. There are two numerical types--"real" and "integer"--a "string" type that takes arbitrary character strings, and a switch type that is either on or off and takes no argument. These are explained below. Please take note of the weird map type explained in section 5.3.6. 5.3.1 - Switch Parameters Parameters which do not accept a value are called switches. If a switch parameter is not specified, it's value "off" or "unset"; if the switch is specified, it is "on" or "set". E.g. the "-verbose" flag (alias "-v") increases program verbosity (informational textual output to stdout) if set. Note that some switches are defined as negatives. E.g.t he "-nofits" switch means file output will be raw frame files; i.e. FITS file output has been suppressed. Expert note: If a switch is set in a parameter file so that "on" becomes its default state (see section 5.3), it can be reset to "off" with the syntax "-xxx=0". E.g. if, as might actually be useful, the "-verbose" flag is set in a parameter file, it could be turned off for a particular run with "-v=0" or "-verbose=0" since the command line overrides parameter files. 5.3.2 - String Parameters (AKA "str", or "file", or "char"). Parameters of type "string" accept arbitrary text strings. Remember you need to protect characters which the shell regards as special by backwhacking them (i.e. prefixing a back-slash '\') or enclosing the string in quotes. 5.3.3 - Integer Parameters (AKA "int") Parameters of type "integer" accept any integer value that can fit in a signed 4-byte integer vaariable/ I.e. their range is -2,147,483,648 to 2,147,483,647. Integer values specified by the user are type checked and an error will occur if a non-integer value is given. 5.3.4 - Real Parameters (AKA "double", or "dbl") Parameters of type "real" can accept any numerical value within the range of a double precision (8-byte) variable. A real parameter has a precision of roughly 15 decimal digits. Real parameters are type checked to look like numbers; non-numerical values will generate an error. 5.3.5 List Parameters Any of the types (except switches) described above can be specified with the "list of" modifier that indicates it can accept multiple values. The multiple values are specified as a comma-separated list. If a string type is a list, the user must be careful with white space and other characters the shell treats specially. For string types, the strings may not include a comma. As an example, the "-input_file" parameter can accept a list of comma-separated file names, e.g. "-if xxx,yyy,zzz". 5.3.6 - Maps (AKA map, mapping, list of map(s)) The value is a comma separated list of key=value mappings. I.e. each comma-separated component of the list contains a key and a value separated by an equals sign. The values can be of any type except switch (i.e. string or numeric); there is no type checking. Each key represents a "sub-parameter" which wisedecom can refer to separately. Key=value mappings are used to collect closely related parameters under one main parameter for purposes of compactness. Usually mappings are used to get rarely used or very closely related parameters out of the way to improve clarity. Since map parameters are by their nature lists, adding the "list of" modifier has no effect. Mapping parameter definitions allow for a list of allowed keys (visible in the help output after the parameter type). All the mapping parameters in wisedecom use these lists. This means an illegal key--one not in the legal key list--will generate an error immediately. In wisedecom, the parameters "rice_params" and "-other_opts" are key=value mappings. The user should not normally find it necessary to use them. As an example of use, if the user wanted reset the VTC epoch from zero to, say 1000000 and also indicate that wisedecom should introduce bit errors in the telemetry, the specification would look like '-opts vtc_epoch=1000000,ber=1'. 5.3.7 - Named (AKA names, name, list of name(ds)) The value is a comma separated list of names, usually meaning a subset of a specific set of valid identifiers specified in a valid name list with the parameter definition (visible in the help output after the parameter type). All the name list parameters in wisedecom use these validation lists. This means an illegal name--one not in the legal name list--will generate an error immediately. Since named parameters are by their nature lists, adding the "list of" modifier has no effect. In wisedecom, the parameter "-rice_flags" is a name list. The user should not normally find it necessary to use them. As an example of use, if the user wanted to disable CHIP emulation and switch to nearest neighbor decompression (something one need ever do in real life), the specification would look like '-rice_flags -CHIP,NN'. 5.4 - Parameter Files All parameters that can be specified in a parameter file in the execution directory called "All.params". This file could contain experiment-specific settings that correspond to the current directory. The format of each line of the parameter file is -wisedecom: ...parameters... where "...parameters..." is to be replaced with one or more parameters exactly as they would be specified on the command line, although you don't have to worry about protecting special characters from the shell except for spaces. As many lines as necessary can be used provided each is prefixed with "-wisedecom:"; they are processed in order. Parameters specified on the command line override those given in a parameter file. It can be dangerous to use the "All.params" file since it's easy to forget it's there, leading to unexpected and confusing behavior ("Hey! I didn't say to do that!"). Employ caution in abundance. A safer alternative to the "All.params" file is to use explicitly-named parameter files, which is done with the syntax "-@filename", where "filename" is a parameter file employing the same syntax as described for the "All.params" file. E.g. wisedecom -@std.params -runid run2 experiment4/*.bin This will read parameters from the file "std.params" which is looked for in the execution directory. A relative or absolute path can come before the file name to specify a file residing anywhere. E.g. wisedecom -@param_files/std.params -runid run2 experiment4/*.bin This looks for the parameter file in the 'param_files subdirectory of the execution directory. In both examples above, after the parameters are read from the file, the "-runid" parameter is added with a value of "run2". If a run ID is defined in the file, this will override that definition. Next, the list of input files is provided; in this case, all files ending in ".bin" from the "experiment4" subdirectory will be processed. 5.5 - Parameter Descriptions The parameter description below are from the output of '"wisedecom -help2", reformatted, lightly edited and expanded on in places. Parameter: input_file Aliases: infile,if,f,i Type: list of file Default: None Name of one or more comma-separated file names containing raw frame data or packetized telemetry. May also come from trailing, space-separated unnamed arguments, but not both. If a file name is exactly '-', the data is expected from stdin. On;y one raw frame file but any number of telemetry files may be sent via stdin. Raw compressed and uncompressed files must be accompanied by the -band parameter. Uncompressed raw frame files must be accomanied by the -nodecom switch. Parameter: run_id Aliases: runid,run Type: string Default: None String to replace %RUNID% with in o/p file names. Also replaces the %INBASE% tag if input comes from stdin. Will be written to the RUNID FITS header value in FITS output files. Parameter: dump_pkt_hdrs Aliases: dumphdrs,hdrs Type: integer Default: 0 Dump packet header information from source packet telemetry files. 0=no header output, 1=first packet in each frame, 2=all packets. Useful for debugging data problems. Parameter: band Aliases: bnd,b Type: integer Default: None or from packet headers Band (1-4) of frame data. Must be specified for raw compressed or uncompressed frame data. Not necessary for telemtery files, but if given will be used of the packet IDs. Parameter: no_decompression Aliases: nodecom,nod Type: switch Assume input data is not compressed. Must be specified for uncompressed raw frame data. Not necessary for telemetry files, but if specified will be used regardless of the packet IDs. Parameter: no_packet Aliases: nopkt,nop Type: switch Input data is not packetized. A raw frame pixel array is assumed. Must be specified for raw frame files. Parameter: no_fits Aliases: nofits,nof Type: switch Write a raw frame array, not a FITS file. This may be useful for debugging data problems, but make viewing the output more difficult. Parameter: truth_table_file Aliases: ttfile,tt Type: file File with lines in the format "truth-md5 truth-path". "truth-path" is the path name of a raw frame file and "truth-md5" is the md5 checksum of that file. This is the output of the md5sum command. If this file is given (and exists) the decompressed images will be md5 checksumed and the checksum used to identify the truth image. If checksum doesn't match any in the table, an error is reported. All raw frame data, compressed and uncompressed, should be run through md5sum and its output captured in a file which will then become the trith file for all subsequent experiments. Parameter: out_base Aliases: obase,ob,o Type: string Default: %INBASE%-%BAND%-%PKID%-%PKTIME%-%SEQ% This string provides the base name of output files. It can be a simple base name (no %-tags) in which case that name is used exactly (meaning it will be overwritten if multiple frames are being written). Any %-tags given are replaced on the fly. A path prefix may be given to indicate an output directory other than the execution directory. In normal use, the execution directory is the expected output directory. A .fits suffix is added for FITS files, and .frm for raw frame array files (see the -nofits switch). Compressed raw frames have .frm.E appended (see the -nodecom switch). Obviously, compressed frames cannot be written to a FITS file. Since the ".fits" and ".frm" suffixes are applied automatically by wisedecom, don't put it in the base name you supply. Any "." suffixes supplied in the base name will have the ".frm" and "fits" appended. Tag meanings: Tag | Replaced By -----------|----------------------------------------------------------------- %INBASE% | Name of input file exclusive of the directory name | and trailing suffix after a "." %BAND% | Band name (w1-w4). %PKTIME% | VTC seconds in the pkt header. Removed for | non-packet data. %PKID% | The packet ID in hex. Removed for non-packet data %DATIME% | The current date and time at program start. %RUNID% | Manually set (-runid) run ID string. %TOTFRAME% | Sequential frame number for all input telem. streams | or raw frame data. Cumulative across all files. %FRAME% | Sequential frame number for given telem. stream | or raw frame data. Starts at one for each file. %SEQ% | A unique sequence number. The current directory | is searched for files matching the base name | and, if found, the highest sequence number found | is incremented and used for the current file. Parameter: rice_params Alias: params Type: map Override Rice decompression parameters to match input data. This option should rarely if ever be used. Available keys: Key | Alias key | uses/usds parameter | Default -----------------|-----------|---------------------|-------- bits_per_pixel | bpp | -n | 15 pixels_per_block | ppb | -j | 16 blocks_per_ref | bpr | -br | 32 refs_per_packet | rpp | -rp | 1 Parameter: rice_flags Alias: flags Type: names Default: DEFAULT Identifiers representing overrides to the default Rice decompression flags to match input data. Should rarely if ever be used. A leading - means disable. These identifiers correspond to various command line switches to the usds command. Available Flags (* = set by default): Identifier | uses/usds flag | Comment ------------|-------------------|----------------------------- RESET | n/a | Reset all flags to false DEFAULT | n/a | Set all flags to default values NN | -nn | EC | -ec | EP | -ep | TWOD | -2d | TWOL* | -2L | RR | -rr | BYPASS | -by | MS | -ms | NOSIGN* | -pos | NEG | -neg | CHIP* | -chip | See the usds help output for more info. Parameter: verbose Alias: v Type: switch Add printed output to stdout to track program operation. Parameter: log_file Alias: log Type: string Default: +1 Log file output if not '0'. Can be an explicit file name for the log output, but normally "WISEdecom.log" will be used and magic numbers are used to direct what is to be done. If a different log file is desired for each run (not a bad idea), name it explicitly; e.g. one might use the run ID: wisedecom ...other params... -runid test23 -log test23.log ...etc... Value | Means ------|------------------------------------------------------------------------ name | append to the given file name +1 | enable log output, append run output to the standard log file .1 | enable log output, overwrite the standard log file 0 | disable log output Parameter: max_errors Aliases: maxerrs,nerrs Type: integer Default: -1 Maximum errors to allow before program terminates. Normally, wisedecom continues processing each file as far as possible, and will continue processing new files even if errors have occured. To change this behavior, set this value to a n on-negative integer. '0' would mean terminate after the first error. '10' would mean terminate after the 11th error. '-1' (the default) means never stop. Parameter: input_dir Aliases: indir,idir,id Type: string Default: . # Directory containing input files. "." means the execution directory. This directory is prefixed to the given input file name(s) unless the name begins with a '/' (i.e. a path is already prefixed), in which case the given directory is used. Defaults to the execution directory. Parameter: other_opts Alias: opts Type: list of double, key=value mappings Default: (see below) Catch-all parameter for grouping rarely-used parameters. Descriptions are brief since it is not expected they will normally be used. Key | Default | Meaning ----------|---------|---------------------------------------------------------- noout | 0 | write no output files if not '0' nx,ny | 0,0 | expected size of output frames; '0' means normal size bpp | 0 | bits per pixel in frame data; '0' means normal (16) seqno | 0 | fixed sequence number to use in o/p file names vtc_epoch | 0 | seconds to add to VTC to get a time ber | 0 | number of bit errors to introduce to a compressed frame bit | -1 | bit error location; -1 means choose randomly 6 - Wisedecom Execution Flow In this section we will briefly describe the execution flow within wisedecom so informatory output has more context. This is not a detailed, line-by-line description of wisedecom's code. It may be worthwhile keeping in mind that wisedecom is a hybrid program; it uses a Perl main program which internally utilizes C-code libraries for decompression, packet decoding, and FITS file writing. Thus parts of the execution path (and possible messages generated from them) will be from Perl code, others will arise from within binary C code being executed. All parameter handling, main textual output, file I/O (except FITS file writing) are done at the Perl level. Heavy bit manipulation is done in faster C code. In the process flow description below, it is assumed parameter values are set to their default values unless otherwise specified. 6.1 - Module/Library Loading At program start-up, Perl extension modules and C-based libraries are loaded. If there is an environment, permission, or configuration error, wisedecom may fail here. There may be a delay if the inlined C-code requires compilation before execution can proceed. 6.2 - Parameter Parsing Next, wisedecom reads and parses its parameters, both from the command line and from any parameter files. Parameters are also checked for consistency. Any errors in parameter parsing or consistency checking will lead to immediate termination of the execution of wisedecom with a non-zero exit status. After parameter parsing is complete, if the "-verbose" flag is set, a "banner" is printed that echoes the defined parameters as well as other execution information, such as the time, the process ID, the program version number, etc. 6.3 - Initialization In the initialization stage, variables that depend on parameter settings are initialized, the truth table (if defined) is read in and parsed and checked for obvious syntax problems (there shouldn't be any if the file is straight output from the md5sum command). 6.4 - Main File Loop After initialization, the input files are examined one at a time in a loop. Each input file is processed to completion before the next file is processed. The input file name is modified by the value of the -indir" parameter and then the file read in. In the verbose output, as each file is read in, the line ">>>>>>> Processing file ..." is printed followed by the file name and file size. The input file base name is established at this point. The base name is the name of the file after the path (after the last "/" if any) and before any suffix following a ".". This is the string that will replace the %INBASE% tag in the output file name. The expected band and compression state of the input data is established at the top of the file loop by the presence or absence of the "-band" and "-nodecom" flags. For source packet data (simulated telemetry) the band and compression state are established by the packet IDs, so unless the "-nopkt" flag is present, indicating the input is from raw frame files, the "-band" and "-nodecom" flags should NOT be set. If they are set, they will override the packet ID. 6.5 - Per-frame Inner Loop For source pocket data, each file might generate many output files. There is an inner loop within the main file loop which iterates until the input file is out of packet data. For raw frame files (i.e. the "-nopkt" flag is set) the inner loop is degenerate and simply executes once per input file. 6.6 - Packet Header Decoding and Image Assembly Packets used to generate each output file are read one at a time. No read-ahead is required since the packet headers contain the necessary information to determine exactly how many packets should be read. As frame data is assembled the packet header information is checked for consistency and completeness. Any errors in reconstructing the frame data will be printed at this point and further processing for this frame terminated, but the input file will continue to be read in case later frames can be processed properly. It's possible a synchronization error of some sort (such as corrupted packet size or science data size information) will make all subsequent frames uninterpretable too, but this will not be known until the attempt is made (i.e. wisedecom isn't smart enough to recognize some kinds of errors which should terminate file processing). Frame data size, VTC, band, and compression state from the packet headers are saved for later use (such as constructing the output file name) and are printed if the "-hdr 1" parameter is given. If "-hdr 2" is given, header data for EACH packet is printed out as it is processed. 6.7 - Rice Library Invocation and Decompression Unless the "-nodecom" flag was set or, for telemetry data, the packet ID indicates the frame data are not compressed, the Rice decompression library is called on the assembled frame data and it is decompressed. If the decompression fails, an error is printed and the next frame is processed. If the decompression completes without error, the resulting pixel array size is compared to that expected for the band associated with the data. For data wih bit errors, some decompression failure modes will result in decompression completing, but returning a pixel array of the wrong size. If this occurs, an error is printed and the next frame is processed. Lastly, if the "-tt" parameter has been defined, the MD5 hash of the pixel array is compared to the hashes found in the truth table. If a matching hash is not found, an error is printed and the next frame is processed. In verbose mode, compressed and decompressed pixel array sizes are printed. At this point, the output file name is established. Depending on the string passed to the "-obase" parameter, various runtime information, such as times, packet header values, input file base names, etc. may be substituted into the output base name, which may also have an output directory prefixed by the user. The ".fits" suffix is applied for FITS file output, and the ".frm" suffix is applied if the "-nofits" flag was specified. 6.8 - File Output For FITS file output, run time data similar to what can be included in the output base file name but more extensive, is written to the FITS header. Pixels are byte-swapped (since the FITS library expects x86/64-type little-endian integers) and written out by the cfitsio library. For raw file output, the pixel array data is written as is to a file I/O errors for opening or writing files, if any, will printed here and processing will continue to the next frame. 6.10 - Termination and Error Handling Error referred to above are counted and when they exceed the limit set by the "-nerrs" parameter, wisedecom execution will terminate with a non-zero status. If the number of errors is unlimited (the default), processing will continue until all input files have been processed, but if there were errors, the termination status will also be non-zero. Successful runs (no errors encountered) always have an exit status of zero. 7 - Annotated Sample Wisedecom and Utility Command Lines and Output A few sample runs of wise decom have been made and saved for the user to play with. These runs can be duplicated and the output compared to the pre-generated sample directories. Each sample below consists of a command line which will be explained along with the output expected. A detailed examination of the output can be had by examining the sample directory contents. Each sample directory contains a file called "CmdLine.xxx.txt" which contains the command line executed, and "Out.xxx.txt" which contains the output sent to stdout and stderr. In both cases "xxx" will be replaced by the sample run name (e.g. "sample1_1"). All input files and output files, as well as the log file will also be present. 7.1 - Directory Setup The user may find it useful to setup three directories for using wisedecom: the telemetry input directory, which will hold the output from the HRDP; the truth file directory, which contains all truth frame data, compressed and uncompressed, which will be matched by wisedecom to output data it generates; the execution directory, which is where wisedecom will be run from and where all output files will be generated. The first two may well never need to change throughout HRDP testing, but a different execution directory may be desired for each experiment. The truth table can be generated in either the execution directory or the truth file directory as long as the complete path to its location is given to the -tt parameter. For the samples below, the truth files will be in directory "XXX/wise/samples/truth". HRDP "output" (more like anticipated HRDP output) will also reside there. The execution directories will be "sample1", "sample2", etc., all under directory "XXX/wise/samples". When running the samples yourself, you should create a subdirectory under the sampleX directory and copy the "All.params" and "truth.tbl" files into that new directory. Sample directories will have symlinks to the truth data, from which input data will be read. When making run directories for yourself, you'll need to override the "-indir" parameter to read from the parent directory's symlink, like this; "-indir ../truth". The results of your run should correspond well to the sample runs. Output FITS header meta-data should be the same except in details such as dates, times, and file directory names. Frame pixel data should be identical. 7.2 - Parameter Setup To make the samples easier to explain, the output base name will be changed from the complicated default value to "%RUNID%-%BAND%-%TOTFRAME%. This will require us to always specify the "-runid" parameter on the command line. This parameter definition will be placed in the file "All.params" in each sample directory (see section 5.4 on parameter files, especially with regard to the caution there on using "All.params" files). To make command lines in the samples simpler, we will also pre-define the location of the truth table, the input directory, and the "-verbose" flag. Here is the content of the "All.params" file: -wisedecom: -v -ob %RUNID%-%BAND%-%TOTFRAME% -indir ../truth -tt truth.tbl 7.3 - Sample Command Lines 7.3.1 - Decompressing CRF Files (sample1_1, sample1_2) The truth data include the compressed raw frame files "LetterF5.raw.E" and "ramps.raw.E". To decompress these files, and place the results in FITS files, execute wisedecom -runid sample1_1 -band 1 -nopkt LetterF5.raw.E ramps.raw.E (all on one line). The output files will be "sample1_1-w1-1.fits" and "sample1_1-w1-2.fits". These files can be viewed with ds9. Start ds9: ds9 & Use the File->open dialog to select "sample1_1-w1-1.fits" for viewing. Go to the Scale menu and select log and 99.5%. Use the zoom menu to zoom in and out. Use the mouse to pull the pan box around. Click "center" and return the zoom level to 1. Then, under the frame menu, select new, then use the file menu again to view "sample1_1-w1-2.fits". Select the same scale parameters as before. Use frame->next or frame->prev to move back and forth between these frames. Select File->Display Fits Header to look at the file's header meta-data. Use Frame->delete to stop viewing the second file and view only the first. Now use the File->open dialog to navigate to the truth file directory "XXX/wise/samples/truth" and open "LetterF5.raw". This should be identical to "sample1_1-w1-1.fits" except for header data. Scale the new image as before. Now select Frame->blink to blink these two images. You should not see any change in pixel values between the images. A small variation of this same command line is to add the "-nofits" option to write out raw frame data that can be matched directly to the truth file: wisedecom -runid sample1_2 -band 1 -nopkt -nofits LetterF5.raw.E ramps.raw.E (Note the new run ID used.) Output files will be as before except they will end in ".frm" and start with "sample1_2". Confirm the files are identical by typing cmp -l sample1_2-w1-1.frm ../truth/LetterF5.raw | more There should be no output. We already know the files are identical because we specified a truth table in the run and wisedecom ended without any errors. 7.3.2 - Depacketizing and Decompressing SPT Files (sample2_1, sample2_2) The anticipated normal use of wisedecom will be to run it on the output from a given test of the HRDP. In the truth directory "XXX/wise/samples/truth", the file barsPackets.bin is a simulated source packet telemetry file containing band 1 frames. The simplest way to run this file through wisedecom is: wisedecom -runid sample2_1 barsPackets.bin This produces output files called "sample2_1-w3-XXXX.fits" where XXXX varies from 0001 to 0030. You may examine these files in the same manner you did the files from sample1 using ds9 and cmp. (If ds9 is already running from a previous sample, there's no need to restart it. Just navigate to the new sample directory.) A variation on this is to add the "-nofits" flag to get raw frame output. wisedecom -runid sample2_2 -nofits barsPackets.bin (Note the new run ID used.) 7.3.3 - Decompression errors (sample3_1, sample3_2) To see what happens when there is a decompression error, we will introduce a bit error into a raw frame file in two different places to see what happens. Here are the unusual command lines that artificially generate examples of decompression errors: Rice error: wisedecom -runid sample3_1 -band 1 -nopkt -opts ber=1,bit=1520802 ramps.raw.E This command line will result in an error from the Rice decompression library that begins "*** USDS ERROR: usds lost ...". No decompressed bytes were generated and the run ends at this point with no file output. No Rice error: wisedecom -runid sample3_2 -band 1 -nopkt -opts ber=1,bit=1538332 ramps.raw.E This error is more interesting. A significant fraction of the time, in the presence of a bit error, the decompression process succeeds, but generates erroneous data. In this case, no error from the Rice library occurs and the run continues up to and including writing an otuput file (note that the output file name is modified with the text "-BER1" to indicate a bit error was artificially introduced). The error is not caught until the frame data is compared to the truth table through their MD5 hashes, at which point an error is generated. Follow the procedure in section 7.3.1 to blink the wisedecom moutput against the original in the truth directory. Can you find the location of the bad data? Where does the "cmp" utility place the first mismatching data? Appendix A - Error and Warning Messages Error messages written by wisedecom are written to both the log file (unless suppressed) and the stderr output stream and are generally prefixed by '*** WISEdecom' to increase their visibility (though messages referred to in the mappings paragraph in A.1 are Perl errors and will not be so prefixed). Here is a list of all error messages and there meanings. For the messages below, the error prefix has been excluded. Some messages will have the end truncated and replaced with a '...' in the descriptions below. Usually the missing part is either context information (file names, frame numbers, etc.) or a system call error message indicating why some I/O operation failed ("no such file" or a permission error, etc.). Actual error text is in quotes. A.1 - Parameter Specification Errors Most parameter errors are either parameter name typos, format errors (most often from failing to protect spaces in strings and other special characters from interpretation by the shell with quotes), or type errors (e.g. not giving a proper number to a numeric parameter) . Numerous errors prefixed with "*** Wisedecom/PARAMS" These are errors generated by the low-level parameter parser. Such errors are followed by a call stack trace and are thus multi-line and very ugly. They generally indicate the user has either mistyped a parameter alias or value, or left off a value where one was needed, or added a value to a switch parameter type. Here are the errors which will be most commonly encountered: "Unrecognized parameter: 'xxx'." "Integer parameter 'xxx' has illegal value 'yyy'." "Real parameter 'xxx' has illegal value 'yyy'." "Can't input from 'xxx' ..." The last occurs when a parameter file is specified using the "-@filename" syntax but the file could not be read. . Errors Related to Mappings and Names (see section 5.3.6,7) If you forget to use key=value format--e.g. if you give a simple numerical value to '-opts' ("-opts 12")--you will see this opaque message: "Map parameter rice_params value doesn't look like a mapping" If you use a key for either names or maps that isn't in the approved list, you'll see this error: "Value for parameter xxx has illegal key(s) 'xxx'." Also, since mapping values aren't type checked, you could get away with accidentally put an arbitrary string where a number is expected, which may eventually lead to a message like "Argument "xxx" isn't numeric in addition (+) ..." and probable misbehavior of wisedecom. . "Unable to start logging ..." For some reason the log file could not be opened. A preceding error message from a Perl module should indicate the specific problem. . "Illegal character(s) in run ID ..." The run ID may be used for part of output file names, so it must not contain characters that could be confusing to use in a file name on Linux. The excluded characters are: /, ;, \, |, `, ', and ". . "%RUNID% found in output file model, but '-runid' not specified" If the output base name pattern includes the %RUNID% tag, the user must supply a run ID (duh). . "Illegal band specified" Only the integer 1 to 4 are allowed. . "Unpacketized raw frame input data specified, but no band given" Non-SPT files (i.e. URF or CRF files) must have a band specified with the '-band' parameter. . "Unnamed parameters found with leading '-' ..." After all named parameters were processed, an unnamed parameter was found prefixed with a '-'. Named parameter processing is terminated when a string not prefixed with '-' and not taken as an argument to a previous named parameter is seen. After this point, command line arguments are assumed to be file names to be used as input. If one or more begins with a '-', it is an error. This is to trap a common parameter specification error where a parameter argument is misplaced or forgotten, or there is a typo. E.g. wisedecom -v -nod 1 -nof input1 input2 The '-nod' parameter doesn't take an argument (it's a 'switch' type parameter), so parameter processing stopped at the '1', which is assumed to be the name of the first input file. When the '-nof' flag is seen, it is an error. . "Both '-infile' and unnamed input files found ..." Input files can be specified as either a comma-separated list as an argument to the '-infiles' parameter, or as a list of space-separated file names after all named parameters. To avoid common parameter specification error and typos, only one of these is allowed. . "RunID required for STDIN input" If input data is coming from stdin, a run ID must be provided to provide a base name for output files. . "Rice: Options key=>val not paired" A key=value mapping to the "-rice_params" parameter is malformed and not in proper "key1=value1,key2=value2,..." format. (Why are you trying to use this parameter anyway?) . "Rice: Option key unknown" A key specified in a the key=value mapping given to the "-rice_params" parameter is unknown to the usds decompression code. (Why are you trying to use this parameter anyway?) . "Rice: Option flag unknown" An identifier string to the "-rice_options" parameter is unknown to the usds decompressor. (Why are you trying to use this parameter anyway?) A.2 - "File I/O Errors" I/O errors almost always arise from one of two causes: On input, the file you named doesn't exist. Probably the directory is wrong (did you forget to put the absolute or relative path to the data?), or there was a typo in the filename. On output, the problem is usually that you don't have permission to write where you want to write. Permissions will generally be pretty wide open in normal user space, so probably you accidentally tried to write somewhere you shouldn't. A common boo-boo is to prefix a filename or relative path with a "/", inadvertently making it an absolute path. E.g. this wisedecom -obase /%RUNID%-%BAND%-%TOTFRAME% will try to write output files in the root directory, which is a definite no no. Removing the leading slash will write to the execution directory, which is what is usually intended. . "Truth checksum file not found or empty" Self-explanatory. . "Bad line in truth file " A line in the truth table did not match the format 'md5hashfilename'. . "No lines read from checksum truth file " The specified truth table has no data lines. . "Unable to open ..." Self-explanatory. The system error will be appended. . "Read nothing from file " An input file had no content. . "Can't write to directory 'xxx' ..." Output was attempted to a directory that either doesn't exist or for which you don't have write permission. . "Failed to open output FITS file ..." Self-explanatory. The system error will be appended. . "Unable to write pixels to FITS file ..." The frame pixels could not be written to the FITS file. There should have been a preceding error indicating the problem. Either of the above two FITSIO errors will be accompanied by errors from the FITSIO Perl module prefixed with "*** FITSIO/open" as well as from the underlying library that begin with "FITSIO status = ...", also there'll be stack trace. All-in-all it's a pretty voluminous and ugly error. You can't miss it. . "Frame is incorrect size for FITS output ..." One can only write full, decompressed frames to FITS files. Wisedecom knows how big these frames should be and has detected that you are trying to write data of an inappropriate size. Usually this happens when you specify the '-nodecom' flag on compressed data, such as when the output is a CRF file or source packet data containing compressed images (packet IDs 26-29). This message implies FITS output is skipped for this frame's data. . "Failed to open output raw file ..." Self-explanatory. The system error will be appended. . "Unable to write pixels to raw file ..." Self-explanatory. The system error will be appended. . "Unable to write (close) pixels to raw file ..." Self-explanatory. The system error will be appended. A.3 - Packet Parsing Errors . "Failure depacketizing frame ..." This is the top-level catch-all error given when one of the error below is tripped during packet parsing. . "Short pkt ..." A packet was smaller than the the standard source packet size and wasn't the final packet for a frame. . "Bad pkt hdr ..." One or more of the fixed packet header bit sequences contained an unexpected value. The fixed bit sequneces are for the version, type, secondary header, spare, and apid values. See 4.1.3 for what these values should be. . "Science data exceeds packet size ..." The science data size expected in this packet doesn't fit in the indicated packet length. Possibly either the packet length or science data length bits have been corrupted or mis-specified. . "Partial packet ..." The science data size and the packet length match, but the packet isn't full. I.e. the packet length isn't that of a standard source packet. . "Data doesn't fill non-terminal pkt ..." The science data doesn't fill the source packet, but this is not a terminal packet (i.e. the grouping flag is not 2). A.4 - Decompression Errors These errors occur when compressed data could not be correctly decompressed. . "Unable to decompress frame $nfr of file ..." This is the catch-all, top-level error given when the usds code returns without completing the decompression. This message will be preceded by an error message directly from the usds code prefixed with '*** USDS ERROR'. This will be the most common failure mode for corrupted data. . "Frame size of bytes is unexpected ..." Sometimes the usds code is able to decompress, correctly as far as it's concerned, but ends up returning to few or too many bytes for a correect WISE instrument data frame for the given band. When that occurs, wisedecom traps emits this errors. This will be the second most common decompression failure mode. . "Unable to match checksum to truth table ..." Occasionally, the usds is able to decompress successfully (as far as it knows) and provide data of the correct size, but which is not the same as the inout data causing a failure to match an MD5 hash in the truth table. This is the error emitted in that case. This is the least common decompression failure mode. A.5 Internal Errors The following errors indicate a bug in wisedecom. It will ned to be fixed. . "Rice: opt_flag value not a ref" . "Rice: opt_flag value not an array ref" . "unpack_bits: <= zero bits requested" . "unpack_bits: atbit or atbit+nbits-1 out of range" . "unpack_bits: Result too large" . "unpack_bits: result size != 1,2,4 or 8" A.6 - Warnings . "Whitespace in run ID replaced with '_'" A run ID was specified with spaces or tabs. These have been replaced with an underscore. . "-runid specified, but no %RUNID% found in output file model" A run ID was specified, but no %RUNID% tag was specified in the output base name. This is not a problem, but might indicate the user didn't specify the output base name as intended. Appendix B - Acronyms C&DH - Command and Data Handling CCSDS - Consultative Committee for Space Data Systems FITS - Flexible Image Transport System FTP - File Transfer Protocol GB - Gigabyte GDS - Ground Data System GMT - Greenwich Mean Time GSE - Ground Support Equipment HDU - Header Data Unit HRDP - High Rate Data Processor IT - Information Technology JPL - Jet Propulsion Laboratory LSB - Least Significant Byte MOS - Mission Operations System MSB - Most Significant Byte MUB - Mission Unique Board N/A - Not Applicable NASA - National Aeronautics and Space Administration SDIP - Science Data Ingest Processor SFTP - Secure File Transport System SRD - Software Requirements Document SwRI - Southweset Research Institute TBD - To Be Determined TBR - To Be Resolved TBW - To Be Written USDS - Universal Source Decoder for Space USES - Universal Source Encoder for Space VTC - Vehicle Time Code WISE - Wide-field Infrared Survey Explorer WSDC - WISE Science Data Center Appendix C - Sample FITS Header SIMPLE = T / file does conform to FITS standard BITPIX = 16 / number of bits per data pixel NAXIS = 2 / number of data axes NAXIS1 = 1024 / length of data axis 1 NAXIS2 = 1024 / length of data axis 2 EXTEND = T / FITS dataset may contain extensions COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H LONGSTRN= 'OGIP 1.0' GRELEASE= 'NONE ' BAND = 1 RUNID = 'NONE ' DATIME = '070113T223027Z' VTC = 1001100 PKTIME = 10011 PKID = 42 BER = 0 INFILE = './b1u.bin' FRAMENO = 2 TOTFRAME= 5 INBYTES = 2097152 TRUTH = '/users/tim/wise/tim/data/sim/01042007/09295000000-1.raw.Q' RICEOPTS= 'opt_flags=>[DEFAULT]' TRUTHTBL= 'truth.txt' OBASE = '%INBASE%-%BAND%-%PKID%-%PKTIME%-%SEQ%' RUNDIR = '/users/tim/wise/tim/data/sim/01082007' INDIR = '. ' EXEC = '/users/tim/wise/base/deliv/ops/bin/wisedecom' EXECVSN = 'v1.0 ' EXECTIME= 1168725401