Procedure for on-orbit flats.

Last modified: 01/05/2010 by F. Masci


NOTES: * for your convenience, steps (1) - (5) below have been performed
         under caustic:/wise-ops/01/wise/fmasci/NedSim_May09/compflat_forStef/
         It is advised you repeat these in your private area on caustic.

       * the input test data used below are random scans selected from the
         most recent 30-orbit simulation (v3.2 testbed). The pre-processed
         frames were available at the time of writing. Please ensure they
         still exist before proceeding.
	 
       * the number of frames used in the test runs below is not necessarily a
         realistic (optimum) number for each band. These runs are only to
	 demonstrate software functionality. Suggested numbers for on-orbit
	 flats optimized using the 30-orbit simulation are in the test report
	 referenced in (8) below.


(1) Ensure on-orbit frames for a desired number of scans have been processed
    through the "dynacal" pipeline, more specifically, we're interested in
    intermediate "linearized" and mask frames from the initial run of
    "wsfical" in dynacal with the -svlin switch turned on.

    E.G., suppose we want to create flats from two scans worth of frames,
    we first make a local directory called, e.g., "testdata" and copy over 
    the frames:
    mkdir testdata;
    cp /wise/tops/scans/7a/01107a/fr/*/cal/01107a*-lincor-1a.fits testdata;
    cp /wise/tops/scans/7a/01107a/fr/*/cal/01107a*-msk-1a.fits.gz testdata;
    cp /wise/tops/scans/8a/01108a/fr/*/cal/01108a*-lincor-1a.fits testdata;
    cp /wise/tops/scans/8a/01108a/fr/*/cal/01108a*-msk-1a.fits.gz testdata;

    The pipeline operators (or Frank) can help with querying a list of 
    consecutive scans (acquired in time-order) or within some time interval.
    The scan numbering is not necessarily in 1-to-1 corresponderence with time. 

    At the time of writing, it is not known whether the pipeline operators will
    create a gzipped tar-ball of all the intermediate frames over N scans
    following an incremental processing run. If so, this will make the copies
    easier. If the pipeline disks are cleaned before the frames are retrieved,
    then you'll need to ask that "wsfical with the -svlin switch" be re-run
    for the desired scans.
    

(2) Create band-dependent filelists of the above frames and masks:
    ls testdata/*-w1-lincor-1a.fits > ImageListB1.txt;
    ls testdata/*-w1-msk-1a.fits.gz > MaskListB1.txt;
    ls testdata/*-w2-lincor-1a.fits > ImageListB2.txt;
    ls testdata/*-w2-msk-1a.fits.gz > MaskListB2.txt;
    ls testdata/*-w3-lincor-1a.fits > ImageListB3.txt;
    ls testdata/*-w3-msk-1a.fits.gz > MaskListB3.txt;
    ls testdata/*-w4-lincor-1a.fits > ImageListB4.txt;
    ls testdata/*-w4-msk-1a.fits.gz > MaskListB4.txt;


(3) Set your environment to dev to pick up the latest s/w:
    caustic% newcfg dev
    Ensure you can see the "compflat" script and "flatcal" module, e.g:
    caustic(dev)% which compflat
    /wise/base/deliv/dev/bin/compflat
    caustic(dev)% which flatcal
    /wise/base/deliv/dev/bin/flatcal


(4) Take a look at the "makeflats" script in this directory. All parameters 
    have been tuned for the 30-orbit simulation (v3.2 testbed). You need
    only change the variables: "band" [=1,2,3, or 4] and "meth" [=1 for
    gradient method; =2 for classic stacking method]. From simulations,
    method 1 is optimal for bands 2,3,4, and method 2 is best for band 1
    (see test report referenced in (8) below). Familiarize yourself with
    all the I/O and options available for compflat. 


(5) Once input parameters are checked, you can execute "makeflats" in the 
    current directory by typing:
    ./makeflats
    Output products will appear in generic directories named:
    "Meth<1or2>Band<1or2or3or4>output/".
    E.g., the outputs for band 3 method 1 will appear in: Meth1Band3output/
    

(6) The main flat-field products in the output directories are (e.g., band 1):
    
    flat-w1-est.fits [main flat-field product]
    flat-w1-unc.fits [corresponding uncertainty map]
    flat-w1-msk.fits [corresponding mask image]
    
    The ./qa/ subdirectory contains useful plots and diagnostic FITS files.
    The ./meta/ subdirectory contains meta-data tables of QA metrics. See
    documentation referenced below for an explanation. All other files are
    to support analysis and QA. Method 2 generates a slew of intermediate
    files for further diagnosis.


(7) A word of caution: the "makeflats" script is set up to only use frames
    that are suitable for flat-fielding purposes via the "-filt" switch.
    This switch filters against the value of the FDYNAFLG keyword in
    FITS headers. If there are too many unusable frames (with FDYNAFLG = 0),
    then Frank will need to revisit the wsfical pipeline logs to explore which
    filtering conditions are not satisfying their thresholds, and then adjust
    the thresholds accordingly. There is a plan to tune all thresholds in IOC.


(8) Documentation/reference material:

    Brief online description of all "compflat" command-line I/O: type
    following at the caustic prompt:
    compflat -help2

    Documents explaining all I/O and algorithms:
    http://wise2.ipac.caltech.edu/staff/fmasci/sds-wsdc-D021-compflat.pdf
    http://wise2.ipac.caltech.edu/staff/fmasci/sds-flatcal.pdf
    ***N.B: the "flatcal" module is automatically executed from within the
            "compflat" script, never standalone.

    Flat-fielding logistics for mission:
    http://wise2.ipac.caltech.edu/staff/fmasci/dynacal_plans.txt

    Testing/tuning on 30-orbit simulation:
    http://wise2.ipac.caltech.edu/staff/fmasci/FlatTesting.html