======================================================
(RE)GENERATING PIXEL-CALIBRATION PRODUCTS FOR NEOWISER
======================================================

F. Masci, 1/20/2016

ssh caustic
newcfg dev

---
1. copy the previous (most recent) calibration products directory 
   to a new local directory, e.g:
   cp -rf /wise/data2/fmasci/ical_bckup/ifr_ops/031414v7.5/66113a001-99999z999/ YYYYYyYYY-99999z999/

   Go to step 2.

---
2. Create inputs for calibration of electronic-gain and read-noise.

   example:
   ./forgainrncal 2015-12-21T03:20:34.158 1296000
   where 1296000 sec => go back 15 days from supplied date/time range.

   Outputs: two files W?gainrndata.txt (3 columns each):
   intFuzMode intSigLTMADFM intMed16ptile;

   Go to step 3.

---
3. Using output files from step 2, compute gain and read-noise by
   performing a linear fit to lower envelope of points on plot:
   y = intMed16ptile**2 versus x = intFuzMode. 

   The linear fit will be:
   y = s*x + c where
   s = slope; c = intercept; 
   gain = 1/s [e-/DN]
   read-noise = sqrt(c) [DN]

   will need to use eyes to check that lower envelope is fit.
   For examples, see products in:
   http://wise2.ipac.caltech.edu/proj/fmasci/neowiser_gainrn.html
   
   Make a note of the gain and read-noise values for use below.

   Go to step 4.

---
4. Update the file YYYYYyYYY-99999z999/wise-meta.tbl
   * ensure "ical:debgain" values are set to gain values from step 3
     for bands 1 and 2. 
   * ensure "ical:uncscal" values are all 1.0

   Go to step 5.

---
5. Create read-noise maps by executing:

   ./makernmaps <sigstack.fits_for_w1> <sigstack.fits_for_w2> <rnsig_w1>
                <rnsig_w2> <cal_outdir>

   where:
   sigstack.fits_for_w? are the "*sigstack.fits" cal products from 
   a recent trending run; rnsig_w? are read-noise values from step 3;
   cal_outdir = YYYYYyYYY-99999z999. See top of the makernmaps script
   for an example.

   Go to step 6 *only* if calibration products related to pixel-uncertainties
   (gains, read-noise maps and unc-scaling parameters) need to be updated on
   ops. Otherwise, if other calibration products need updating (masks, darks,
   flats..), skip to step 8.

---
6. Send Tim and Ron a list of ~20 recent scans from ops to copy to pops for
   testing. Also notify Tim about your new preliminary cal directory:
   YYYYYyYYY-99999z999 for replicating under the pops environment.

   Following this initial test run under pops, go to step 7.

---
7. Following the initial pops-test run (following either step 6 or step 10,
   depending on the type of calibrations to be updated), create a file
   named e.g., "testscans.txt" containing the test scans. See supplied
   example for format. 

   Compute the uncertainty scaling factors by executing:

   ./uncscalesfrompops testscans.txt

   The values you are interesting in will appear at the end, e.g:
   global chi median for w1 = 1.001234
   global chi median for w2 = 1.000123

   write these values into the "ical:uncscal" fields of the 
   following table: YYYYYyYYY-99999z999/wise-meta.tbl

   Now skip to step 11.

---
8. You reached this step via step 6 only, i.e., because other calibration
   products also need updating besides pixel-uncertainty parameters.

   The following script auto-magically creates new masks, flats, and darks
   using products from a recent trending run. This script will update all
   relevant cal products under YYYYYyYYY-99999z999

   ./makefltdrkmsk <sigstack.fits_for_w1> <sigstack.fits_for_w2>
                   <stack-w1-est.fits> <stack-w2-est.fits>
                   <flat-w1-est.fits> <flat-w2-est.fits>
                   <cal_outdir> <testscan_list> <lownu_flatcor_flag>
                   <lownuonly_flag>

   where:
   the various *fits* files are from a recent trending run; see the top
   of the makefltdrkmsk for an example. cal_outdir = YYYYYyYYY-99999z999

   For this step, the makefltdrkmsk script should be executed with the
   following settings:
   <testscan_list> = "testscans.txt" (not used here, but specify as dummy file) 
   <lownu_flatcor_flag> = 0 (=> don't generate low-frequency flats here)
   <lownuonly_flag> = 0 (we want all preliminary cal products here; not only
                         low-frequency flats)

   When preliminary flats, darks, and masks are generated under
   YYYYYyYYY-99999z999, go to step 9.

---
9. Send Tim and Ron a list of ~20 recent scans from ops to copy to pops for
   testing. Also notify Tim about your new preliminary cal directory:
   YYYYYyYYY-99999z999 for replicating under the pops environment.
   Following this initial test run under pops, go to step 10.

---
10. Following this initial pops-test run, create a file named e.g., 
    "testscans.txt" containing the test scans. See supplied example for format.

    Now execute makefltdrkmsk again, using the same input fits files as
    the initial run in step 8, but now with the following setting:
    <lownu_flatcor_flag> = 1
    This tells the script to also generate and apply low spatial frequency
    correction flats to the high spatial frequency versions.

    The relevant products under YYYYYyYYY-99999z999 are replaced.

    Now go to step 7 (yes 7).

---
11. Tell Tim and Ron to recopy the updated cal directory YYYYYyYYY-99999z999 
    to pops and request that the same list of test scans be reprocessed.

    Now go to step 12.

---
12. Following the second (and hopefully final) pops run, you can perform
    the following two checks to validate the final calibrations. 

    * check #1 (pixel-uncertainties: ensure median chi values are ~1):
      ./uncscalesfrompops testscans.txt

    * check #2 (check that the low spatial frequency residual flats are ~1):
      these can be generated using the makefltdrkmsk script using the same
      input fits files as in step 8, but with the following settings: 
      <lownu_flatcor_flag> = 1
      <lownuonly_flag> = 1

    If the above ouputs look "good", you're done!

    Go to step 13.

---
13. Notify other analysts to check the products from the second pops run.
    If everyone is satisfied, tell Tim to deploy the new cal product
    directory to fops.

    Following deployment, go to step 14. 

---
14. After a successful calibration delivery, update the following page:
    http://wise2.ipac.caltech.edu/proj/fmasci/neowiser_cal_status.html