Dynamic Calibration Plans 

                           F. Masci, 6/2/2009


1. We have two types of dynamic calibrations. Applied in ICal in this order:
   - responsivity (flat) correction. Inputs are "linearized" frames; 
   - skyoffset (or illumination) correction. Inputs are "flattened" frames;

2. Caveat:
   We need the best flats to make intermediate (flattened) products for
   the skyoffset calibration product.
   => too much latency incurred to make and apply running time dependent
      flats since we expect needing ~1.5-2 orbits worth of frames to make
      a good S/N flat.

3. Proposal for *first* processing pass:

   3.1 for 1st pass processing: make the best representative flats offline 
       with allowance for dependence on expected transients, e.g., anneals.
       Make once for application to a whole range of orbits and periodically
       check stability (see 4 below). Apply these in ICal PL and exercise
       the automated dynamic skyoffset calibration (latter concurrent).

   3.2 while above is happening, always save intermediate (linearized)
       products. Filtering (usability) flag for flat creation is already
       assigned by ICal. These products need to be moved off the cluster
       periodically to avoid filling up the disks or deleted when a flat
       is made therefrom. Alternatively, when an analyst is ready to create
       a new flat from data in a specific timespan, they can just ask that
       the ICal pipeline be rerun with the intermediate products saved.

4. Ongoing activity concurrent with 3 (manual at first then automated):
   periodically make flats over time-windows, tag with time-range of  
   applicability and QA, and archive for use in 2nd pass processing.
   Possibly also update flat products in 1st pass processing.
   
5. *Second* pass (final) reprocessing:
   Apply appropriate flat using time-window matching and execute
   dynamic calibration pipeline for skyoffset creation/application.
   Flat archival shall use a simple time-tagged dirctory structure.