|
Investigation of WPHot flux biases | |
The following tests involved running WPHot on simulated images of a set of point sources superposed on backgrounds with various noise characteristics. The point source responses were generated using Ned Wright's PSF prescription. WPHot was then run on the image. The plots below show the resulting magnitude difference (estimated - true) as a function of assumed signal to noise ratio, where the latter represents the ratio of source flux to RMS background noise.
These plots show a progressively worse flux overestimation at the faint-source end. This is unexpected, since for pure Gaussian noise there should be no bias. In order to check whether this is an effect of incorrect background estimation, the background calculation in WPHot was overridden with the correct background value (1.0) for all sources. The simulation was repeated using a perturbed value of 1.001. Here are the results:
It is interesting that the WAPP fluxes are unbiased for SNR values down to 7.5, but below this the bias increases suddenly.
Dec 10 update:
Tom traced the sudden apparent onset of WAPP bias to the fact that sources whose aperture flux is less than 2 sigma are quoted as upper limits in the WPHot output source table, rather than estimated flux values. Changing the threshold to "0 sigma" produced the following result:
There is thus no bias in the aperture photometry. There is still a small bias in the profile-fitting photometry though:
WPRO result for the same data.
One thing which does seem clear is that the flux overestimation at the faint end is not a background effect, since it does not disappear when the correct background is used, and furthermore, it cannot be removed by applying offsets to the background.
Dec 16 update:
The WPRO flux (overestimation) bias at the faint-source end goes away when the source positions are fixed at their true values:
WPRO result for fixed source positions.
This is consistent with the hypothesis that the effect is due to WPRO going off and locking on to neighboring noise bumps when the S/N is low. However, what is not clear is why this effect gives a net positive bias, since noise bumps are just as likely to produce spurious negative fluxes as positive ones.
Dec 23 update:
The positive bias could be due to the
"horizon" effect. Specifically, if the estimated source position is
outside (or close to the edge of) the "fitting circle" containing the data
pixels, the responses of the PSF at the locations of those
pixels are systematically lower than for a PSF centered at the true
source position. After dividing out this response during the flux estimation
process, the result is a flux estimate which is systematically too large.
This is illustrated by the results of a simulation using the band 1 PSF
with two different values of the fitting radius.
Simulated flux biases due to the horizon effect
So the bottom line appears to be that the WPRO flux overestimation at low
S/N is due to flux errors induced by position errors. These errors are
positive due to the horizon effect with respect to the fitting radius.
2009 Jan 12 update:
If the horizon effect is responsible for the
low S/N flux overestimation, then we can eliminate it by restricting
the estimated positions to within the fitting circle, i.e. the
magnitude of the offset of the estimated position from the candidate
position should be less than rfit. To test this, here are the results
of restricting the maximum position offset to rfit/2, where rfit in
this case is 7.5 arcsec:
Flux bias result with
position offsets limited to a maximum of 3.75 arcsec
Unfortunately, that did not cure the problem. So the flux overestimation
at low S/N is NOT due to the horizon effect. Interestingly, even
though we get a bias when we allow the x and y position offsets to
vary up to a maximum value of rfit/2 (3.75 arcsec in this case), we
do NOT see a bias when the position offset is FIXED at rfit/2, as
the following plot shows:
Flux bias result with position offset
FIXED at 3.75 arcsec
The flux, in this case, is underestimated because the source position
has been deliberately offset from the true position, but there is
no trend with S/N.
To summarize these results:
1. The flux bias occurs at low S/N when the x,y position offsets are regarded
as variables in the profile fitting.
2. The flux bias is still present when the maximum position offset is
less than rfit, thus eliminating the horizon effect as a possible cause.
3. The flux bias does NOT occur when the x,y position offsets are fixed.
The estimated flux then is, in general, reduced from the true value, but
does not show the trend of increasing flux with decreasing S/N.
For the special case where the offset is forced to be zero, we do not
see any bias in the WPRO fluxes.
Since the bias is associated with the position solution at low S/N
and is not due to the horizon effect, it would seem that the only remaining
possibility is that at low S/N, the profile fitting
algorithm wanders off to neighboring noise bumps, and if it finds one
whose flux exceeds that of the true source, it adopts that as the
solution instead.
So we have come back to the Dec 16 hypothesis, and are still left with
the question of why the flux bias should be positive, given that the
noise has both positive and negative values. Could it be that there
is a hidden positivity bias in WPRO? To test this, the simulation
was rerun using NEGATIVE sources. The result was as follows:
Comparison of flux bias results for positive and
negative sources
As is apparent, a positive source results in the estimated flux going more
positive, while the converse is true for a negative source. This is what
should happen, so this indicates that there are no hidden positivity biases.
Evidently, the bias has the same sign as the flux of the true source.
The reason is that the
algorithm chooses a source position for which the absolute value of the
estimated flux is maximized. This is also why the peak value of a
matched filter occurs at the source position.
If WPRO is indeed preferentially fitting neighboring noise bumps, then
the chi squared values for these solutions should be lower than the
values which would be obtained by fixing the source position at the true
position. The following is a plot of the mean value of reduced chi squared
as a function of S/N for the two cases:
This plot shows that the fitted positions give lower values of chi squared
than do the true positions, which is the expected behavior if the algorithm
is being influenced by noise bumps.
Note: The reason that the reduced chi squared
is much less than unity for large S/N is that PSF errors were not simulated
during the generation of the synthetic data, and therefore the fit at large
S/N is much better than would be expected based on the WPRO noise model.
CONCLUSION: The WPRO flux overestimation at low S/N is due to the
procedure locking on to a neighboring noise bump whose flux happens
to exceed that of the true source. The flux error is still consistent
with the error bars of an individual source, but when the results for
a large number of sources are averaged, a net bias shows up.
The maximum likelihood procedure is guaranteed to give an unbiased
result for a linear measurement model, and in fact it does so
when the position variables are not solved for. However, the measurement
model is nonlinear in position and so all bets are off in that regard.
The flux overestimation at low S/N is then the penalty that we pay
for including variables for which the measurement model is nonlinear.
Jan 14 update:
If the above explanation is correct, then the bias should eventually
go to zero as the S/N decreases further below 3. In the limit of,
zero S/N there should be exactly zero bias, since there is no reason
for the algorithm to prefer positive noise bumps over negative ones.
To test this, synthetic data were generated for a range of source
fluxes, including zero flux, and the additive noise was held constant
at a value corresponding to a flux sigma of 1.0. A plot was made
of the mean value of the difference between estimated and true flux
as a function of S/N, as follows:
(Estimated flux - true flux) versus S/N
The filled circles (with error bars) represent the mean flux difference
as a function of S/N. These means represent the averages of about 3000
independent source extractions in the presence of Gaussian noise. The
results show that:
1. The bias is present over the S/N range 1-10. (Note that it starts
increasing again above S/N = 30, but the effect is unimportant in that range
since the bias then becomes a negligible fraction of the flux itself).
2. The bias goes to zero as S/N goes to zero, as we would hope.
For comparison on this plot, the solid line is a plot of the flux difference
normalized by the true flux. The solid line thus represents approximately
the magnitude bias. Its behavior is consistent with the previous plots
in that it shows essentially no bias for S/N > 10, but the bias increases
for lower S/N. For very low S/N, the solid line becomes meaningless,
because even after 3000 averages, the flux uncertainty is much larger than
the flux itself.
CAN ANYTHING BE DONE ABOUT THIS BIAS?
In principle, the answer is yes, because any systematic bias can
be at least partially removed, by definition. The bias in this
case is approximately equal to sigma/4 for 1 < S/N < 10. After
applying a correction of -sigma/4 to each flux (where sigma is the
WPRO flux uncertainty), the plot of flux difference versus S/N becomes:
(Estimated flux - true flux) versus S/N,
after correction by 0.25 sigma
The bias in the range 1 < S/N < 10 has now largely disappeared, although
we are left with a residual bias below S/N = 1. This result
is nevertheless encouraging, and suggests that by restricting the
range of the correction to 1 < S/N < 10, we could suppress the bias
for the fluxes of detected sources but prevent contamination of the
upper limits for the nondetected sources.
If we now regenerate our plot of WPRO magnitude bias versus S/N
using the "corrected" fluxes (after tweaking the correction factor
to 0.26 sigma), we obtain:
WPRO magnitude bias plot,
after correction by 0.26 sigma
The WPRO photometry is now essentially unbiased. Note that no change was made
to WPRO itself -- the correction was applied after the fact. This would be
the preferred option for WISE data also. Further work would be
necessary to derive a reliable correction, however, and this correction would
not necessarily improve the global performance -- the maximum likelihood
estimates do, in principle, give the least mean squared error over the
ensemble of possible source fluxes and noise values. So it is
not obvious that one would always want to apply a correction.
In any event, the
effect is only a quarter of a sigma, so even the uncorrected fluxes are still
correct within the error bars. If correction is required for some
purposes involving statistical analyses, the effect might be better left as
an item for the Explanatory Supplement, with a prescription for its correction
if desired. Hopefully this lays the issue of low S/N flux biases to
rest, at least for now.
Patches of sky
background were cut out of 01301b027-w1-int-1b.fits (the level 1b version of
one of Ned Wright's simulated images). The cutout portion was then randomly
shifted so as to fill in the whole focal plane, and appropriately
normalized.
Case 1: 3' x 3' background with minimal star
content.
Case 2: 15' x 15' background with
substantial confusion (many moderately bright stars).
Case 3: Same 15' x 15'
background; LBACK = constant.
Case 4: Same as Case 2,
except using a smaller fitting radius.
1. The flux overestimation for faint sources is not due to an incorrect
estimation of the background.
2. The midrange flux underestimation (the local maximum at moderate SNR) is a
confusion-related effect in WPRO, and can be mitigated by proper choice of the
fitting radius.
B. Backgrounds obtained from Ned Wright simulations:
This shows the flux
overestimation at the faint source end, but no evidence of the midrange
underestimation (the local maximum evident in Roc's plots).
This shows the local maximum
in the vicinity of SNR = 60, and demonstrates that it is
confusion-related. It does not show up in
the corresponding plot for
WAPP,
suggesting that it is in the profile-fitting. Note that
both aperture and profile fitting photometry show the flux overestimation
at very low SNR values, suggesting a common origin for that effect.
In this simulation, the local
background estimation in WPHot was overridden by the true (assumed) background
level which was set to a constant value over the field of view. The fact
that the midrange understimation persists indicates that it is not an artifact
of background estimation. It therefore suggests that the underestimation
is a confusion-related bias in
the profile fitting. One might therefore expect that such an
effect might be reduced by decreasing the fitting radius.
Specifically, the
fitting radius was reduced from 1.25*FWHM to 0.75*FWHM, where FWHM
is the PSF width. The midrange underestimation has now gone, thus supporting
the hypothesis.
Conclusions
Last update - 2009 Jan 14
K. A. Marsh - IPAC