Notes on coregistration of MOSES orders m = -1, 0, +1

C. Kankelborg, 2006 June 3 -

2006 June 3

Previous notes on alignment may be found in align_orders200605.html.

A new version of COREGISTRATE is online. Since my last entry on May 31, I 've discovered and fixed bugs in handling of missing data, both in the input filtering and in image warping. Following is a summary of new and revised routines:

COREGISTRATE Revised Improved handling of missing data
CK_CONVOL New Convolution with extensive options for missing data handling (demonstration).
DISPLAYSCALE New Scales data to 8 bits, with special handling of missing data. Incorporates a custom color table (demonstration).

Kicked off a new run of MOSES_ALIGN_ORDERS, moses_align_orders.log060603.184715

10pm. The alignment run is now on exposure 5. The old problem of huge chi squared has gone away. Reduced chi squared is roughly proportional to exposure time, which indicates that the residuals are mainly systematic differences between the zero and minus images (rather than random noise).

exposure nominal
duration 		reduced
chi squared
0 0.25 1.04
1 0.75 2.58
2 1.50 5.40
3 6.00 23.10
4 12.00 49.58
5 24.00 105.21
(est.)

2006 June 4

MOSES_ALIGN_ORDERS completed for the first time. It took 15.5 hours to align the negative and zero orders for all 27 exposures. Much of this time was taken by the 24s exposures: about 1 minute per call to CHISQ_BADNESS. My best guess is that POLY_2D takes a long time to process the NaN values encountered in saturated pixels.

Above is the diagnostic image from ALIGN_SUMMARY. All four tie points are positioned repeatably to sub-pixel accuracy.

Thoughts:

2006 June 12

Spent the last few days fixing problems with my test cases for COREGISTRATE. Intrinsically, COREGISTRATE is easily capable of < 0.05-pixel accuracy, and even better repeatability, even with smaller images and more missing data than we have with the MOSES analysis.

To do:

Looking at residuals

Ran EXAMINE_RESIDUALS on aligned data from minus and zero orders, Note that the zero order drift is still in, though the minus order has been aligned to the zero order.

Since the m=0 frames aren't aligned yet, I will stop here.

2006 June 17

I wonder whether the actual mapping from m=0 to m=-1 can be represented adequately using 4 control points. To investigate the possible need for quadratic terms in the horizontal axis, I have written DISTORTION1. This is only a back-of-the-envelope calculation, but it suggests that the second derivative contributes perhaps 0.13 pixel. It may be worth trying coregistration with 9 control points.

Is there any systematic trend of control point coordinates vs. number of saturated pixels? That might be one signature of systematic error associated with the above idea.

Can't seem to restore mosesAlignOrders.sav. Something is wrong with the save file, so I'll have to re-do coregistration:

IDL> restore,'mosesAlignOrders.sav'
% RESTORE: Unexpected item code found in save file.
% Wrong number of tags defined for structure: !ERROR_STATE.
% RESTORE: Structure not restored due to conflict with existing definition:
           !ERROR_STATE.
% Wrong number of tags defined for structure: !MAP.
% RESTORE: Structure not restored due to conflict with existing definition:
           !MAP.
% RESTORE: Unable to restore structure, tag type disagrees with existing
           definition: !PLT.
% Wrong number of tags defined for structure: !WARN.
% RESTORE: Structure not restored due to conflict with existing definition:
           !WARN.
% Wrong number of tags defined for structure: !AXIS.
% RESTORE: Structure not restored due to conflict with existing definition:
           !AXIS.
% Wrong number of tags defined for structure: !AXIS.
% RESTORE: Structure not restored due to conflict with existing definition:
           !AXIS.
% Wrong number of tags defined for structure: !MAP.
% RESTORE: Structure not restored due to conflict with existing definition:
           !MAP.
% Wrong number of tags defined for structure: !YS_IDLSYS_02.
% RESTORE: Structure not restored due to conflict with existing definition:
           !YS_IDLSYS_02.
% Wrong number of tags defined for structure: !AXIS.
% RESTORE: Structure not restored due to conflict with existing definition:
           !AXIS.

2006 June 19

Extreme errant pixel values from any source are likely to bias coregistration. Since there are a few particle hits in our images, and since the flat fields do not include some particle contamination that occurred during operations at WSMR, I have written IMSCRUB to clean up the images. This MOSES_PREP, along with several other improvements:

Significant Update to MOSES_PREP

THE ORIENTATION OF THE LEVEL-0 IMAGES HAS CHANGED TO SOLAR ORIENTATION.

Kicked off MOSES_PREP: moses_prep.log060619.122328.

Looking at the results. On many exposures, especially the short ones, it took many smoothing iterations to fill the bad pixels. Often went to 16 without converging. IMSCRUB identified way too many bad pixels on the short exposures.

I modified IMSCRUB to scale the threshold with the square root of the signal rather than linearly. Trying MOSES_PREP again: moses_prep.log060619.154000. This time, no bad pixels at all were found on many images, and still there were up to 12 smoothing iterations on other images. This behavior seems worse than before.

Changed IMSCRUB back to a threshold linear in the median-smoothed signal. Here is how I call IMSCRUB:

	print,'Exposure ',i,' order m = -1'
	cube_minus[*,*,i] = imscrub(cube_minus[*,*,i],        $
		image_marked=image_marked, /expand_bad)
Default settings are THRESH=0.5, MEDWIDTH=5. New run: moses_prep.log060619.173513

The results are still not good enough. The results depend strongly on exposure time. Too many bad pixels are found in the underexposed images. Where the intensity is near zero (off the limb), most of the pixels are deemed bad.

Reluctantly, I've written a revised digital filter with more adjustable parameters: IMSCRUB2. New run of MOSES_PREP: moses_prep.log060619.215318. The log file should now contain information on all tunable filter parameters.

The number of bad pixels appears to have stabilized at a few thousand per image, and most exposures require about 3 smoothing steps to interpolate values for the bad pixels. Still seeing a large number of smoothing iterations within IMSCRUB2 for the outboard orders on th 24s exposure (exposures 5 and 21). I don't understand why, but of course this is exactly the set of images that are most deeply exposed. Perhaps related to blooming. The filtered images and the bad pixel maps look reasonable, except that there are still more bad pixels than there really should be of for the 0.25s exposures, especially in m=0 since it is less exposed.

Kicked off a new run of MOSES_ALIGN_ORDERS: moses_align_orders.log060619.224236

2006 June 20

Alignment is still proceeding. Minus order is done, and plus order has just started. I've written a shell script, alignstatus, to check the progress of the latest run. Example:

mithra-kankel% alignstatus
Performing minus-to-zero coregistration for exposure       25
% COREGISTRATE: Starting alignment at Tue Jun 20 10:28:11 2006
% COREGISTRATE: Alignment completed at Tue Jun 20 10:35:33 2006
ncalls =          268
Performing minus-to-zero coregistration for exposure       26
% COREGISTRATE: Starting alignment at Tue Jun 20 10:37:41 2006
% COREGISTRATE: Alignment completed at Tue Jun 20 10:44:27 2006
ncalls =          228
Performing plus-to-zero coregistration for exposure        0
% COREGISTRATE: Starting alignment at Tue Jun 20 10:46:54 2006

Following is an alignment summary for the -1 order:

No improvement is evident from the addition of the flat fields. In fact, the consistency from exposure to exposure is slightly worse. This may have something to do with the digital filtering step; perhaps too aggressive?

2006 June 21

ALIGN_SUMMARY now does both outboard orders. Following is an alignment summary for the +1 order:

Now working on...

Final Alignment

...by which I mean having datacubes for all three orders mutually aligned to a single image (perhaps the first 12s m = 0 is a good choice).

I just found that restoring my level one data (mosesLevelOne.sav) was throwing errors on two of three Sun compute servers, probably because of IDL version conflicts. I think the issue is the /all keyword, which causes save to include system variables and other useless things. Recreating LevelOne data from scratch, to ensure that everything works correctly. This should be no different from what was used for MOSES_ALIGN_ORDERS, so no need to repeat that step, though the file creation times will now be out of order.

Back to the main point. The alignment work is to be completed by MOSES_ALIGN_FINAL, which calls DRIFT_COMPENSATE2. Running in batch mode; log file is here: moses_align_final.log060621.180647.

So, something is wrong with the badness function in DRIFT_COMPENSATE2. Badnesses in the thousands.

2006 June 22

Fixed the bug and ran. moses_align_final.log060622.115422.

Found more bugs after making a movie (see directory cck_science/). Put in fixes to MOSES_ALIGN_FINAL and DRIFT_COMPENSATE. New run moses_align_final.log060622.233842.

2018 Jan 1

Found mosesLevelOne.sav and mosesAlignFinal.sav missing! Note that the flat fields have been improved (per comments in moses_prep.pro) since last entry in this log. Re-ran MOSES_PREP today, see moses_prep.log180101.184859.

Ran MOSES_ALIGN_FINAL. Logfile: moses_align_final.log180101.201122.