Data reduction I

[FeII] line+continuum emission (1644 nm) in the planetary nebula Hubble 12

program names, file names
variable names
prompts, commands, program code

• Preface


• The runparams common block


• fnamegen


• Loading a flatfield


• Preliminary reduction...


• Background subtraction I


• Background subtraction II

Preface

This section begins the discussion of the actual mechanics in the reduction of data. The steps taken in this process are to be done in the same order as given in this documentation. Right now we'll assume that everything is in place to begin reducing data, ie., we'll assume that someone has already prepared the following data for the observing run during which the data were taken: linearity and saturation frames, a linearity weight frame, the saturation fraction, flatfields, distortion correction parameters, and platescale. Later chapters of this documentation will discuss how to prepare these data if they do not already exist.

One thing to start thinking about is how often you should save your work. I like to reduce several images at once and save as infrequently as possible. This saves disk space and avoids the confusion of having to sort through too many files. My personal preference is to save after having completed everything up to and including the steps described in the distortion correction chapter. If the data does not suffer from optical distortion, then I save after completing the steps described in the second data reduction chapter. Then I save again after tessellating and calibrating, and again after continuum subtraction (for narrowband images). But this is all a matter of personal preference---do what you think is appropriate, but try to strike a balance between saving work often enough that nothing is lost (in the event of a power failure, IDL crash, inadvertent user error, blah, blah) and taking up too much disk space (which is sure to incur the wrath of the local system administrator).

The runparams common block

The directory ~/idlpro/obsrunparams contains several files with names resembling the observing run to which they correspond. For example, oct98slow.pro is associated with the data taken with the "slow mode" (low Vbias) operating parameters on the oct98mlof observing run, and oct99.pro is associated with the oct99wiro observing run (which had only one data acquisition mode).

Running one of these files loads several items necessary for the reduction of data for that observing run. Included in these items are the linearity frame, a linearity weight frame, the saturation map, the saturation fraction, and the parameters used for the correction of optical distortion (not necessary for WIRO data -- only MLOF images).

The linearity weight frame is something I've found useful from time to time. It's a plane of ones and zeroes, the zeroes corresponding to pixels in the linearity frame which have values that tend to adversely affect the data images. This isn't always necessary, but the linearity program expects it. If a weight plane isn't necessary, I just use a plane of ones (which have no effect) as the linearity weight plane.

A file of data reduction parameters like this must be created for every new observing run, with the appropriate values and filenames supplied. Obviously, this can only happen after linearity (and the distortion correction analysis, if it's an MLOF run) has (have) been done.

After running the program, the user must type the following command:

IDLprompt> common runparams

This puts all that run data in a place in memory where the data reduction programs know where to look for it. This way, for example, the linearity program may be run by only sending it the data image to be linearized -- the saturation fraction and the linearity, linearity weight, and saturation frames don't have to be explicitly given to the linearity program, because the program knows where to get those things.

fnamegen

Typically, it's easier to reduce a large number of images all at once than to reduce them one at a time. Let's pretend that you want to reduce a bunch of K-band images in /home/carl/obs/oct99/24oct/s235a/ , images s235a.005 through s235a.015 . Your first step should be to run the ~/idlpro/fnamegen program:

UNIXprompt> ~/idlpro/fnamegen obs/oct99/24oct/s235a/s235a. 5 15

(notice that this is run from the UNIX prompt, NOT in IDL). This will edit the file ~/idlpro/flist to make a list of the files whose images you want to reduce. Make sure that the directory name (the first argument above) goes out to (including) the period -- the program will supply the file numbers (from 005 to 015, in the above example, the output of which is shown here).

A very important thing to keep in mind is that the directory name should be specified relative to the working IDL directory. If the directory names in ~/idlpro/flist differ from the working directory in IDL, IDL won't be able to locate the files listed in ~/idlpro/flist.

As a second example, let's assume that you want to reduce a lot of data that was taken non-consecutively. Let's pretend that you want to reduce images 5 through 15 in /home/carl/obs/oct99/24oct/s235a AND image 12 through 18 in /home/carl/obs/oct99/28oct/s235a . You'd run fnamegen twice, with an extra argument the second time, so that the second bunch of files is appended to the end of the first (otherwise fnamegen overwrites anything in ~/idlpro/flist):

UNIXprompt> ~/idlpro/fnamegen obs/oct99/24oct/s235a/s235a. 5 15 UNIXprompt> ~/idlpro/fnamegen -a obs/oct99/28oct/s235a/s235a. 12 18

(note the -a switch in the second command).

fnamegen has several other features for more complicated filenames. See the documentation in the file (fnamegen) for details.

Loading a flatfield

In general, image data must be flatfielded. Let's assume that the flatfield to be used for the data to be reduced is /u/irdata/dsp/oct99wiro/flatfields/ 2166sky.fts. To load the flatfield, run the flatfile program:

IDLprompt> flatfile, '/u/irdata/dsp/oct99wiro/flatfields/2166sky.fts', flatpackname

The single quotes ARE necessary -- this argument is the filename and is treated as a character string by IDL. The final argument is the name of the variable which will reference the flatfield image package (flatpackname). This name is completely arbitraty and at the discretion of the user. A flatpack is similar to an impack, but there are subtle differences -- a flatpack MUST be loaded with flatfile, NOT rfits.

Depending on how the flatfield was originally stored, flatfile may or may not ask for a nickname for the flatpack. If it asks for a name, type in something meaningful, like oct99wiro 1644nm CVF flat. Or (if asked for a nickname) you can just hit then ENTER key, and flatfile will just use the filename (/u/irdata/dsp/oct99wiro/flatfields/2166sky.fts in the above example). Flatfile may not request a nickname -- in this case, an identification appears in the flatpack header. The importance of the nickname is that it is written to the header of any data images flatfielded with the flatpack.

Preliminary reduction:   load_list -- linearizing and flatfielding

The load_list program loads a series of files from disk (specified in the ~/idlpro/flist file, as generated by ~/idlpro/fnamegen), linearizes them, and (perhaps) flatfields them. The output of the program will take the form of two arrays of impacks, one of which will be the original images, and the other will be the linearized and (perhaps) flatfielded images. The previous statements say (perhaps), because you can run load_list without a flatpack, in which case the images are not flatfielded. The load_list program is run with the following command:

IDLprompt> load_list, raw, lf, flatpackname

As mentioned in the section about the fnamegen script, load_list must be run from the directory such that the filenames in ~/idlpro/flist have the correct relative pathnames. After runtime, raw and lf will each be an array of impacks. The elements of raw are the original, unchanged images (as specified in ~/idpro/flist), and the elements of lf are the linearized and (perhaps) flatfielded images. If the final argument of the above command is omitted, only linearization occurs. The header files of the impacks in lf will reflect the changes made by load_list.

The individual elements of raw are referenced just like the elements of any other IDL array: raw(0) is the impack corresponding to the first FITS file listed in ~/idlpro/flist . lf(0) is the linearized and (perhaps) flatfielded version of the image. The following commands may be used to view the fourth original image and the header of the sixth reduced image:

IDLprompt> v, raw(3), 2, sig=5, /c IDLprompt> v, lf(5), /h

IMPORTANT NOTE: the images in raw do not have weight planes, but the images in lf do have weight planes. A weight plane is added during linearization. Any pixels that the linearization routine finds to be saturated are "turned off" by setting the corresponding pixels in the new weight plane to zero. Nothing happens to the weight plane, however, during flatfielding---only the image plane is affected.

Impacks can also be linearized and flatfielded individually:

IDLprompt> linez, impackname IDLprompt> flatfield, impackname, flatpackname

Both programs alter the contents of impackname, and both programs alter the header file to document the changes. load_list calls linez and flatfield, so the header files record the data reduction whether or not load_list is used. Again, linez introduces a weight plane (which is unaffected by the call to flatfield).

The attentive reader will have recognized that load_list, linez, and flatfield are all IDL procedures (as opposed to functions).

Background subtraction I

We'll talk about two ways to deal with background-subtracting the data. The second way is useful if you can somehow use a single image as the background frame for all your data images. This might be the case if you choose to median-filter a bunch (or all) of your data images to derive the background frame. This way is convenient, as it lends itself to a more automated approach. The other method, which we'll discuss first, is appropriate if your data needs more individual attention than is allowed by the more automated second method Background subtraction is accomplished using the bkgsub function:

IDLprompt> newimpack = bkgsub( src_impack , bkg_impack )

where src_impack and bkg_impack are impacks with the source and background images, respectively. bkgsub creates an impack whose image is the background-subtracted image and whose header is a modified version of the header from src_impack -- this header is altered to specify the filename of the image used as the background image. The impack created by bkgsub always has a weight plane. If both inputs had weight planes, the result's weight plane is the logical AND of the input weight planes. If only one of the two inputs has a weight plane, that weight plane becomes the weight plane of the output impack. If neither input has a weight plane, the output impack has a weight plane whose every pixel has a value of 1.

I find it useful to run bkgsub in a batch file, especially if I've used load_list to make a large array of impacks, each of which needs to be put into a background subtraction. A batch file in IDL is like a batch file in other contexts, or like a script file in UNIX -- it's just a list of commands to be run by the interpretter, one after the other. It just saves the user the trouble of typing each command individually.

Let's say that lf is an array of six linearized and flatfielded impacks that need to be background subtracted, and we'll assume that the impacks alternate between background and source iamges. We could create a file called idl.bat whose contents are the following:

blf1 = bkgsub( lf(1) , lf(0) )
blf2 = bkgsub( lf(3) , lf(2) )
blf3 = bkgsub( lf(5) , lf(4) )

Background subtraction II

This second method is less technically complicated, because it leaves the data images in an array of impacks very much like the output of the load_list procedure. The IDL routine of interest here has the catchy name bkgsubimpackarr (rolls right off the tongue, doesn't it?). Let's assume that we've run load_list from a list of 14 data images, so that we have an array lf of 14 image packages. And let's assume that we have magically made an image package bkgframe whose image plane we can use to background-subtract all the images in lf. Consider the following four commands:

IDLprompt > use_arr = [ 2 , 3 , 4 , 5 , 7 , 9 , 12 ] IDLprompt > lfb1 = bkgsubimpackarr( lf , bkgframe ) IDLprompt > lfb2 = bkgsubimpackarr( lf ) IDLprompt > lfb3 = bkgsubimpackarr( lf , use_array = use_arr )

lf is not changed by any of these commands. lfb1, lfb2, and lfb3 are all arrays of image packages, each containing 14 image packages (where 14 is the number of impacks in lf). bkgsubimpackarr creates lfb1 by simply subtracting the image in bkgframe from each image in lf (and by putting a generic annotation in all the headers). For the third command, bkgsubimpackarr makes a background frame by median-filtering all the images in lf, subtracting this background from each image in lf, and assigning the result to lfb2. The fourth command does nearly the same thing as the third command, except that for the fourth command bkgsubimpackarr uses only the image packages indicated by the elements of use_arr (these numbers are treated as array indices for lf), so the background frame will be made by median-filtering seven of the images in lf.

Should you want to keep a copy of the median-filtered background frame, you can use the medfilter function -- the following two commands

IDLprompt > bkgimpack = medfilter( lf , use_array = use_arr ) IDLprompt > lfb4 = bkgsubimpackarr( lf , bkgimpack )

have exactly the same effect as

IDLprompt > lfb5 = bkgsubimpackarr( lf , use_array = use_arr )

(lfb4 and lfb5 will be identical).