Set processing -- multiple image instruments

In the simplest case, each exposure made on an instrument will read out one CCD using one amplifier/ADC and provide you with one data file containing an array of pixels which represents a single data or calibration frame. However, many instruments are now more complicated than this, and may feature multiple amplifiers reading out different parts of the same chip, or multiple chips on the same mosaic camera for each exposure, or a combination of the two. The resulting data product may consist of one or more data structures (e.g. NDF structures or FITS Header + Data Units (HDUs)) in each of one or more containing files.

Two problems arise when dealing with these sorts of data. Firstly, during the data reduction stage, calibration must be carried out in the same way for data read from the same bits of hardware and differently for data read from different bits; a separate master bias frame must be generated for each CCD in a mosaic camera and each data frame must be calibrated using the corresponding one. Secondly, during the registration stage, alignment information which corresponds to the positioning of the chips on the focal plane of the instrument may be accurately known and not subject to change, and this must be communicated to the software which is attempting to align separate data frames using other methods such as object matching; there's no point trying to identify matching features on two frames which you know are glued next to each other on the focal plane.

Both of these can be addressed to a large extent by careful use of the facilities of CCDPACK described so far, but to make processing of this kind of data more painless, CCDPACK provides the concept of a Set of images. The idea is that each exposure from a given instrument produces a single Set of data which can contain several frames. All Sets from a particular instrument will consist of a fixed number of members: each member of a given Set will share the same Set Name attribute indicating its origin, and has its own Set Index attribute indicating its rôle, or position, within the Set. Every exposure generated by a four-CCD mosaic camera for instance would produce a single Set with a unique Set Name, and four members with Set Index values of 1, 2, 3 and 4.

Using this functionality is quite simple: the user tells CCDPACK how all the data files are grouped into Sets, and from then on it can make sure that processing is done accordingly. The way this works is that a CCDPACK Set header is added to each file (in the .MORE.CCDPACK.SET extension of the NDF) which contains two items, the Set Name attribute (a string) and the Set Index attribute (an integer, usually 1, 2, 3, ...). Set alignment coordinates may also be added as an attached coordinate system, with the Domain name CCD_SET, if a known alignment is to be associated with the Set (because you know how the chips are arranged on the focal plane). The Set itself does not exist as a separate file or other entity, so you never need to specify a Set as such when giving input parameters to CCDPACK programs, but when presented with a list of input files, they will read and make use of Set header information if it exists. In fact in most cases, after adding the Set header in the first place, you can forget about Sets altogether and let CCDPACK worry about them for you.

All the CCDPACK programs which can do Set-sensitive processing consult the USESET global parameter to see whether they should use Set header information if it exists. To make CCDPACK use Set headers, you should use CCDSETUP to set USESET true like this:

% ccdsetup useset=true accept

All subsequent commands will then use Set headers where they have been added. Alternatively you could specify USESET=TRUE on the command line of each CCDPACK program. Since files without Set headers are always treated the same whether USESET is true or false, it is harmless to have set it true all the time, but in certain cases (especially if a non-native data format such as FITS is being used) it may result in slower operation.

Note that Set Name attributes don't need to be (and in general will not be) totally unique between different Sets; for instance DEBIAS will produce output files which have the same Set Names as the corresponding input files. However, no ambiguity will arise as long as only one Set of files with the same Set Name is presented to any given CCDPACK program at the same time. Normally, because of the flow of data between applications, that situation will not arise, but if for some reason two Sets with the same Name do need to be presented in the same context, MAKESET can be coerced into assigning a given Name to a Set by use of its NAME parameter.

The following sections give some detailed description and examples of how to manipulate Set header information and how CCDPACK goes about using it.