### B Specdre

#### B.1 Introduction

This appendix contains information on version 1.1 of the Specdre package (for SPEctroscopy Data Reduction) which has been added to Figaro. Sections B.6 and B.7 give examples of using several applications together.

Specdre is a package for spectroscopy data reduction and analysis. Some of the general features of the package are:

• Hyper-cubes: The Specdre data set is in general a hyper-cube where each row or hyper-column is a spectrum. Even where a single spectrum is required as input, this can be an appropriate section of the hyper-cube cut out “on the fly” as the application accesses the data.
• Coherent storage of fit results: The results of line or continuum fits are stored along with the data. In the case where a hyper-cube is a coherent set of spectra, fit results will also be stored coherently. For example, in a three-dimensional data set the two-dimensional map of line integrals is immediately available to display routines.
• Bad values and variance: Bad values (or quality information) are recognised and ignored or propagated, as appropriate. If present, variance information is propagated or used in the processing, e.g. for statistical weights. It can optionally be ignored. Where covariance is created (namely re-sampling), an approximate measure of this is stored along with the data. Other applications (namely fit routines) will use the ordinary variance or the measure of covariance, as appropriate.

The topics addressed by the applications are mainly:

• ASCII I/O: The data and errors of hyper-cubes can be written to or read from printable/editable tables. Bad values are converted between the two formats. Single spectra can be read even if the axis data are not linear or monotonic.
• Graphics: Display applications allow full control of the plot, including font, colour, line styles, error bars, etc. Overlay on previous plots according to their “world coordinates” is possible. This includes overlays on grey/colour/line plots made by KAPPA, Pongo, etc.
• Cube manipulation: You can extract averaged hyper-planes from hyper-cubes, assemble hyper-cubes from hyper-planes, or fill in a hyper-cube from several given hyper-cubes.
• Arc line axis (wavelength) calibration: While full user interaction via graphics is granted, automatic arc line identification is also possible.
• Re-sampling: The application for re-sampling can either re-sample all spectra in a hyper-cube, or re-sample and average into one spectrum any number of input spectra. It allows information about the covariance between pixels to be carried through to a line fit routine.
• Spectral fits: You can fit polynomials, blended Gauss or triangle profiles. Fit results are stored along with the data and can be turned into fake data sets for later subtraction, division, etc.

Specdre uses the NDF data access library, which allows you to specify sections rather than the whole data set. Also, for the special requirements of spectroscopy data reduction and analysis, an extension to the NDF format is used which stores additional information with the data, thus allowing much enhanced communication between Specdre applications.

#### B.2 Specdre’s use of parameters

Some parameters used by Specdre are common to several commands. The device parameter is sometimes associated with the global parameter GRAPHICS_DEVICE. When it is, it usually defaults. And these parameters are really global, in the sense that other packages may use and change them, too.

Where in and/or out are NDFs, they are mostly associated with the global DATA_ARRAY. The effect is that the default input is usually the output of the previous command.

info and dialog are always associated with SPECDRE_INFO and SPECDRE_DIALOG. These parameters control the amount of information and user interaction of many applications. Once info is switched off all applications will become quiet until the parameters are set true again.

varuse is a defaulted parameter to many applications, but not associated with a global parameter. By default it is true. Sometimes it has to be set false in order to ignore variance information in the input data.

Other parameters like start, step, end occur naturally in several applications. In some instances they may be scalars, in others vectors. Often their defaults are set by the application with knowledge of the data set at hand.

#### B.3 Graphics

The management of graphics output closely follows that of KAPPA. To make full use of the graphics capabilities, you will need to use some KAPPA commands; and you will find Pongo extremely useful to add almost anything to your plots. For normal use you can get along without KAPPA or Pongo. The actual plots are achieved through a combination of AGI, SGS, PGPLOT, and GKS. The graphics is done with PGPLOT, but the device is handled via AGI and SGS, and GKS is the low level package underlying them all. AGI stores information about the graphs produced in a data base. This data base tells co-operating applications where and on which device a plot was made and what its world coordinates were. Other packages use and update the same data base so that a consistent display administration can be achieved, even when different packages are used in turn. The overlay options in Specdre applications use this information and allow you to plot with Specdre in the right place on top of, for example, an image displayed in grey scale or colour with KAPPA.

Usually a command whose main task is to produce a plot, has a parameter device which is not prompted for and which is associated with the global parameter GRAPHICS_DEVICE. The value of this global parameter will be the name of a graphics device, and the named device will be used for the display. But you can always specify the device parameter on the command line, thus overriding the global parameter:

ICL> specplot device=graphon
ICL> specplot device=xw
ICL> specplot device=ps_l

This will also change the global parameter so that next time you use the same device automatically.

There are other Specdre commands that may or may not use a graphics device. These will prompt for the device parameter and offer the null value as default. This can be accepted to avoid using a graphics device. When specifying parameters on the command line, device=! may not always work, but including the accept keyword will have the desired effect.

Some applications not only display graphics, but use a graphics display plus mouse and keyboard to conduct a dialogue with the user. This is usually optional and controlled by the dialog parameter. dialog is a character parameter. It is always allowed to be one of the letters {Y,N,T,F,y,n,t,f}. Sometimes it may also be G or g for graphics dialogue. y,t may or may not mean the same as g.

If you specify a printer or PostScript device, this may fail for the graphics dialogue case. But otherwise all plots can be sent to files that you print later. There is one important difference, you have only one screen, but the printer has many sheets of paper. Your plot may be in a number of printer files and each printout starts on a new page. If you have done overlay plots on the screen and want the same on the printer, then you can use as graphics device an Encapsulated PostScript device like epsf_l. You still get a number of files, but they can be merged into one (without form feed) using psmerge (cf. SUN/164). Usually the output is a complete PostScript file ready to be printed.

#### B.4 Slicing data sets

Specdre uses the NDF routines to access data in Starlink Data Files (SDF). This allows the user to specify sections of NDFs rather than whole (i.e. base) NDFs. Some applications need one-dimensional input data, but by themselves offer no means to take a subset of a larger or multi-dimensional data set. The desired effect can, however, be achieved by NDF. As an example, fitgauss will fit a spectrum only and rejects two-dimensional input. You still can use a 2-D data set by specifying a section thereof as input to fitgauss:

ICL> fitgauss in=myndf(,5)               { 5-th row of 2-D input
ICL> fitgauss in=myndf(5,)               { 5-th column
ICL> fitgauss in=myndf(20:30,5)          { pixels 20 to 30 of 5-th row
ICL> fitgauss in=myndf(6540.0:,5)        { pixels beyond coordinate 6540.0
ICL> fitgauss in=myndf(6450.0~10.0,5)    { coordinate range 6540.0 +- 5.0

(For fitgauss the sub-setting within the row is not necessary, because it does its own masking of the given 1-D data.) It should be mentioned here that the NDF fed into an application need not be at the top level of its container file. Once your NDF got a Specdre Extension (Section 3.2.7) with fit results in it, you can e.g. plot the second fit parameter versus the row number:

ICL> specplot in=myndf.more.specdre.results(2,1,)

#### B.5 Cube manipulation

A spectrum may be thought of as a one-dimensional data set. But spectroscopists are also aware of the two-dimensional space of sky position and might use time as an axis in data sets. So the data handling aspects of spectroscopy are more demanding than image processing – Specdre applications may encounter data with any dimensionality. Two-dimensional detectors are often used to take spectra and where observations are not very time-consuming three-dimensional data sets may be quite common.

Specdre can work on data with any dimensionality. The limit is 7-D due to the HDS data access routines. In practice the limit may be 6-D since one structure in the Specdre Extension has one dimension more than the main data array.

However, Specdre is about spectra. An N-D cube is taken as a set of spectra, each spectrum is a row or a column in the cube. A row extends along the first axis of the cube, while a column extends along any axis of the cube. In any cube Specdre assumes that there is exactly one spectroscopic axis, by default this is the first axis. The Specdre Extension specifies which axis is the spectroscopic one.

Specdre’s handling of N-D data falls into three categories.

• Some applications work on one spectrum at a time. They will insist on 1-D input, but you can specify a column to work on. Often the column must extend along the spectroscopic axis. These applications can be used successively on several or all columns of a cube. Insofar as they produce output will they work “in situ” and modify the input file.
• Other applications take on a whole cube at once and work on each row in turn. Usually spectra have to be in rows and the spectroscopic axis must be the first axis. These applications may also refuse to work on only a section of the input.
• Then there are applications that don’t deal with spectra at all but are tools to manipulate cubes. It is important that Specdre has such a set of tools, since the Specdre Extension may have a number of cubes in it as well. The main and extension cubes must be manipulated in a consistent manner, or the Extension becomes invalid. The fact that the cube manipulation routines handle only version 0.7 of the Specdre Extension and not version 1.1, means that the COORD component of the Extension is lost when these routines go about their business.

subset is very similar to KAPPA’s ndfcopy or to taking an “on the fly” section as input to an application. The differences are that subset also takes subsets of NDFs in the Specdre Extension (v. 0.7) consistent with the subset of the main NDF, and that it removes “degenerate” axes. Consider the command

ICL> subset image(5,1:10) row

When subset gets the input it is still an image of size 1 by 10. But in the output the degenerate axis has been removed so that it is also officially 1-D.

Where subset may remove axes, grow deliberately adds new axes – degenerate or genuine ones. So we could reverse the command above with

ICL> grow row expand=[1,0] stapix=[1,0] endpix=[1,0] size=[1,0] ~
new=y out=image

The zeros as second vector elements just show that the second axis of image matches the axis in row. expand shows which of the output axes are and are not in the input. Normally those new axes will of course not be degenerate, so size might be [5,0]. In that case row could be copied into any of the output columns, into one of them or repeatedly into a whole range of columns. The main idea of grow is that you assemble rows into images, images into cubes etc. So when new=n then you will copy into an existing file. The following puts two spectra and one image into a common cube. Whatever part of the cube is not copied to, remains filled with bad values.

ICL> grow row5_1 expand=[0,1,1] stapix=[0,5,1] endpix=[0,5,1] ~
size=[0,5,3] new=y out=cube_x_by_5_by_3
ICL> grow plane2 expand=[0,0,1] stapix=[0,0,2] endpix=[0,0,2] ~
new=n out=cube_x_by_5_by_3
ICL> grow row3_to_5_3 expand=[0,1,1] stapix=[0,3,3] endpix=[0,5,3] ~
new=n out=cube_x_by_5_by_3

If grow creates and expands new axes, xtract collapses existing axes. This is done by averaging the pixels along each collapsed axis. Note that this is an average and not a sum, which makes a big difference for the use of input variances and the meaning of output variances. Basically an average assumes that all values entering the mean are equal and scatter at random.

grow copies input into output of higher dimensionality, the common dimensions must match. fillcube is different. It copies input into output of the same dimensionality. Dimensions need not match, the copy is positioned in output by matching centre coordinates. Indeed the copy may not be contiguous in output. The output is an existing file, so you can fill it successively from different input files. This is mosaicing in N-D.

resample, too, plays a rôle in cube manipulation, since it can homogenise and linearise the coordinates along the spectroscopic axis. When used in mode=Cube it re-samples each row of a cube individually. Afterwards all rows have the same linear coordinate grid as expressed by the new vector of centres for the spectroscopic axis. Any spectroscopic values in the Specdre Extension are thus obsolete and removed. Sometimes it is necessary for other applications that grids are linear or that there is no array of spectroscopic values in the Specdre Extension.

#### B.6 Spectral fits

Specdre has a number of applications to fit analytical functions to spectral features. Two are line fit routines for Gauss and triangular profiles. These can fit up to six components at a time. The lines can be blended and the line parameters can be free, fixed, or tied to the corresponding (free) parameters of another component. A similar routine fits up to six diluted Planck curves. Finally, a polynomial fit can be performed, the order can be up to 7.

The fit routines can run with a (non-graphic) dialogue or not, they can display data and fit graphically at different stages of the fitting process (masking, guessing, fit residuals).

These fit routines work on one-dimensional data only. But you can pass an NDF section that is (part of) a row or column in an image or cube. For the fit only data inside the union of up to six masking intervals are used. The fit results go first of all to the standard output (the terminal), but can also be recorded in (appended to) an ASCII file. In addition fit results will always be stored in a results’ NDF in the Specdre Extension. Those results can be used to generate a model data set. You could then subtract that from the original data.

Here is an outline of a complex example showing how you might proceed with a long-slit spectrum (but note that the example will not work as presented since some parameters are omitted for brevity). Also, the telluric correction may not be the correct way of doing things. The purpose of the example is to illustrate how you can inter-operate Specdre and KAPPA applications.

...
ICL> evalfit  image conti comp=1                                       { 2)
ICL> div      image conti normal                                       { 3)
...
ICL> evalfit  normal tellur1 comp=3                                    { 5)
ICL> sub      normal tellur1 stellar1
ICL> cadd     tellur1 1.0 tellur2                                      { 6)
ICL> div      normal tellur2 stellar2
• For each row of the image use the abscissa ranges [1;2], [3;4], and [5;6] to fit a polynomial continuum. The two intervals excluded here probably contain spectral lines. For each row the parameters describing the fit are stored as the first component in the results’ NDF in the Specdre Extension of image. You would probably run the fit routine with dialogue and graphics the first time round to play with the mask intervals.
• For the whole image use the stored result to generate a corresponding data set representing the fitted continuum.
• Use a KAPPA command to divide the original image by the continuum.
• For each row of the normalised image use the intervening abscissa ranges and fit three Gauss profiles to them. These are stored as components 2, 3 and 4 in the Extension. Again you would use dialogue to play with masks and parameter guesses in the first use of fitgauss. For subsequent image rows you would try without dialogue and just recall the command line to edit the row number.
• The second line fitted, the third component stored, is not stellar but terrestrial, subtract it with the KAPPA command sub.
• Use KAPPA’s cadd to turn the simple telluric lines into proper telluric spectra with continuum at 1.0. Then divide the normalised spectra by the telluric spectra.

#### B.7 Arc spectrum axis calibration

There are four applications to do a wavelength calibration, or at least a calibration into spectroscopic values anyway. You can use frequencies or photon energies if you feel like it. You can also use nanometre, micrometre or Ångström, as you please.

At the heart of this axis calibration is an algorithm written by Mills (1992) to automatically identify features in an arc spectrum. For this to work, you must have a data base of arc feature identifications rather than just a simple line list. You can use arcgendb to convert a line list (as distributed with Figaro) into a “feature data base”. Unfortunately the data base takes some time to build and is also rather big, 10 to 100 times bigger than the simple list. So there may be a point in taking only the relevant wavelength range from long line lists like Th-Ar and converting it into a feature data base.

With such a data base at your disposal, you still cannot run the auto-identifier. This is because the calibration procedure as performed by Figaro’s arc is broken up into three steps:

• In the first step you go through the arc spectrum and locate features, that is you find out where in pixel coordinates the observed line features are. For this first step you normally use arclocat. Usually this first step will be done in two passes. In the first pass you run arclocat dialog=f so that it tries on its own to find un-blended narrow emission features in the arc spectrum. In a second pass you run arclocat dialog=g and use the graphic dialogue to improve the set of located features. You may add features not found before, or delete features that are blended or known to be absent from the feature data base. In general you should locate as many features as possible, you can always leave some un-identified.

Instead of arclocat you can also use fitgauss or fittri, in case that the simplified line fit in arclocat is not good enough. arclocat has modes Gauss and triangle, too, but it fits only one line at a time.

• Only in the second step do you auto-identify the features, that is the Mills algorithm will try to establish for some of the located features what their identification might be in terms of wavelength, frequency or whatever you use in the feature data base. This second step is performed by arcident. The result must be regarded with some scepticism, since there is a small chance that arcident will find a grossly wrong solution or make a slightly unfavourable selection of features to identify. Any such inadvertencies can be corrected in the third step.
• The third step again may or may not be in a graphics dialogue. arcdisp dialog=g will display a plot of wavelength, frequency etc. versus pixel coordinate. It does not show the spectrum. Instead vertical lines indicate unidentified (but located) features, horizontal lines indicate all possible identifications from the feature data base, and crosses indicate identified features. Finally the would-be dispersion curve is displayed. You can now add or remove identifications with mouse and keyboard by clicking on the intersection of the feature location (vertical line) and potential identification (horizontal line).

The improvement of feature identification is one goal of arcdisp. The other is to do a polynomial fit to the pixel-wavelength relation and to convert pixel coordinates into wavelengths using that fit. This latter goal can also be achieved without the graphics dialogue with arcdisp dialog=n. This does not replace the pixel coordinates of the main NDF, but creates a new array of spectroscopic values in the Specdre Extension.

arclocat, arcident and arcdisp in general work not on a spectrum, but on an image or cube that has spectra in its rows (rows, not columns). arclocat dialog=g allows you to switch from one row to another as you please, arclocat dialog=n will scan through all rows in sequence. arcident will do an independent auto-identification on each spectrum, so it is suitable for a collapsed échellogram where successive extracted orders are in the rows of an image.

arcdisp dialog=g will work on each row in sequence, you can work on one row and proceed to the next in your own time. You can quit at any time; this will leave the file without spectroscopic values, but any improved feature identifications are kept. If you step through all rows, then the spectroscopic values will be kept as well. (You can have spectroscopic values in the Specdre Extension either for all rows or none at all!) You may want to set the label and unit for the spectroscopic values after arcdisp with KAPPA’s setlabel and setunits.

So far you have only succeeded in producing an array of calibrated spectroscopic values in the Specdre Extension of the arc spectrum. You will want to copy that array into the celestial observation you are actually interested in. This can be done with ndfcopy, provided the sky spectrum does have a Specdre Extension and does not have a SPECVALS component in it. That status can be achieved by two editext commands. Finally you may want to re-sample each spectrum (row in a cube) so that all use the same linear grid of spectroscopic values. To that end you use resamp mode=cube. Here is the whole procedure. (The commands are not complete; parameters are missing. Also there may be no Th-Ar line lists available for photon energies in the MeV range.)

ICL> arcgendb thar.arc thar_arc
ICL> arclocat dialog=n arc1
ICL> arclocat dialog=g arc1
ICL> arcident arc1 arc2 thar_arc
ICL> arcdisp arc2
ICL> setlabel arc2.more.specdre.specvals "particle energy"
ICL> setunits arc2.more.specdre.specvals "log10(10**9*eV)"
ICL> editext "create" sky1
ICL> editext "delete specvals" sky1
ICL> ndfcopy arc2.more.specdre.specvals sky1.more.specdre.specvals
ICL> resamp cube sky1 sky2

#### B.8 References

• Bailey J.A., Chipperfield A.J., 1991, ICL – The interactive command language for ADAM, Starlink Guide 5
• Currie M.J., 1992, KAPPA – Kernel application package, Starlink User Note 95
• Currie M.J., 1994, HDSTRACE – Listing HDS data files, Starlink User Note 102
• Eaton N., 1995, AGI – Application Graphics Interface – A subroutine library for accessing the graphics database, Starlink User Note 48
• Harrison P., Rees P., Draper P., 1994, PONGO – A set of applications for interactive data plotting, Starlink User Note 137
• Meyerdierks H., 1992, Fitting resampled spectra, in P.J. Grosbøl, R.C.E. de Ruijsscher (eds), 4th ESO/ST-ECF Data Analysis Workshop, Garching, 13 – 14 May 1992, ESO Conference and Workshop Proceedings No. 41, Garching bei München
• Mills D., 1992, Automatic ARC wavelength calibration, in P.J. Grosbøl, R.C.E. de Ruijsscher (eds), 4th ESO/ST-ECF Data Analysis Workshop, Garching, 13 – 14 May 1992, ESO Conference and Workshop Proceedings No. 41, Garching bei München
• Shortridge K., 1991, FIGARO – General data reduction and analysis, Starlink Miscellaneous User Document
• Terrett D.L., 1991, GKS – Graphical Kernel System, Starlink User Note 83
• Terrett D.L., 1993, PSMERGE – Encapsulated PostScript handling utility, Starlink User Note 164
• Terrett D.L., 1995, PGPLOT – Graphics subroutine library, Starlink User Note 15
• Wallace P.T., Terrett D.L., 1995, SGS – Simple Graphics System, Starlink User Note 85
• Warren-Smith R.F., 1995, NDF – Routines for accessing the Extensible N-Dimensional Data Format, Starlink User Note 33
• Warren-Smith R.F., 1995, HDS – Hierarchical Data System, Starlink User Note 92