orac009-ufts Version: 01 Original: 27November1997 Modified: 2 December 1997
This document provides a brief summary of the data reduction scripts that will be provided for testing during the commissioning of UFTI.
The first UDR data reduction scripts to be produced by the Orac project are those that will be needed to reduce data at the telescope during the commissioning of UFTI in March 1998. There are of course very many different methods of obtaining and reducing even simple point source photometry, and indeed even for data that has been obtained using the same method, the reduction algorithm may be determined by personal preferences. Obviously in the long term the use of scripts for UDR builds in the flexibility for many different reduction recipes to be developed. However for the purposes of developing and commissioning UDR it is necessary to agree on a small number of standard procedures and scripts that will be used in the first instance. UFTI will be commissioned while the full capabilities of UDR and the observatory control system are still being developed and so careful prioritisation of effort for script writing is essential, or there is a danger of Michelle having inadequate reduction facilities and the full automation being incomplete.
In section 2 the degree of automation of the data reduction scripts during UFTI commissioning is described, and in section 3 very brief specifications of the scripts that will be produced initially are presented.
The full automatic functionality of UDR is dependant on the new UKIRT Observatory Control System in a number of ways, e.g.:
The Observatory Control System is to be delivered and commissioned with Michelle and so will not be available when UFTI first goes on the telescope.
Initially the UFTI data reduction recipes will be run in a semi-automated mode i.e. the user will start the script by manually typing in a command, and after that the script will reduce the data by following the recipe without user intervention. For example a user who wants point source observations using a std_phot_fnt Exec with an x point jitter mosaic, may have to reduce this data by issuing a data reduction command such as reduce_std_phot_fnt x nn, where x is the number of data frames making up the jitter pattern, nn is the number of the first observation and any other necessary parameters are on the command line. Such data reduction commands would be issued at any time after the Exec has been started. An alternative is that some limited automation could be "fudged" for UFTI by asking the user to enter a UDR recipe name as part of the object name or in the Exec somehow. Whether or not this can be achieved in a sufficiently robust manner as to make it worthwhile is tbd.
The full system is being developed to handle error conditions such as observing sequences finishing prematurely in a clean manner with no or little user intervention. This again depends on the relationship between UDR and the OCS, which will not be in place during UFTI commissioning. However it is a requirement for UDR that the user can monitor the reduction process and interrupt at any time. It is therefore expected that initially the user will use this manual method for stopping and starting data reduction scripts when observations are aborted or stopped. (This is what currently has to happen for IRCAMDR and CGS4DR and so it is not a backwards step....).
In summary, the main differences between the initial UFTI version of UDR and the fully functional one are that the user may have to start and stop reduction recipes by hand, whereas eventually they will be started up automatically as the data is written to disk, and that only a limited number of recipes will be supported.
A number of email discussions, on the UKIRT DR circulation list, outlined 6 -10 different methods for obtaining flat fields to reduce photometry of point sources, some of which are better in some circumstances, or preferred by some users to others. It was agreed however that these could be reduced to a few standard procedures that would cover most normal observing and the development of the more sophisticated algorithms and special cases would be done at a later date. In addition reduction scripts will be needed for some types of calibration runs and to reduce maps of extended sources. These scripts are enumerated and prioritised in the following sections.
These scripts are those that are deemed essential to examine and reduce basic UFTI data during commissioning and in the first few months of operation.
There are ~ 10 recipes described in total below - probably about as many as can reasonably be expected to have scripts provided for in 3 months.
As for IRCAM and CGS4 it is assumed that a short Exec to check the read noise and dark current on the array will be run at the beginning of the night. A UDR script to reduce this data and report to the screen and a file the values for dark current and read noise derived from the observations will be needed. In the first instance the data reduction recipe (and the scripts produced for it) will assume that exactly the same array_tests Exec will be used for UFTI as is currently the case for IRCAM3, and it will calculate the read noise and dark current by following exactly the method used by the array_tests procedure in IRCAM3DR.
In practice during commissioning it may be found that the exposure times on the DARK observations will need to be changed and the UDR script will be flexible enough to allow the corresponding changes to the arithmetic. It is reasonable to assume however that the general method (observe a BIAS and a few DARKS of pre-determined standard exposure times) will not change.
UKIRT staff have concluded that the best way to ensure that the telescope is in focus when observing with UFTI is to do focus runs in the infrared. i.e. the telescope secondary mirror position will be stepped through a series of positions while observing a moderately bright star and an UFTI data frame taken and stored for each mirror position. These frames will then be analysed by a UDR script to determine the position of best focus. The details of the Exec to take this data has yet to be specified, but tests with IRCAM3 will determine the method - how many focus steps, size of focus step, how bright a star, etc. Tests of whether it is better to use measurements of FWHM of the source or of Strehl to determine the position of best focus will also be carried out. Once these have been solved a specification for the reduction recipe and script will be written.
UDR will initially provide one data reduction recipe and the corresponding scripts to reduce the data from the single standardised focus procedure that is expected to result from the tests. The output of the script will be a table of observation number against measure of image quality, unless the acquisition is able to write the focus position into the header, in which case it will be of focus position vs. image quality.
As noted above, there are many ways of obtaining and reducing such data. The main problem is that there is no good flat field source for the near infrared cameras at UKIRT other than the sky. It is of course more efficient if blank sky does not have to be observed separately, and very little sky is truly blank in any case, so various jittering and median filtering and source rejection algorithms have been developed. UKIRT will have a small number of photometry Execs for different brightnesses of source which will be recommended to user and for which UDR will provide the scripts for standard data reduction methods. Brief descriptions of the observing sequence and reduction method are given here. The names are the short-hand ones that have been used in the UDR distribution list email discussions. Full specifications will be the subject of a different document - note however that aside from the method of flat generation a specification for many of the "frame combining" steps is given in the discussion of "Scenarios" in Orac002-sen. Also note that in each of these the individual reduced frames also need to be available so that they can be used to check the photometry.
This is the same as 3.1.3, except that the reduction of the data is carried to the next stage. The first three recipes described in 3.1.3 will have an extended version that goes on to derive automatically the magnitude of the source. Users will start the photometry Execs after having positioned the source at a standard position on the array which the UDR scripts will assume (it is possible to search whole frames for sources, but that means you might get photometry of the wrong one if there happen to be two sources on the frame). There will be a simple method of informing UDR if this standard position is changed. A profile fitting algorithm will find the star and derive an instrumental magnitude for it. This will be converted into a preliminary reduced magnitude using a standard zero point and airmass correction. The instrumental and preliminary magnitudes so derived will be logged to a file so that the user can monitor photometric quality more easily than at present.
For mapping extended sources and photometry of crowded fields, separate observations of an area of "blank" sky are needed to form a flat field. A general rule of thumb is that for faint structures at least as much S/N is needed in the flat as is obtained for the source frames - thus the Exec for obtaining a flat will do 5-9 jittered background limited exposures on the SKY. The SKY observations may be taken several times a night depending on the flat stability and accuracy required. The sky observations are median filtered using a "sigma clipping" method and the result normalised to form the flat.
A mosaic of the source is obtained with offsets such there is a significant overlap (~ 20%) between the frames. Given the 1.5 arcmin field of view of UFTI mosaics of more than 3x3 cannot be made by offsetting the crosshead while autoguiding, so there may need to be two versions of the mosaic reduction depending on how the offsetting is done. As each frame is taken it is dark subtracted, flat fielded and displayed. When the next frame is available it is aligned with the first either by using the offsets in the header and the known array rotation and plate scale, or by identifying and aligning sources in the overlap regions. Ideally scripts for both these recipes will be available, but if time is short the former will be the first priority. (because even if this means there is smearing of sources in the overlap region it will at least enable the user to see if there are extended structures that are continuous across the frames and hence whether the map extent is reasonable). Data in the overlap region will be used to scale photometrically the images before combining. This effectively determines a mean sky level for the set and the value will be reported to the user so that they may optionally subtract it from the final frame. The combination of frames will be done when all the frames in the mosaic have been obtained to prevent the creation of artificial gradients.
At the end of the night a summary file of the objects observed, time, filter etc. needs to be produced.
Some of these low priority scripts are simply wraps for facilities that are readily available in standard data reduction packages, or they are for observing sequences that are less common. They are listed here in priority order. It is not expected that very many (if any) will be written by March 1998 when UFTI is commissioned, but if effort becomes available this list can be worked through.
For point source photometry that has been reduced using the recipe described in 3.1.4 to generate a photometry summary file, it is very desirable to have a script that can read in the instrumental magnitudes, identify the photometric standards, calculate zero-points and extinction curves for the night and calibrate the photometry with the result. Such a script would normally be run at the end of the night, or after several standards had been observed. It would improve the observers ability to monitor photometry during the night, and also to quickly achieve fully reduced photometry (essential if there are 100 stars observed in a night), but is not essential for many UFTI programmes, many of which will be more concerned with image structure than with photometric magnitudes.
This is in the "nice to have" but available by ordinary commands in data reduction software category. It would be very useful to have a script that plotted histograms of data or errors and allowed you to select bad pixels by clicking on regions of the histogram with the mouse. Also a combine bad pixel masks that would do a logical OR would be useful. This is not a script that would be used in on-line data reduction, but the ability to run a script that simplifies bad pixel identification, putting masks into the right directories etc. would be useful to run in parallel with observing. Its likely that there will be a lot of bad pixel mask creation in the first few nights of commissioning!
Again this can be done easily from any favourite data reduction package. Non-the-less a script that reproduced the functionality of the IRCAMDR "disp" command would be handy - i.e. a script that reduces the subtraction of two images and plotting of the result to just one command with only the observation numbers as input would be very helpful. "nsigma" type auto-scaling is fine for this - the purpose is just to subtract to images and look to see if there is a source there. Something like diffplot nn mm saves the user from having to worry about remembering commands or file names.
The scripts that will be produced as first priority assume that the whole array will be used. It would be nice to have scripts that reproduced the recipes summarised in 3.1.4 and 3.1.5 for point sources which could be used with sub-array readouts. The shapes and forms of these sub-arrays are of course tbd.
Many UFTI programmes are likely to be about high resolution imaging to look for structure around fairly bright point sources. The full analysis of such data is a complex subject, and is not well suited to automation. All these observing runs will however be dependant on frequent monitoring of the PSF by observing standard stars. It would be useful therefore to have a script that would find the star (in the standard box) calculate the Strehl and FWHM and plot a profile, perhaps also putting the information into a file so that the user can easily see changes and know when to give up or start looking at the star more frequently. These facilities are of course all in the usual data reduction packages, so this is another case where simply wrapping them up into a single command with an observation number as parameter would be nice.
The goal of this is again "quick" result monitoring rather than careful reduction. It is desirable to have a script that when asked to subtract images in two different filters will correct for the difference in plate scale, identify and match point sources and then subtract. In time a much more sophisticated method than this will be needed, but this might be useful first pass.
This is the generalised form of the "quadrant jitter" described in 3.1.3, in which the source is moved to n positions on the array and the flat is determined for n-1 corresponding segments. It seems like a useful technique for general photometry reduction but is as yet untried. Probably the script is needed before it can be tested, but this is unlikely to happen during UFTI commissioning - when other tests will probably be higher priority.