SOME NOTES ON OBSERVING WITH SuperAAPS

SuperAAPS is a DramaTCL GUI that links together several of the AAT’s systems which are used for Planet Search observing. It aims to combine them together into an integrated observing system that improves the efficiency, reliability and quality of AAPS data. The systems that SuperAAPS makes use of are

1. •The ODC (ie. Detector Controller),
2. •PTCS (Portable Telescope Control System),
3. •Guider,
4. •Exposure Meter,and
5. •UCLES (via its underlying task known as UMIT)

However, they were not exactly designed with automated interaction in mind. Interoperability has improved in the last few years, and saaps now works reliably about 95% of the time.

saaps is software written by the AAPS team for use by us. It is not supported by the AAO. So first read the 'frequent problems' sections below to try and fix the problem. If it can't be fixed, you need to report it to me - there is no point reporting it to AAO staff, as they can't debug it.

To contact me at night, you should email me with details, then text me at 0416 092 117 to say there's a problem and you've sent an email, and then if I don't respond call me on that number.

To report a problem, send me as much information as you can (with times, detailed descriptions of what you were doing at the time, copies of the text in the SAAPS message window, and (if a TCL error has triggered a pop-up) a copy of the "stack trace" text) so I can track down the remaining instabilities in the system.

If I don't find out what's going wrong, I can't debug the issue, and I can't fix it.

Chris T.

07/11/2014 .

WHAT'S NEW

November 2014

✦On the Oct 2014 run it was observed that UCLES could not be precisely positioned (i.e. to within less than a pixel) to match earlier configurations. This was revealed by a large difference in the echellogram plate scale in the cross-dispersion direction (~7.5% change). This was resolved as being due to the cross-dispersing prisms not correctly positioning, and resolved by replacing a controller card in the relevant mechanisms electronics.

Please watch out for the inability to match the current observing ThAr to the archival ones! If you see this effect, report it to AAO staff and try to work out what is going on.

August 2011

✦The Apogee CCD in the exposure meter has been replaced, and the meter has been recommissioned and is back in operation.
✦CYCLOPS or UCLES - The SuperAAPS system can now run both regular AAPS observing with the UCLES slit, and CYCLOPS observing with the fibre bundle. saaps will try to work out whether you are using CYCLOPS of regular slit mode from the drama version you have started. If it gets this wrong, you can change the system being used from the UCLES_CYCLOPS menu in saaps.
✦Observing Run Codes - Please use run code 'w' from now on for regular AAPS observing, run code 'c' for CYCLOPS observing, run code 's' for sub-giant planet search observing (Rob Wittenmyers' program) and any other letter (except for 'u') for other programs.
✦Tracking Tool - The tracking tool 'Configure Runs to Track' option now sorts the runs by their run code (in reverse order) so 'w' runs appear the top and 'c' runs appear at the bottom, making it easier to select recent runs for each program
✦Target Catalogues - The loading of your own target catalogues now works, as does the use of that catalogue with the tracking tool.
✦Beware of Pattern Noise - The CCD Controllers developed a pattern noise problem during the Monster Run in 2009. This is intermittent, and causes excess noise to appear in each image - something that the Doppler analysis code picks up. So check for excess noise in the overscan region have been implemented in the usual post-processing - if the noise is within nominal limits, you'll get a green report on the post-processing window.If its excessive, the report is red, and a note appears in the observing log. When this happens you need to get technical help to (1) reset the controllers, or (2) have the cards from the controllers removed and jiggled - it seems to be some sort of annoying intermittent connection bug.
✦A bug appeared in which (on 5 times on one night) the EXPOSED keyword in the FITS files (which is supposed to give us the exposure time) was set to 0. I have implemented a check for this in the post-processing script, and if the EXPOSED keyword is different from UTEND-UTSTART time by more than a few seconds, then UTEND-UTSTART is used to give the exposure time instead. In any case, the exposure meter mid-time is still correct.
✦The EXPOSURE_METER continues to have a subtle bug we can't identify that causes the EXPOSURE METER to freeze during an exposure, and the saaps tool to then stop in its tracks after the exposure has read out. That exposure will be fine, but it will NOT have a mid-time correctly calculated in it. So you will need to shut everything down and start again. Changes to the underlying code seem to have reduced (but not eliminated) the incidence of this bug.
✦The GUIDER has been much more robust in its automatic acquisition in the last year or so - nonetheless, it is not perfect, so glancing at it when a new target is slewed to is always worthwhile!

At present it looks like this bug is due to a cumulative "memory leak" style problem, so that it just happens after a certain number of exposure meter exposures. It is being examined.

HARDWARE AT THE START OF A RUN

Pre-slit area hardware - at the beginning of an AAPS run, AAT technical staff will begining the set-up for all the 'standard' AAT equipment we use. However, you should check that the additional hardware we use in the pre-slit area of UCLES is all connected abd powered up. These are

1.Power up rotating pick-off mirror power supply and set to 1.6V. The rotating pick off is powered by a dark coloured variable power supply. It needs to be plugged in and set to deliver 1.6V to the rotating mirror, which sends 4% of the light exiting the slit o a CCD camera for monitoring by the exposure meter. (NB: Not needed while exposure meter is not operational).
2.Power up Exposure Meter CCD by connecting its plug-pack. The exposure meter uses an Apogee CCD camera to collect light picked off from the beam exiting the slit. It doesn't have a power switch, and so is powered up by connecting its supply. (NB: Not needed while exposure meter is not operational).
3.Connect up Iodine Cell temperature controller - make sure the temperature controller is plugged in. In general you'll want the cell to be cold when doing the set-up, so don't power it up just yet.

AAPS Set-up : the AAPS setup has 6 main steps ...

A. Start the UCLES ODC software, and then start saaps
B. Roughly re-position echelle ET,EG, SA to that used on the last AAPS run.
C. Check the detector rotation, and adjust if required.
D. Precisely position ET,EG, SA
E. Check the instrument focus, such that you can get a best focus in the range CF=??-??
- if you can't then re-shim the detector
- if you can, then move on.
F. Precisely position ET,EG, SA again.
Note that this process is only needed at the beginning of a run.
Once the setup is done you should try to never move the echelles for the rest of the run, except for very small tweaks, if your nightly check of the ET,EG shows that a small tweak is required.
In particular, if (after having done the setup) UCLES ever says it wants to move the echelle, then always say "No"!

A. Start up UCLES ODC and saaps

1. Connect to aatlxx (if not already connected) from the triple-headed Linux desktop (aatxdb) being used as an X-display and log in (username and passwords on the wall)
2. Start drama. Logging into aatlxx will automatically start drama. You may need to use a 'special' version (if so instructed by AAT staff or so noted on the white-board). For example as at August 2011, we are using ucles_cyclops_latest, so enter this command to start the new drama

   drama -v ucles_cyclops_latest

otherwise, just enter

drama

The following is not necessary any more

(Then enter
   setenv EM_SAVE_IMAGE 0
   setenv EM_CAL_FILE "/instsoft/instusers/aatinst/saaps/spectral.cal"

to save time by not having the exposure meter write out every image it takes, and to get the right spectral calibration file.)
3. The following is not necessary any more.

Up until recently it has been necessary to use a 'private' version of the exposure meter task, that fixed a variety of flaws in the drama-standard version. To make this happen use the following commands (easiest to cut and paste from here)

setenv EXPOSUREMETER_DIR /home/cgt/acmm/ExposureMeter-2-20
setenv EXPOSUREMETER_LIB /home/cgt/acmm/ExposureMeter-2-20
setenv EXPOSUREMETER_DEV /home/cgt/acmm/ExposureMeter-2-20

Hopefully the need to do this will have been removed by the next run.
4.Start UCLES with an ucles1 or ucles2 command (depending which controller is in use - see the white board) and the following extra options which will ensure you can talk to the exposure meter and guider

   > cd ~
   > ucles2 guider em

If the exposuremeter is not operational for any reason you need to remove the "em"

   > cd ~
   > ucles2 guider

Similarly, if the GUIDER is not operational from the telescope console and you want to run without it (say during the day to do calibrations) use

   > cd ~
   > ucles2 em
5.After entering the ucles1/2 command, the System Loader window will appear and all the tasks needed to run UCLES for SuperAAPS will commence, with the observing GUI, and Exposure Meter window and a Skycat window appearing last. Note that the cd ~ is important. If you try to start ucles from anywhere except the home directory, it will crash with a really unhelpful error message.

When the exposure meter first starts up after a power-up, it will need to cool-down to its operating temperature of -15C from room temperature. This can take up to half-an-hour.

It might be useful to know what the options to the ucles1 or ucles2 commands are
•em - start the exposure meter task
•guider - talk to the guider task on the new AAT TCS
•ptcsSim - start a PTCS simulator. This option means you can do pretend observing with a pretend telescope
•To do observing or tests with a pretend guider, you need to start a special 'guider simulator' before you enter the ucles1/ucles2 commands above, as follows

> /instsoft/instusers/aatinst/guisim & (NB: this didn;t work in Oct 2014)
6.Start a GAIA window for examining images as they are taken (it offers additional functionality for measuring seeing and SNR over the Skycat window) and a ptcs window so you can see where the telescope is.

> gaia &
> tel
7. Start SuperAAPS

> cd saaps ; saaps &

This will start the SuperAAPS GUI. The first thing you'll see is a pop-up informing you that the "Guider will attempt to acquire objects from now on". Just Click OK, and the GUI will appear.
8.If this is the second or subsequent night of observing, use the Commands->Reconfigure menu pull down in the System Loader to advance the first run number past the last run number from last night.
9. Then 'Initialise' a new night to start a new logsheet in saaps with 'Commands->Initialise New Night'. You will get a pop-up saying "Are you sure you want to fnalise and archive this log?" Make sure the name of the log-file is correct for the last nights observing and say yes. If you are unsure say no and check the files out in ~/saaps/final_logs/.

After you say 'Finalise' the old log will be changed to be read-only, so it can't be accidentally deleted in the future.

Make sure you then update the 'UT Date' entry correctly in the log header for the next night - you will not be able to write a log file until that entry is a legitimate one.

B. Roughly re-position echelle.

1. Make sure the 31l/mm echelle is in use. If it isn't then do a manual configure in the UCLES ODC, requesting the 31l/mm echelle, with a central wavelength of 6050A with the "peak" option set to no, and SLAUTO set to "no".
2. Look for the most recent AAPS run in the logsheet directory, and examine it.

        cd ~/saaps/final_logs
        ls -altr w*.log
        more w36.140517.log

    which will give some text like the following. Use the ET,EG,SA from these and manually enter it into the UCLES ODC to get the rough positioning right.

            UCLES Spectrograph Observing Log - CYCLOPS2 HATS
_____________________________________________________________________
Observer: RPB                            Tape: w36     Telescope: AAT
UT Date: 17/May/2014                   Chip: EEV2     Gain=1.3 e/adu
Windowing: Cols:4096   Rows: 2048 Binning: 2x1    AAPS
ET: -0.00010 Offset: +0.00000 EG: -1.02030 Offset: +0.00000
Slit: 15.82 pix (3.97 arcsec) x 87.02 pix (14.24 arcsec)
Slit Angle: -6.25    Slit Angle Offset: +0.00
Col Focus: -4.98 (fwhm=1.5 with 0.5pix slit                         )
I2 cell temp: 60.0    Shims: Unknown
---------------------------------------------------------------------
Run    Object       I2    UT   m/p Exp S/N         Comments
---------------------------------------------------------------------
900   ThAr0.5px    y 03:32:30 m   30     2
901   Wideflat     n 03:37:35 m 120   360
902   Wideflat     n 03:40:42 m 120   362
903   Wideflat     n 03:43:49 m 120   362
904   Wideflat     n 03:46:56 m 120   362
905   Wideflat     n 03:50:03 m 120   362
906   Wideflat     n 03:53:11 m 120   362
907   Wideflat     n 03:56:18 m 120   362

C. Check the detector rotation, and adjust if required

There are two ways to do this. In either case, keep repeating the process until you have the rotation done.
1. I have written a perl script that can do do position and rotation adjusments between a master image and a new image. It identifies lines in those images, and then does a similar triangles match. It has fairly tolerant of the images being up to ~hundred pixels out of alignment, and it also fits for a detector rotation.
Take a set-up ThAR in saaps. Note the location of the arc file just taken. Then on aatlxx terminal ...
% cd ~/saaps
% AlignUCLESAAPS -d AlignUclesAAPSMaster.fits NewSetUpArc.fits
This produces a lot of information, and in particular includes
Transformation rms : 0.011 ( 63)
Transformation coeffs : a -0.460006905 b 0.924396838 c -0.000288152 d -0.254891875 e -0.019440143 f 0.999764761
Transformation scales : s_x 0.924396882911217 s_y 0.999953747178962
Transformation offsets : x -0.460006905 y -0.254891875
Transformation rotation: 0.00893008967895595
Align by
Rotating detector : 0.00893008967895595 degrees (for binned pixels with UCLES slit)
X pixel movement : -0.460006905 pixels
Y pixel movement : -0.254891875 pixels

Align by ....
Suggested Rotation of detector : +0.01 degrees - NO MOVE NEEDED
Equivalent micrometer change = +0.02 mm

X direction movement (=EG=perp to orders) : -0.46 pixels
Y direction movement (=ET=along orders) : -0.25 pixels
EG offset slave->master from file headers was : +0.16236
ET offset slave->master from file headers was : -0.00592
Suggested EG movement : -0.0005 - NO MOVE REQUIRED! (< 0.9 pix)
Suggested ET movement : +0.0002 - NO MOVE REQUIRED! (< 0.9 pix)
Slit Rotation measured to be +28.04 (deg)
Adjust slit angle by -28.04 degrees.
The top block has information on the co-ordinate transformation derived. If the rms about the transformation is small (<0.1 pix) and is based on a reasonable number of arc lines, you can trust the results. (You can also check the plate scales s_x and s_y make sense).
The "Align by ... Suggested Rotation" block tells you how to adjust the detector rotation.
The "Suggested EG/ET Movement" blocks tell you how to adjust echellogram position"
2. OR you can take a flat field, then adjust the detector rotation until a selected order is vertical on the detector. This is fairly fiddly, but it will always work. If you can't get the above to work for you then this is the fall-back procedure.
Take a single flat with saaps. Use the image display to check that order 105 (located at pixel value X=705, on the EEV) is aligned to be vertical. Note however that there is about 0.3 pixels in curvature of the order itself (ie. centre of the order is at ~0.3 pixels lower X value than the top and bottom, even when the order is perfectly vertical).
If this order is not "straight" (ie top and bottom has the same X-value to within ~0.2 pixels, noting that the centre will always be about 0.3 pixels lower in X) then get the day or afternoon technician to change the chip rotation (until the top and bottom of the order nearest column 105 for EEV are aligned).
The moves required are
Decreasing the micrometer by 1.00mm (ie rotating the EEV dewar anti-clockwise when viewed from behind) caused the top trace to DECREASE by 9 pixels relative to the bottom trace.
Decreasing the micrometer by 2.00mm (ie rotating the EEV dewar anti-clockwise when viewed from behind) caused the top trace to DECREASE by 24 pixels relative to the bottom trace.
Increasing the micrometer by 0.20mm (ie rotating the EEV dewar clockwise when viewed from behind) caused the top trace to INCREASE by 2 pixels relative to the bottom trace.
Alternatively, the magnitude of the micrometer change can be calculated by trigonometry:
delta micrometer = 130 * tan (theta) mm,
where theta is the angle by which the orders lean (noting the detector is usually binned by 2 in x).
The sign of the change is given by this simple rule:
If orders
lean LEFT INCREASE micrometer setting
lean RIGHT DECREASE micrometer setting
You should be able to get orders straight to within 0.2 pixel (which is limited by ability to make small changes to micrometer setting).

D. Precisely Position Echelle.

There are two ways to do this. In either case, keep repeating the process until you have the rotation done.
1. Take a Setup ThAr in saaps. Then use AlignUCLES AAPS (as above).
Make the ET,EG,SA changes as recommended.
The script also produces a pop-up of the 'slope' of a selected line to estimate the slit angle. The script makes suggestion on the slope of that line so as to make it flat. But be careful and look at the plot to make sure the change makes sense - the estimation is not perfect. But in general apply the suggested SA changes as well.
Repeat until the script says no more changes are required.
2. Or take a Setup ThAr in saaps. Paul's FOC routine in IDL on the machine that IDL runs. This script is pretty picky about how close the positioning needs to be - you need to have the position right to within about 30 pixels.
% cd ~rpb/focus
% idl
% IDL> device,pseudo=8 ; MIGHT help with colors.
% IDL> foc,/plt,inpfile='/data_lxx/aatobs/OptDet_data/141030/ccd_3/30oct20001.fits',/mark
Then apply the ET,EG changes recommended.
Repeat until the script says no more changes are required.
3. If either of the above is not working then, you may need to fall back on a more "old fashioned" approach. This PDF (NOTES Pictures 2014.1.pdf) shows the AAPS setup graphically - you can manually move the echellogram to get the "Magic Line" to its location with the following guidelines.
Adjust (it is simplest to only adjust one parameter at a time) :
Ech_Theta (ET) to move in direction of dispersion
Ech_Gamma (EG) to move in perpendicular to dispersion
       *** EG + changes move orders/lines to decreased X value on CCD
            (ie perpendicular to orders)
            Changes of 0.0011 in EG typically cause ~1 pixel movement on EEV2
       *** ET + changes move orders/lines to increased Y pixels on EEV2
            (ie. moves along order)
            Changes of 0.0007 in ET typically cause ~1 pixel movement on EEV2
So for example,
        In order to:                                 You Should:
        ------------------------------                ----------------
        Decrease X pos. of line by 10        EG --> EG + 0.011
        Increase X pos. of line by 10        EG --> EG - 0.011
        Decrease Y pos. of line by 10        ET --> ET - 0.007
        Increase Y pos. of line by 10        ET --> ET + 0.007

E. Check the instrument focus.

At the moment, the best way to estimate FWHM in an image is with Paul's IDL FOC script. So focussing involves taking exposures at CF values from -10 to +5 (with the I2 cell installed and cold), and then making sure the best focus falls in the range -2.0 to +5.0. (Best focus without the I2 cell in place is 8.0mm less than with it in place. So if you want to be able to take templates without the cell in place, you'll need to be able to reach best focus. Hence we aim to be focussed in the range -2.0 to +5.0).
The easiest way to get the exposures done is to take a SetUp ThAr, then once that's done execute the AAPS_Focus.tcl sequence in the ODC, which will take arcs at CF=-10.0 to +5.0 in steps of 2.0mm. Go and have a coffee while its doing that.
Once in hand, run the FOCU routine on each of those images, noting the FWHM and CF values, then do the minimisation however you feel is best. (I like to make a little graph in by log-book, but I'm old-fashioned) to determine the best focus.
With the 0.5pix slit for the Setup ThAr you should get a best focus in the range 1.45-1.55 pix.
If you can't reach best focus, you may need to shim the detector. Some notes on that follow:
SHIMMING: Fine adjustment of the spacing between the UCLES detector and the camera focal plane is achieved by inserting shims between the detector dewar and its mount on the UCLES camera. Due to seasonal temperature temperature variations (and variations in the amount of force used in screwing down the dewar), these shims may need to be changed from run to run to permit a good focus to be obtained within UCLES' allowed collimator motion(CF=[-10,+5]. Or more specifically with the collimator in the rnage -2.0 to +5.0 with the iodine cell in place).
IF YOU NEED TO CHANGE SHIMS
•If the best focus is off the negative end of the collimator travel (<-10) you need to make the shims thinner (ie move the detector towards the camera).
•If the best focus is off the positive end of the collimator travel (>+5) you need to make the shims thicker or insert more shims (ie move the detector away from the camera).
•The magnitude of shimming required is 0.5 thousands of an inch ("half a thou") per millimetre of collimator travel.
•If adjusting the shimming, try to get the best focus in the range -2.5 to 5, so that you can also get a good focus WITHOUT the cell in place, in case you need to take templates (usually best focus without the cell in place is about 8.0mm less than the best CF with the cell in place).
SHIM TABLE
Colour            Inch             mm               microns
=============================================================
Silver            0.0005           0.0127           12.7
Amber             0.001            0.0254           25.4
Purple            0.0015           0.0381           38.1
Red               0.002            0.0508           50.8
Green             0.003            0.0762           76.2
Tan               0.004            0.1016           101.6
Blue              0.005            0.1270           127
Clear/Transmatte 0.0075           0.1905           190.5
Brown             0.01             0.2540           254
Black             0.0125           0.3175           317.5
Pink              0.015            0.3810           381
Yellow            0.02             0.5080           508
Solid White       0.025            0.6350           635
Coral             0.03             0.7620           762
=============================================================

F. Final Precise Echelle Position.

Once you have got the detector rotated correctly, the detector shimmed so you can reach best focus, the best focus set,and the slit angle correct ... then its time to do a final reposition using the same procedure as step D above.

AAPS Observing

Once you've done the setup, then you should make sure to turn on the iodine cell and set it to run at 60C.

Then every day you should aim to take

Afternoon

At least 5 and preferably 20 Wideflats

One Setup ThAr

Several narrow flats

Several iodine flats

Nightime

Fill the dewar at sunset

Take some iodines

Start observing targets and away you go

At end of the night take an iodine and ThAr before filling the dewar.

Morning

Fill the dewar, take a ThAr and iodine. Then setup the queue to do at least 20 Wideflats.

Go to sleep.

Some Further notes on TCS Issues, Daytime Staff, and the End of the Night

It may be that AAT staff will want to do testing on the TCS during the day that will require killing and re-starting the TCS and/or guider. This will cause the UCLES ODC to stop taking data (eg. flats) because it needs to talk to a TCS task to fill in file headers etc. To stop this becoming a problem, I often shut down saaps, and start it back up in simulation mode, before I start the 40-60 wideflats we take at the end of the night.

✦Shut down saaps (File->Close), and then shut down the ODC (File->Exit - making sure to NEVER park the echelles - then when prompted cleanup 'Local Nodes' at the pop-up).
✦Restart ODC as above, but with the following commands to start a fake TCS and a fake guider

cd ~

/instsoft/instusers/aatinst/guisim &

ucles2 guider em ptcsSim

cd saaps

saaps &

You can then happily take flats regardless of what happens to the real TCS. Of course, the following afternoon (usually after dinner), you will need to shutdown saaps & ODC again, then restart ODC and saaps as per usual.

Start up UCLES : the SuperAAPS observing process

1. Shutdown SuperAAPS. Make sure there are no queue observations running, and use the "File->Exit" menu or Ctrl-Q in the SuperAAPS window.
2. Shutdown UCLES from the System Loader. You must shut-down UCLES from the "AAO CCD Loader" window using the "File->Exit" menu pull down. Shutting down from the observing GUI itself doesn't work.

You will be asked "Do you really want to do this?" - say yes.

You may then be asked "Do you want to park the echelles?" - say "NO". This is very important. If you accidentally say yes, you'll have to re-do the fine positioning of the AAPS set-up again. This pop-up is a recent addition, and we don't like it, and are trying to get rid of it. Until we do, "Just Say NO to Echelle Parking"

Then you'll be asked do you want to do a clean up. Select "Local Nodes" and do the cleanup.

Everything should then clean-up.
3.Emergency Shutdown. If the CCD Loader window is unresponsive, or the clean-up won't work, you can use a cleanup command in the same terminal window you started ucles from. But this is not the preferred mode of shutdown.
4.Warning - After you've exited SuperAAPS, you'll be sitting in ~/saaps. If you try to restart UCLES from this directory, you'll get an error message like "Can't talk to aatvme15" which arises because you're starting UCLES not in the home directory. So do the cd ~ and try again.

Start up UCLES : the "old" observing process

For the record, here's how do to start up and shutdown when not using SuperAAPS.
1. Connect to aatlxx from the triple-headed Linux desktop (aatxdb) being used as an X-display and log in (username and passwords on the wall)
2. This will automatically log you into drama. If you have been told you need to need use a 'special' version (if so instructed by AAT staff or so noted on the white-board) then enter (eg.)

> drama -v ucles_cyclops_latest
3. Start UCLES with an ucles1 or ucles 2 command (depending which controller is in use - see the white board)

> ucles2

after which the System Loader window will appear and all the tasks needed to run UCLES will commence, with the observing GUI and a Skycat window appearing last.

NB: If this is the second or subsequent night of observing, use the Commands->Reconfigure menu pull down in the System Loader to advance the first run number past the last run number from last night.
4.Start a GAIA window for exmaing images as they are taken (it offers additional functionality for measuring seeing and SNR over the Skycat window)

> gaia &
5.Start observing from the GUI

Shutdown UCLES : the "old" observing process

1. Shutdown UCLES from the System Loader. You must shut-down UCLES from the "AAO CCD Loader" window using the "File->Exit" menu pull down. Shutting down from the observing GUI itself doesn't work.

You will be asked "Do you really want to do this?" - say yes.

You may then be asked "Do you want to park the echelles?" - say "NO". This is very important. If you accidentally say yes, you'll have to re-do the fine positioning of the AAPS set-up again. This pop-up is a recent addition, and we don't like it, and are trying to get rid of it. Until we do, "Just Say NO to Echelle Parking"

Then you'll be asked do you want to do a clean up. Select "Local Nodes" and do the cleanup.

Everything should then clean-up.
2.Emergency Shutdown. If the CCD Loader window is unresponsive, or the clean-up won't work, you can use a cleanup command in the same terminal window you started ucles from. But this is not the preferred mode of shutdown.

Observing with SuperAAPS

OBSERVING WITH SuperAAPS

Overview

SuperAAPS has 10 major components in three major windows. Sitting in front of the triple headed monitor you might see something like this with the AAO CCD Loader on the far left, then the ODC GUI, then SuperAAPS, then the SuperAAPS exposure post-processing results, then GAIA.

In the main SuperAAPS GUI we have

1. A Catalogue of target stars and calibration observations.
2. A set of Queue Actions ("Start/Stop", "Pause/Continue" , "Abort") that control the queue
3. A set of current exposure actions ("Stop","Change","Abort") to stop, change the time of, or abort the current exposure.
4. A bunch of status information on what everything is doing, and a few more action buttons (Comment and Note buttons).
5.A queue of objects loaded from the Catalogue in 3 panels "Now Observing", "Next Target" and "Rest of Queue". Objects will be observed from "Next Target" and "Rest of Queue" in the sequence you provide.
6. A Messages window that tells you what the queue thinks its doing, and where errors and problems will be reported.
7. An Observing Log window with information on completed observations including the photon-weighted mid-time, seeing, SNR and comments you've added on individual runs.
8. A Notes window for notes you enter in things that happen during the night.
9. Menu options for less commonly used actions (Finalising the current log into a human friendly text output like that we used to keep in u85.logsheet3 format, Initialising a new log at the start of the night, Asking for images to have their seeing/SNR re-estimated, etc)

In addition to those GUI components, there's also
10.An exposure meter progress window, showing real-time SNR, count-rates and current exposure meter temperature
11.A perl/PGPLOT pop-up window where SNR and seeing estimates are shown for the last exposure taken. The code that plots this also works out the mid-time, gets the photon-weighted mid-time from the file (if present) and flags if the two are significantly different. (below)

General Principles.

In general, the idea is that you select a set of stars you want to observe (or calibrations you want to do), rearrange their order, adjust their exposure times, and hit "Start". Observing should begin, with the telescope slewing to the first target, setting up UCLES, beginning guiding and starting an exposure. When an exposure starts, it gets promoted from the "Queue" window to the "Now Observing" window, and the next planned observation gets promoted to the "Next" window.

When an exposure finishes, the "Now Observing" window will grey out, and readout will start. If the "Next" object is at a different position from the "Now Observing" one the telescope will start slewing to it. Once the readout (and slew and acquisition if needed) is finished, the "Next" object will get promoted to "Now Observing", and so on.

As each observation is written to disk, it will be post-processed by a perl script to produce a new display of seeing, SNR and some useful cuts through the image in the PGPLOT window, and the log window will be updated.

Setting time automatically (A)

The exposure meter not only determines the photon-weighted mid-point of exposures, but also estimates the accumulated SNR per wavelength pixel for our target stars in the iodine region. In principle this can be used to adjust the length of exposures to obtain a desired SNR.

The GUI also has a model for observing efficiency of AAPS stars as a function of V magnitude, seeing and S/N target. It uses this to estimate how long you'll need to observe a target (given the last measured seeing) to reach the S/N=100, 200 or 400 target set (for each object) in the catalogue. It then tries to break down that exposure into exposures that will be less than the maximum allowed exposure time (currently 450s, though that is a parameter you can change from the Observing pull-down menu).

If the exposure meter is being used to set times, it then works out how that split breaks up the S/N requirement, and sets a S/N target per exposure.

You can play with the specific target for an object by selecting it (once its in the queue) and hitting modify. By Entering the object in the pop-up as (A)uto and providing a S/N target (say 400 in 1 exposure, or equivalently 200 in 4 exposures), it will calculate a new split of the times to reach that target.

Setting time semi-automatically (S)

Sometimes that split is not quite right - you can override it and simply tell the GUI to do 3 S/N=200 exposures by using "Modify" with the (S)emi-auto tag. Semi-auto objects will still get their exposure times determined by the exposure meter ... but you get to determine the target S/N and how many exposures yourself.

Setting times manually (M)

In practice Auto and Semi-auto modes just don't work for some targets, and you'll want to hard-wire an exposure time and a number of exposures. For example on the Monster Runs, where every object gets observed for 20 minutes, its easiest just to manually put in exposures that add up to 20 minutes.

Use "Modify" and the (M)anual option for this - note that some objects are flagged in the input catalogue as only to be observed manually - usually these are faint objects like M-dwarfs where we just have to take what we can get in 1800s.

For AAPS observing, you should almost always be using the Manual or Semi-Auto mode as they are more robust.

Manipulating the queue

Accessibility: The Catalogue gets the telescope's current position once a minute and updates the HA and airmass each target would have if observed now. It 'greys out' objects that will be too far over to observe (though you can still select them and try to queue them).

Status and SNR Target: Each object in the catalogue has a 'status' flag - 'K' for known planet, 'H!' for hot (things we have flagged in the past for observing every night), 'P!' for interesting (get once per run). Each object also has a SNR target '4' for OMPS stars (SNR=400), '3' for TMPS stars (SNR=300), '2' for everything else (SNR=200) except for faint object like M-dwarfs, where the number is a maximum exposure time to use.

Some stars are also flagged 'No' under the Q? column - these are objects where user interaction is required to ensure you observe the right one, so they won't be acquired automatically - instead a pop-up will appear and you'll have to dismiss it to indicate the telescope is guiding on the right object.

Queued Objects: Once an object is queued the actual exposure time proposed (or the manual ones you've over-ridden its estimates with) are shown in the 'Time' and 'N' columns. And entry in the 'M' column means you've indicated the times are manual ones, and the exposure meter will not truncate or extend exposures based on SNR - it will just observe for those times.

You'll also see columns for 'UT', 'HA', 'AM' and 'Slw'. 'UT' is the UT time when its estimated this target will get observed, based on things ahead of it in the queue, 'HA' & 'AM' are the HA and AM it will have then, and 'Slw' is the slew time from the previous object in the queue to this object. You can use this information to play with the 'travelling salesman' problem of minimising slew times for objects in the queue.

Once objects are in the queue, you can select them and promote or demote them in the queue (up and down arrows), remove them ("Remove"), remove everything ("Clear"). There is also a button "Make Next" which will promote the selected object into being next in the queue. (Noting that if no slew was initiated when the readout started at the end of the current exposure, then making a new object 'Next" will result in the slew to the new target not starting until after the readout is finished. OR if the target you've just 'Made Next' is different form the one currently being slewed to, you'll have to wait for it to finish getting to the old target, before it will slew to the new one.)

Calibrations

The most commonly used calibrations for AAPS observing are listed at the top of the catalogue - Setup ThAr, ThAr, Flats, and Iodines. These can be queued, modified and rearranged like any other observation. There is also a "PAUSE" you can put in the queue that will make it stop and wait until you hit "Continue".

Changing your mind

So you've started observing an object, and you want to do something different or stop observing. What can you do? Well you actually have a lot of options. Common situations are

Cloud has come over and you want to stop observing, but wait and be ready to go again - well you can't close the shutter on an AAPS exposure and open it later ... your exposure mid-time would be useless. So what you should do is "Pause" the queue, then either "Stop" the current exposure (which will read out the data obtained to date) or "Abort" the current exposure (which is somewhat riskier). Then when the cloud parts, you can just "Continue" the queue.

The object I had next in the queue was incorrect, and when the shutter closed the telescope started slewing to the wrong object - this one is easy. Just "Pause" the queue, select the object you DO want to do next, hit "Make Next" which will put it into the "Next Object" entry. Then hit "Continue" - the GUI checks that the object in "Next" is the same as where the telescope is pointed before it starts an exposure. If its different, it will move the telescope to the correct target. Of course, you'll have slewed twice, but I can't help that ...

Maintaining the logbook & post-processing

At the end of every exposure, SuperAAPS runs a perl script (~/saaps/saaps_postproc.pl) which does some simple image processing to estimate seeing and SNR in a 'standard' box in the iodine region of the spectrum on the detector where our AAPS spectra should always lie. It also gets the photon-weighted mid-point and open-close mid-point, compares them and flags if they are significantly different.

That information (mid-time, seeing, SNR) is captured and used to create a logbook line in a format very similar to that we made by hand with the mtfits.pro IDL script. SuperAAPS maintains a 'working' set of this log information in an internal file format, as well as in human readable text format.

In addition, you can add comments to any one of these lines with the 'Add Comment' button, and add a timed 'Note' for the night with the "Add Note" button.

And finally, you can enter all the information we put at the top of our manual logs with the 'Observing->View/Update Log Header' menu.

To initialise these logs at the start of the night use 'Observing->Initialise New Night's Observing Log' from the menu bar. This will do three things

1. 'Finalise' the log data already there - ie. copy the text version of all the log information into a single file called uRR.YYMMDD.log in ~/saaps/final_logs. This is then the equivalent of the uRR.logsheetN files we currently make. You will be warned when you finalise a log. Usually this is OK, since all its doing is overwriting its previous idea of what the final log was with its updated idea. But its worth checking the date in the final log name to make sure it really is the right log for last night. Once finalised the log is changed to read-only so it can;t be accidentally modified in future.
2. Prompt you for new header details - at minimum these will usually need changing for the UT date.
3.Clear the log out and start a new one. When it does so, it creates a date-stamped copy of the working files, so it is always possible to 'go back' to an old state by exiting SuoerAAPS, copying the contents of ~/saaps/final_logs_working/YYMMDD into ~/saaps/working_logs, and restarting SuperAAPS. You will probably have to play with permissions when you do this as well - it is deliberately clunky. Overwriting a finalised log should only be done in extremus!

If during the night you create a crap file, and decide to use 'Reconfigure' to reset the run number and so create a new file with the same run number, SuperAAPS will not (by itself) notice this is a new file and so create a new log entry for it. You can force it to post-process the new object again with 'Observing->Reprocess Images into log'. Indeed you can use this to reprocess as many files as you like. One use for this is that if the 'standard' post-processing box is out by a few pixels, you can tweak it with 'Observing->Adjust post-processing box location', and then reprocess files. Note that reprocessing keeps your old comments. So if they are not relevant delete them by suing "Add Comment" to make the comment for that run empty.

Setting GUIDER Acquisition and EXPOSURE_METER modes

By default the exposure meter will try to set exposure times based on a SNR request. To change this default to using all manual exposure times (and to change back) there are pull down menu items in 'Observing' that will also determine whether the exposure meter is used (a) at all, (b) only to get mid-times and (c) in the default exposure time setting mode.

Similarly, the default behaviour of the GUIDER task is to try and have the GUIDER automatically acquire targets. If you are finding this is unreliable, then you can use 'Observing->Don't use guider to acquire' to turn this behaviour off. In this mode a pop-up will appear when you slew to each new target saying 'Dismiss me when the target is acquired'.

Related options in 'Observing' are those which reset flags telling SuperAAPS to stop polling the GUIDER and EXPOSURE_METER. (If communication with these stops for any reason, continued polling just makes things worse - the SuperAAPS stops polling them after they've failed once. Its then your job to get them connected again by restarting the tasks and/or resetting the AAO CCD Loader, before restarting polling by using the 'Observing->Try ???? again' menu items)

The Tracking Tool

When observing we often want to know what data we've collected for targets on previous nights in this run, on recent runs, or indeed within the last year or so. The Tracking Tool provides a way to get at that information. It reads all the log-sheets that have been made with the saaps system, and makes up a summary in a seperate window. The tracking tool come up automatically when you start saaps. It can be hidden and recreated with the "Tracking Tool->Show..." and "Tracking Tool->Hide..." menubar items.

Within the tracking tool the most important option is the "Configure->Select Runs to Track" menu item. By default, the tool will summarise the status of observations from the current run, putting a coloured box in each row and column for each night on the current run that summarises the S/N achieved on each night. However, you can tell it (as has been done in the example above) to track both this run and and previous runs as the 'current run', so that you can ensure (for example) that all objects get observed at least once over the course of two nearby runs.

The tool also shows you how many epochs (ie. nights) each target has been observed on over the last four months, and also in the last year.

The tool automatically updates if you read in a new target catalogue in the main window.

OBSERVING - SKYCAT and "THE OLD FASHIONED WAY"

Controlling the Telescope from SKYCAT

If observing with SuperAAPS is not working, it is straightforward to go back to the old fashioned mode of observing. Indeed you can do it even while SuperAAPS is running, controlling everything you need from the GUI.

However, what you DO need in this mode is a way of sending telescope co-ordinates. The easiest way to do that these days is with Skycat (NB this doesn't work in GAIA). If you've minimised Skycat then bring it up (and if you killed it then start it with > cd ~ ; skycat &).

Then use 'Data Servers->Local Catalogs->AAPS_Nov07_Target.txt' to bring up our target catalogue. To list the whole catalogue enter a very large number in "Max Radius:". Slewing to a target is then as simple as selecting it, the using 'Telescope->Slew to Selected Object', and responding 'Slew' to the resulting pop-up.

If skycat loses contact with PTCS, you can re-establish it with 'Telescope->Reset Server Connection'

You can use GAIA to do SNR and seeing calcs on our targets with the 'View->Pick Spectrum...' function, which requires you to draw a spectral box on the display (choose an order near X=705 in the centre of the I2 region), and then provides seeing estimates and SNR estimates for the relevant X and Y cuts.

LIMITATIONS

WHAT GOES WRONG (and how to fix it)

Shutdown/Resetting and Restarting/Resetting : The start-up and shutdown notes above contain most of the relevant useful information. Important tricks are

1. If there's a problem communicating with the GUIDER, EXPOSURE_METER or PTCS then the first thing to try is resetting UCLES from the AAO CCD Loader ("Commands->Reset"). This will establish communication with tasks that are still running, and restart any tasks that have died. So its usually faster than a full on kill and restart of UCLES. (But one common side-effect is that it breaks communication of the ODC GUI with the ODC, so you stop seeing the progress bars and times on the ODC GUI - though you will still see them on SuperAAPS)
2.After you've killed the UCLES tasks from the AAO CCD Loader, don't forget to cd ~ before re-starting UCLES. It will fail if you try to start anywhere except the home directory.
3.If you start in ~ and then see a 'Cannot communicate with AATVME15 error' then aatvme15 may need to be reset. If you don't know how to do that, ring for an afternoon technician.

MOST COMMON FAULTS

DITS Orphanned transaction messages : This is a problem that we have been seeing during the night (especially when the GUIDER is being used). It seems to be much less prevalent in March 2008, so not using the sequencer may have solved it. The symptom is that there are message on the SuperAAPS console saying "DITS Orphaned transaction errors". Sometimes these have no effect at all. Sometimes they cause SuperAAPS to crash.

Just start SuperAAPS again with the saaps command - if it won't let you start a new observation, then its time to restart the entire UCLES/ODC system by exiting saaps, and shutting everything down from the System Loader

GUIDER, EXPOSURE_METER, UCLES or PTCS dies: If one of these tasks dies, SuperAAPS should be able to continue on ... though obviously with somewhat reduced functionality! (If the EXPOSURE_METER dies in the middle of an exposure, you won't be getting an exposure mid-time, and the exposure will run out to the maximum length. If the guider dies your object may drift off the slit.)

SuperAAPS makes an effort to continue on if these tasks die, and produces pop-ups, when it gets to the stages in the queue where it needs the dead task, that say things like "Please fix this and hit CONTINUE to go on or STOP to halt queue." You'll also see messages in the startup window like "Skipped polling of GUIDER/EXPOSURE_METER", which just indicate SuperAAPS has noted the relevant task has died and has stopped trying to talk to it).

When this happens you need to fix the problem by (1) getting the dead task running again, and (2) getting it connected to Drama, with the 'Commands->Reset' function in the System Loader, then (3) hit CONTINUE on the pop-up to continue observing. Or use STOP to stop observing and fix everything, then restart observing,

May 2008 - changes in the way these pop-ups work should now make them more reliable. Prior to May 2008 they tended to lock-up the whole SuperAAPS GUI. They don't do that anymore.

PAST FAULTS (May 2008)

An examination of the error logs from March 2008 observing indicated three major bugs that occurred on multiple nights

1.Space in Drama's ring buffer for communication with the detector (RCT) task filled up (6 times in 5 nights). This repeats over the course of a minute, and eventually seems to result in the exposure meter stopping. In May 2008, I increased this buffer size to see if it helped.
2.When GUIDER acquisition failed, the signal that this had happened was not being correctly picked up (6 times in 5 nights), resulting in the queue having to be killed when it was about to start the next exposure. I believe I have (a) identified the possible cause of this bug and (b) made the pop-ups, that come up when the GUIDER is noticed to not be guiding properly before an exposure stars, work properly.
3.The EXPOSURE_METER spontaneously dies (3 times in 5 nights) which was causing SuperAAPS to have to be restarted. I have not yet identified the cause of the spontaneous EXPOSURE_METER deaths, but I have fixed the pop-up system, so that when EXPOSURE_METER does die, SuperAAPS will much more smoothly handle it.