.

 
Example of eyetracking in the real world during Neil Mennie's baufix task.




ASL 501 inside the V8 HMD



ASL 501 for the real world



ASL 210 inside the V8 HMD



ASL 501 (Left) and ASL 210 (Right) as mounted on the HMD optics



Here are some useful links regarding these trackers:

Applied Science Labs Homepage
501 VR Series
5000/501 Series Main Page
310 Series Limbus Tracker (There is no 210 page)

SR Research Homepage
Eyelink II Home

IScan Homepage
Polhemus EyeTracker (Developed by IScan)


Before an introduction into the calibration of the eye trackers and using their software and drivers,
a the hardware setup should be discussed...

The cable connections and general hardware/software setup for the eyetrackers is a bit involved and the
majority of this information is summarized quite well by the manufacturer in the following pdfs:

ASL 501 Manual
ASL 501 Eye-Head Integration Manual
ASL 210 Manual

This tutorial will give a general overview for the devices in the lab and how to get
something up and running with a simple software example but should not be considered a
definitive reference.




ASL 501 Hardware
The 501 series video based eye tracker consists of a host PC, a dedicated
control unit, an eye camera, and a scene camera. All of these items are
located in room 628 - The host PC station is shown below along with the
control unit which resides on the rack.



Two views of the electronics rack in room 628. The ASL tracker is on the 3rd shelf from the top.




The ASL host PC station: (Left) Scene Monitor with color corrector and video mixer on top.
(Middle) Monitor on 4-way switch (ASL/I-Scan/Kart/HiBall).  (Right) Eye video monitor.



The ASL 5000 Control Box on the rack


General Connections and Setup:


There are many connections for the eyetracker but the general flow can be described as follows:

The eyetracker's job is to take a data signal measuring the position of the eye from an eye camera and map
this input onto a secondary video signal (the scene camera), allowing the experimenter to have a representation
 of the point of gaze of a subject from their first person point of view.  The eyetracking system has a dedicated
control box to perform these computations. However, the control box needs to have control software uploaded
to its memory everytime it is turned on - The host PC is used to upload software over a serial connection to the
control box and to provide a GUI for calibration as well as display the output of the controller onscreen.
For the controller to do its job it needs inputs from the (1) scene camera, (2) the eye camera, and (3) a serial connection
with the host PC.   The control unit provides three outputs: (1) A processed eye image, (2) A scene
 image with a crosshair or square showing the point of foveation. (3) Serial data output for data recording.


ASL 5000 Series Control Box

     Inputs                    |              Outputs          
  Eye Camera Video    |     Processed Eye Video
                             Remote Scene Camera Video     |     Processed Scene Video w/ Gaze Overlay            
      Serial link from Host PC    |     Serial Out for Data collection
Power    |               
Head Tracker (optional)       |                                              


The actual connections can vary depending on the application but here is the general connection
scheme (refer to the pictures below):


The 5000 series control unit shown from the back

The most basic connections include power (the switch is on the front),  and the controller port.
The controller is a serial connection that goes from the 5000 box to the host PC (there is nothing
proprietary here any pc with a serial connection and the eyetracking software could do the job).


 
The ASL 5000 with all of the connections plugged in (Not the same unit as the unplugged one above but
both have similar clearly labeled connections)



Setting up the Eye Camera:


The other two important connections are the eye camera and scene camera input.
The large eye camera cable connects to the port labeled camera (See the two pictures below
with the two trackers  and their camera cables). Now The VR tracker and the realworld tracker
eye cameras have differing connections as shown in the pictures below - but the eye video signal
ultimately goes to this connector on the back of the box.



(Left) TheV8 helmet with tracker camera cables and eye camera control box (Right) Real world
tracker with  eye camera cables and Elmo scene camera control box.


Since there are two trackers there is an A/B switch that allows
one to alternate between the two. The connections for this port are already setup so one really should
only need to use the A/B switch and not actually touch the connector on the back of the control box.
This switch is low on the rack and shown in the picture below:


The eye tracker switch box on the the first shelf from the bottom of the rack



If you are using the the real world tracker there is no further setup for the eye camera other than
putting the switchbox in the Real world setting and then adjusting it to obtain a good picture of the eye.
  However, if you are using the tracker inside the V8 helmet there are other steps to setup
the eye camera properly.  The large camera cable goes to the eye camera's control
box which recieves input from  the eye camera in the helmet.   There are four connections that need
to be made between the eye camera control box , the eye camera CCD unit, and the camera cable
The outputs/inputs of importance are:

        Camera Cable (Connected to Camera Port on ASL 5000)  has 3 connectors: DC Power,   Illuminator Power,  Eye Video In
The Toshiba Eye Camera Control Box has 3 inputs:  Video Out, Camera In, Power In
Eye Camera CCD Head (mounted in the V8) has two connectors:  Illuminator Power, Camera Out



The V8 Eye camera control unit  with proper connections

Each connection has a unique plug (with color coding) so the connection is fairly simple. 
First plug the green power cable (part of the large camera cable) into the DCIn12V port
on the Toshiba Control Box.  Now Plug the Red BNC connector to the VIDEO connector.
Now plug the grey CCD camera cable into the camera connector on the front of the box to the
left of the on/off switch.  Finally connect the illuminator power connector from the camera
cable (with the blue/red) tag to the illuminator power connector coming from the eye camera
in the helmet (also with a blue red tag).  Now when powered on the camera should display
a picture of an eye (once the control software is uploaded and someone wears the HMD).


Setting up the Scene Camera:

With the eye camera setup, the last input required is the scene camera - if the real world tracker is
 being used there are two options for the scene camera. There is a low image quality CMOS
camera and a high quality CCD camera available to use as the scene camera. For both cameras
there are a few lenses to choose from depending on the field of view required for an experiment (located
in the grey cabinet next to the 628 entrance.

 
CMOS camera (left) and Elmo CCD camera (right)



The Elmo CC Controller



If the CMOS camera is used its output is actually carried over the main camera cable that also transmits the eye (on the
head band there are two connectors one for video and one for power that attach to the two connectors on the camera

However, if the CCD camera is used the video signal must be sent to the remote scene connector.
To use the elmo camera the CCD head must be plugged into the camera jack.  There is also a power
connector that plugs into the DCin12V.  The video signal is carried through the VIDEO connector -
this video signal should be plugged into the Remote Scene In connector on the rack (see below)

  In the VR setup there isn't a physical camera so instead the video output from the rendering
engine  must be sent to the remote scene input. The rack has a patch panel to make
swapping video signals easier, as such making the direct connections to the back of the
ASL5000 control unit aren't necessary, instead only the patch panel connections need to be adjusted.
90% of connections use BNC cables and connectors extras can be found in the connectors box inside the
grey cabinet as well many cables on the bottom shelf there are also some cables and connectors
for the consumer RCA plug style .


Video Output:



The video source patch panel



The video source patch panel with connections for the VR eye tracking setup

This example shows the following: The processed eye output (ASL Eye) is sent to Eye Mon1
(ie eye monitor). The scene video is sent from Silicon out to ASL Remote Scene In.
The processed scene video output (ASL Scene) is connected to the Scene Monitor. At the
eye tracking computer station a video mixer on top of the scene monitor combines the eye and
scene video in picture-in-picture format. This combined signal is outputted to the color corrector
which in turn sends the video back to the rack at Mixer Out. The Mixer Out signal is sent to the Hi-8
input. The Hi-8 player can record this video and sends its output (Hi-8 OUT) to the the large TV in the lab (TV1).
Please note if any parts of this chain aren't turned on (ie the vcr, color corrector, or mixer) there will
be no video displayed on TV1 - so be sure to do so.   If you want to use the mini-DV just swap the
connections with the Hi-8 in/outs.

Now considering a different setup if the real world tracker were used, the only difference  for the patches
would be that the ASL Remote Scene In connector would need to have the video output from the Elmo CCD
camera sent to it. If the CMOS camera is used no input to ASL Remote Scene is needed.

To view the video streams there are two monitors labeled EyeMon1 and Scene Monitor that allow the
the eye and scene video to be previewed. Typically these individual outputs are combined via the video
mixer to have the first person point of view rendered with a picture-in-picture  window displaying the
eye. This mixed video signal can be sent to the Hi-8 or MiniDV for recording.

Serial Output:

The ASL controller cannot function without having software uploaded to it from a
host PC (shown below).  The Host PC also serves to allow calibration of the system
as well as realtime display of raw data.
 



PLEASE NOTE: The ASL host PC is the second on the right on the floor (soon to be upgraded). The scene tv monitor
is on the left. The eye tv monitor is on the right. The computer monitor, keyboard and mouse
in the center are on a switch that allows them to control four different computers. The switch
is located to the left and behind the computer monitor. The eyetracking computer is #1 - you
can switch between computers by pressing the button on the switch box or alternatively press the
scroll lock key on the keyboard twice and then press the number 1 (non-keypad) to switch to the
ASL, assigned numbers for switchbox ASL(1), I-Scan(2), Kart(3), HiBall(4) ).


Uploading/Starting Control Software

There are two stages in the use of the ASL tracker software. First, once turned on the controller box needs
to have the control software uploaded to it this can be done by double-clicking on the icon labeled "Upload to ASL5000":






The program will open a DOS window (see above) and count down the number of bytes needed to transfer between
the two units. Once this transfer is complete the window will dissapear.  Make sure the upload has been successful
- if you start the EyePos or Eyetrack programs when it has failed they will inform you that the controller
is offline.  The most common error in uploading generates a statement like "Cannot connect to ETE-control unit" in
the Upload text box.  Usually cycling the power for the 5000 control box and trying the upload again will clear this up.
There is a text configuration file caled E5000.cfg in the



Assuming the upload completed - Double clicking on the "EyePos 1.24 DOS" icon
  to open the dos control software. This interface is used primarily by a text prompt




Or Alternatively double-click on "ASL Eyetrack 1.37 (or later Version)" which will
start the windows version which is identical to dos but with a windows GUI).



At this point the subject should be wearing the eyetracker.  In the ASL software click on the check
boxes labeled Power with boxes for both the illuminator and camera (type '/IP <hit enter key>'.  Once the power is on the eye
image will appear on the eye monitor screen (if the eye is lined up , connections are correct, and power is on).
Once turned on the next step is to adjust the camera for a good viewing angle of the eye - of course
with two different eye trackers this means two methods for aligning the eye image.


Obtaining a Good Eye Image


Getting a good image of the eye can be frustrating at times regardless the general goal is to try
and get an image that is centered on the monitor (ie the pupil is in the center when the subject looks
at the center of the calibration pattern).  Additionally one needs to verify that when the subject
looks through all the calibration points that the track remains consistent and there are no dropouts
in the recognition of the pupil and corneal reflections (see below for more details on this point).


Example of typical orientation and position of eye camera in the V8 helmet.

Inside the V8 helmet there are two thumb screws that hold the camera in position. Loosening
the upper knob allows vertical adjustment and some horizontal rotation.  The lower thumb screw allows the camera
to rotate toward and away from the the eye as well as slide up and down about two centimeters along a small groove.
Refer to the pictures in the introduction to see an example of the camera's placement - this picture
does not show a subject wearing the helmet but this position generally will line up with the eye
quite easily - sometimes the  lower thumb screw needs adjustment to get the eye centered in the image
or to avoid reflections caused by the edges of the beam splitter.

The real world tracker tends to be more difficult to get the proper camera angle
Below is an example of a successful reflector and camera angle for the real world setup. 
Of course you will have to judge if the eye image seems ok - but this picture
shows the general position the camera and relfector should be in.



Processing the Eye Image

The control unit processes the incoming eye video to find two reflections, the corneal reflection
(the glint in one's eye)  and a pupil reflection.  The centroid of the analyzed reflections are used as
landmarks that allow eye tracking (two reflections are used so that eye movements can be disambiguated
from head movements that cause the tracker's headband slip).  Once a good eye image has been
acquired thresholds for the Pupil Reflection and the Corneal Reflection (CR) must be adjusted so
that the reflections can be extracted from the processed video with minimal noise.  There are no set
 thresholds to use one must simply test out levels through trial and error to find a set of settings for eye
 tracking to be successful. The main goal is to have two definite circle that segment the eye's two reflections.
Using the up and down arrow keys (in DOS) or moving the pupil slider (windows) will increase/decrease
 the pupil threshold.  Using the left and right arrow keys or moving the CR slider will increase/decrease
the CR theshold.  Examples are shown below of the actual eye image from  the eye monitor and with
examples of a correctly tracked eye, an eye with threshold values that are too high/low.  One may also adjust
the amount or infrared illumination on the eye by pressing the PageUp/PageDown keys or the slider bar. 
In general the illumination does not need to be adjusted from its point in the middle (Level 7).  More often the
threshold settings and the angle at which the eye camera views the eye will cause problems in getting a
reliable track.  With the appropriate threshold settings the computer monitor and the eye monitor will show
 two circles (A large one tracking the pupil and a small one slightly offset tracking the CR reflection See Figure G). 



Above Examples:




These pictures should give some idea for what the processed eye image should look like with
good and bad settings.  Typically one adjusts the settings to look like example G and has the subject look
around the scene to make sure the tracking is consistent across most of the field of view.


Calibration (Mapping the Eye to Scene)

Once the eye image is being tracked the two signals from the eye's reflections must be related to the
visual scene through a calibration process.  The procedure has some variations but the general idea is
 that a series of points in the visual scene can be used as input to a function that will allow the eye
tracker to generate a fixation cursor or crosshair that allows the experimenter to see where the
individual is foveating in the scene.  The calibration procedure between VR and reality are conceptually
identical  - have the subject look through 9 points in the visual scene. There are usually 9 points arranged in
a 3x3 grid pattern - like below but scaled to cover the field of view that the subject will be operating over)


1    2    3

4    5    6

 7    8    9




Example of Calibration Pattern in the Real World - the Nine blue points in the image are
the calibration points - By capturing the eye data while having the subject look at each individual point
will calibrate the system for use.


On each point the eye tracking system captures the data values generated for a given fixation point and
saves this information.  When the subject has looked through all nine points the saved values can be used
to calibrate the video overlay system that will provide the graphical representation of the subject's point of
gaze.   VR programs developed in the lab have code for displaying several calibration patterns inside the HMD. 
Because the HMD is fixed to the head, calibration is fairly straighforward.  Display the pattern in your program
(usually ' / ' key ) and then continue with the rest of the process.  In the real world this is more difficult because the
scene camera must be aligned with the calbration pattern and then subject must remain relatively motionless
during the calibration such that the scene camera's position is almost static in relation to the calibration
 points.  The picture below shows a real-world calibration pattern.  The lab does have a laser calibration pattern
that allows for greater movement during calibration.  There is a on/off switch located a few meters down the
eye tracker's thick camera cable.  If on the laser will project a grid pattern of points of light (generated by a
grating placed in front of the laser emitter.  The laser is aimable such that one can align the pattern with its
 projected surface as needed.   Of course during calibration the subjects head should still remain pointed at the same
depth plane otherwise the pattern will be distorted.   The laser can also be useful to serve as a simple indicator
of head movement - for instance if one wants to calibrate to a table surface off of points on the table
the laser can be aligned with one of the calibration points and the subject will now know if
the moved their head if the laser point moves off of the point it was aligned with.

The actual calibration procedure follows like thius, for instance imagine that the subject sits in front of a table with
point upon which they will be calibrated.  The experimenter aligns the scene camera such that the camera is centered on
the central point in the calibration pattern.  Assuming that the eye and scene cameras are in place the subject should
remain still while the calibration points are input.  In this part of the setup one must click on the sweep target points
icon (or go through the menus:  Calibrate>Set Target Points)  or more simply in the DOS program type "/sw" (without quotes).
 The program will then prompt you to input the target points during this time when the mouse is moved in the
Scene POG window, a cursor on the Scene Monitor will move around.  Place the cursor onto the nine points in
the calibration pattern clicking the left mouse button on each.  During this time the program will count
through the calibration points that have been entered.  Once all nine have been entered you can double
check by clicking on the check targets icon (or typing "/tg" and entering 1-9 to check that point).
Type '/ec <enter>' to start the eye calibration process.  In this mode a cursor will appear on the scene video at the
pointsyou previously clicked on during the sweep phase.  Ask the subject to look fixate point 1 and press the space
key when the eye is stable.  Do this for the nine points - if all goes well the subject should now be succesfully
tracked and the Remote  Scene Video will accurately display their point of gaze, and the serial output will contain the raw data.
Else recalibrate and/or adjust the angle of the reflector.  During the experiment the track may become systematically offset
typing '/EO <enter>' will allow you to off set the POG cursor by pressing the up/down and left/right arrow keys -
just have the subject fixate a know point and align the track to that point.

There are still many fine points about the tracker that are covered in the manual - including
instructions for writing drivers, options for eye-head integration, the option for using
an alternate scene camera (not mounted on the subjects head) with the POG cursor, and discussions of
ASL's analysis software - so read the manuals if you want a more in depth knowledge.