Department of Brain and Cognitive Sciences Department of Earth and Environmental Sciences: Paleomagnetic Research Group MVRL: Multidisciplinary Vision Research Laboratory

An Active Vision Approach to Understanding and Improving Visual Training in the Geosciences

The Field Experience

Mobile Eye-Tracking during University of Rochester's Earth and Environmental Science Department's Courses:

102Q -- “Earthquakes, Volcanoes, and Mountain Ranges in California: A Field Quest”


202Q -- “Plate Tectonics and Active Geologic Processes in California”

“Understanding how the earth works starts with an appreciation of geological processes in action. To observe these dynamic processes such as earthquakes, volcanic eruptions, and mountain formation, earth scientists must travel to areas of geological youth, such as California. In this course, students are introduced to active geology through readings and discussion sections in preparation for a field excursion to California. Students will learn to examine critically ideas on how Earth science systems work and how active processes affect society.” -- From the University of Rochester's course descriptions.

In coordination with the course(s) taught by Professor Dr. John A. Tarduno (of the University of Rochester), students participate in a 10-day field trip across California. We have taken the opportunity to go along on this field experience and invited expert geologists, who have not been to these sites before. Together, the experts and (novice) students participate in visual search tasks at around 20 different sites throughout California, pertaining to landscapes formed through plate tectonics, volcanic activity, and glacial activity. With minimal guidance the observers are all asked to view a particular landscape from an expertly chosen vantage point and attempt to understand and note what signifiers there are that a certain geological process has occurred during the formation of this region. This is our essential experimental setup, and to attempt to discern the way the novices and experts are different, to record how novices learn over the 10 days, and to elucidate what makes an expert an expert, we use mobile eye-tracking technology (particularly models designed by Positive Science, LLC.) to continuously record each observer's (subject's) eye movements during each visual search task.

The mobile eye-tracker is designed to be lightweight, unobtrusive, and perform indoors, outdoors, and with variable lighting conditions. There is an infrared LED and an infrared-pass filter on the eye-facing camera. In this way, the eye camera can record the movements of the eye in the entire range of lighting conditions from sunlight to total darkness, and being an infrared (IR) LED its radiation is not visible (and thus, not distracting) for human observers. The scene-facing camera is of the same design, but with no filter, and so it records (through the Bayer filter and de-mosaicking) directly into tri-chromatic, visible-spectrum color; with both cameras running at 60 frames-per-second, interlaced.

After we return from the field experience, we have terabytes of recorded audio, video, high-resolution scene imagery, and GPS data. This requires a considerable amount of work dedicated to organizing, storing, correlating, managing, processing, and then analyzing all that data. The first step of this process is to use the correspondence and mapping software (from Positive Science, LLC.) to find then relate a subject's pupil position to a point in the scene video (the Point-of-Regard [POR]). With the combination of head-movement calculations from the scene camera, and eye-tracker movements stabilized through corneal reflections of the IR LED, we can use this POR location as our estimate of the subject's gaze into the natural world as imaged by the scene-facing camera.

We are in the unique position, through the Multidisciplinary Vision Research Laboratory (MVRL) in the Chester F. Carlson Center for Imaging Science (CIS) at RIT, of being able to simultaneously record up to 13 subjects in the field. In this way we are able to capture each individual subject's eye-movements during the same visual search task, and based on real-world events that all subjects view, we can also synchronize these recordings to absolute time as tethered to a commonly viewed event (like a professor raising their hand). With individual recordings per subject we are presented the opportunity to make unique analyses per person, but we are also facing the complex task of mapping data into a common domain for analysis.

As part of our focus on education in the geosciences, we also sought to capture very high-resolution panoramic image sets at each site of the geologically significant scenes and landscapes. This imagery is being organized and modified to work together with custom software for our own presentation of virtual field-like experiences. One method of exploring the use of the imagery is through our semi-immersive experiments. The imagery is captured in a 12 megapixel camera that rotates through a full sphere to capture overlapping views at chosen angles. Together, the imagery can be aligned, stitched, and blended to create rectilinear panoramic photos (as shown above, based on an equirectangular projection) or the image set (typically 150-200 images) can be processed individually. Using the imagery as a set or panorama, we can perform or derive mappings of each subjects' POR values from their scene videos onto the full-scene imagery. This creates a common domain for all subjects, and using the same imagery (or deriving mappings between them) we can also present subjects from different years into the same domain. This allows for cross-subject, cross-year analyses and pattern recognition as we aim to discern the pedagogical and information-processing characteristics of novices and experts in geosciences.

We are in the process of developing analyses for the data captured during these experiments. Please check back later for example results and links to any publications.
Thank you!