QUANTIFYING GAZE BEHAVIOR DURING REAL WORLD INTERACTIONS USING AUTOMATED OBJECT, FACE, AND FIXATION DETECTION

Abstract

As technologies develop for acquiring gaze behavior in real world social settings, robust methods are needed that minimize the time required for a trained observer to code behaviors. We record gaze behavior from a subject wearing eye-tracking glasses during a naturalistic interaction with three other people, with multiple objects that are referred to or manipulated during the interaction. The resulting gaze-in-world video from each interaction can be manually coded for different behaviors, but this is extremely time-consuming and requires trained behavioural coders. Instead, we use a neural network to detect objects, and a Viola-Jones framework with feature tracking to detect faces. The time sequence of gazes landing within the object/face bounding boxes is processed for run lengths to determine “looks”, and we discuss optimization of run length parameters. Algorithm performance is compared against an expert holistic ground truth.

Existing System

CNN (Convolutional Neural Network), R-CNN

Proposed System

This project brings together multiple different technologies to enhance our understanding of gaze behavior in real-world situations. Currently, the use of realworld eye-tracking is limited because the first-to-market glasses-based eye-trackers were expensive, and the resulting gaze-in-world data was difficult to analyze in any automated or semi-automated way. The open source model offered by Pupil Labs has made glasses-based eyetracking both affordable and customizable. The system developments described here allow us to automate the count and duration estimate of looks to faces and objects during a social interaction. Because of the prevalence of ASD and its social interaction challenges, together with the subjectivity and difficulty in current methods for assessing the success of therapeutic efforts, investing in objective and quantitative social outcome measures can be useful to measure efficacy of social therapies.

Architecture

BLOCK DIAGRAM