Dynamic Pose-Robust Facial Expression Recognition by Multi-View Pairwise Conditional Random Forests

Abstract

Automatic facial expression classification (FER) from videos is a critical problem for the development of intelligent human-computer interaction systems. Still, it is a challenging problem that involves capturing high-dimensional spatio-temporal patterns describing the variation of one’s appearance over time. Such representation undergoes great variability of the facial morphology and environmental factors as well as head pose variations. In this paper, we use Conditional Random Forests to capture low-level expression transition patterns. More specifically, heterogeneous derivative features (e.g. feature point movements or texture variations) are evaluated upon pairs of images. When testing on a video frame, pairs are created between this current frame and previous ones and predictions for each previous frame are used to draw trees from Pairwise Conditional Random Forests (PCRF) whose pairwise outputs are averaged over time to produce robust estimates. Moreover, PCRF collections can also be conditioned on head pose estimation for multi-view dynamic FER. As such, our approach appears as a natural extension of Random Forests for learning spatio-temporal patterns, potentially from multiple viewpoints.

Existing System

Facial Action Coding System, Non-negative sparse Decomposition.

Proposed System

In this paper, we presented an adaptation of the Random Forest framework for automatic dynamic pose-robust facial expression recognition from videos. We also introduced a novel way of integrating spatio-temporal informations by considering pairwise RF classifiers. This formulation allows the efficient integration of high-dimensional, low-level spatio-temporal information through averaging over time pairwise trees. These trees are conditioned on predictions outputted for the previous frames to help reducing the variability of the ongoing transition patterns. In addition, we proposed an extension of the PCRF framework to efficiently handle head pose variation in an expression recognition system. We showed that our models can be trained and evaluated efficiently given appropriate data, and lead to a significant increase of performances compared to a static RF. The Matlab code used to render the images that we used for training and testing the classifiers will be made publicly available. Finally, we showed that our method works on real-time without specific optimization schemes, and could be run on low-power architectures such as mobile phones by using appropriate paralellization scheme.

Architecture

BLOCK DIAGRAM