DYNAMIC POSE-ROBUST FACIAL EXPRESSION RECOGNITION BY MULTI-VIEW PAIRWISE CONDITIONAL RANDOM FORESTS

Abstract

Automatic facial expression classification (FER) from videos is a critical problem for the development of intelligent human-computer interaction systems. Still, it is a challenging problem that involves capturing high-dimensional spatio-temporal patterns describing the variation of one’s appearance over time. Such representation undergoes great variability of the facial morphology and environmental factors as well as head pose variations. In this paper, we use Conditional Random Forests to capture low-level expression transition patterns. More specifically, heterogeneous derivative features (e.g. feature point movements or texture variations) are evaluated upon pairs of images. When testing on a video frame, pairs are created between this current frame and previous ones and predictions for each previous frame are used to draw trees from Pairwise Conditional Random Forests (PCRF) whose pairwise outputs are averaged over time to produce robust estimates. Moreover, PCRF collections can also be conditioned on head pose estimation for multi-view dynamic FER. As such, our approach appears as a natural extension of Random Forests for learning spatio-temporal patterns, potentially from multiple viewpoints. Experiments on popular datasets show that our method leads to significant improvements over standard Random Forests as well as state-of-the-art approaches on several scenarios, including a novel multi-view video corpus generated from a publicly available database.

Existing System

Pyramid Histogram of Orientation Gradients (PHOG) features from salient facial regions, PCA & k-means clustering.

Proposed System

This system proposed to condition pairwise trees on specific expression labels to reduce the variability of ongoing expression transitions from the first frame of the pair to the other one. Furthermore, similarly to, it can further condition the pairwise trees on a head pose estimation to add robustness towards head pose variations. When evaluating a video frame, each previous frame of the sequence is associated with this current frame to give rise to a pair. Pairwise trees are thus drawn from the dedicated PCRFs w.r.t. prediction for the previous frame. In Extenso, a head pose estimate can be used to draw trees from MVPCRFs for pose-robust FER. Finally, predictions outputted for each pair are averaged over time to produce a robust prediction for the current frame.

Architecture

BLOCK DIAGRAM