Deep Heterogeneous Hashing for Face Video Retrieval

Abstract

Retrieving videos of a particular person with face image as query via hashing technique has many important applications. While face images are typically represented as vectors in Euclidean space, characterizing face videos with some robust set modeling techniques (e.g. covariance matrices as exploited in this study, which reside on Riemannian manifold), has recently shown appealing advantages. This hence results in a thorny heterogeneous spaces matching problem. Moreover, hashing with handcrafted features as done in many existing works is clearly inadequate to achieve desirable performance for this task. To address such problems, we present an end-toend Deep Heterogeneous Hashing (DHH) method that integrates three stages including image feature learning, video modeling, and heterogeneous hashing in a single framework, to learn unified binary codes for both face images and videos. To tackle the key challenge of hashing on manifold, a well-studied Riemannian kernel mapping is employed to project data (i.e. covariance matrices) into Euclidean space and thus enables to embed the two heterogeneous representations into a common Hamming space, where both intra-space discriminability and inter-space compatibility are considered. To perform network optimization, the gradient of the kernel mapping is innovatively derived via structured matrix backpropagation in a theoretically principled way.

Existing System

Approximate nearest neighbor (ANN)

Proposed System

In this study, the query and database are provided with different forms, i.e, still images (points) v.s. videos (point sets), where each face image or video frame is represented as a point in Euclidean space. The core problem of the task is to measure the distance between a point and a set. One straightforward method is to compute the distance between the query image and each frame of the video first, and then take the average or minimum of these distances. However, such a method has two major limitations: 1) All frames representations need to be stored and heavy time cost is brought for computing all pairs of distances between still images and video frames. This would become seriously inefficient in case of long videos and high dimensional image representations. 2) It will heavily suffer from large appearance variations in realistic face videos caused by expression, illumination, head pose, etc.

Architecture

BLOCK DIAGRAM