Object-based Multiple Foreground Segmentation in RGBD Video

Abstract

We present an RGBD video segmentation method that takes advantage of depth data and can extract multiple foregrounds in the scene. This video segmentation is addressed as an object proposal selection problem formulated in a fully connected graph where a flexible number of foregrounds may be chosen. In the graph, each node represents a proposal, and the edges model intra-frame and inter-frame constraints on the solution. The proposals are generated based on an RGBD video saliency map in which depth-based features are utilized to enhance identification of foregrounds. Moreover, our method provides performance comparable to state-of-the-art RGB video segmentation techniques on regular RGB videos with estimated depth maps.

Existing System

Multi-State Selection Graph, Shortest Path Algorithm.

Proposed System

Our method operates on a given video and its depth maps by first generating a pool of foreground proposals for each frame. For each proposal, an RGBD video saliency score is computed and added to the RGBD objectness score calculated in proposal generation to measure the likelihood that the proposal belongs to the foreground. With the proposals and their likelihood scores, a proposal selection graph is built, with each node representing a proposal in a frame. The edges in the graph represent pairwise terms that connect each pair of proposals between consecutive frames, or constraint terms that restrict proposal selection within a frame or between different frames. This selection graph is formulated as a binary integer quadratic program (IQP) problem, which we optimize by using the fixed-point iteration technique. Our method has the following advantages: 1) Provides greater flexibility by handling any number of foreground objects in each frame of the video, which can vary frame to frame due to object occlusions and foregrounds moving in and out of the field of view. 2) Introduces a measure for RGBD video saliency that leads to improved foreground identification in RGBD video. 3) Utilizes depth as a cue that complements RGB features. Besides these advantages, we also show that our technique can improve multiple foreground segmentation in regular RGB video by making use of estimated depth maps.

Architecture

BLOCK DIAGRAM