MONOCULAR CAMERA BASED FRUIT COUNTING AND MAPPING WITH SEMANTIC DATA ASSOCIATION

Abstract

In this letter, we present a cheap, lightweight, and fast fruit counting pipeline. Our pipeline relies only on a monocular camera, and achieves counting performance comparable to a state-of-the-art fruit counting system that utilizes an expensive sensor suite including a monocular camera, LiDAR and GPS/INS on amango dataset. Our pipeline begins with a fruit and tree trunk detection component that uses state-of-the-art convolutional neural networks (CNNs). It then tracks fruits and tree trunks across images, with a Kalman Filter fusing measurements from the CNN detectors and an optical flow estimator. Finally, fruit count and map are estimated by an efficient fruit-as-feature semantic structure from motion algorithm that converts two-dimensional (2-D) tracks of fruits and trunks into 3-D landmarks, and uses these landmarks to identify double counting scenarios. There are many benefits of developing such a lowcost and lightweight fruit counting system, including applicability to agriculture in developing countries, where monetary constraints or unstructured environments necessitate cheaper hardware solutions. 

Existing System

Support Vector Machine (SVM), RGB-based oversegmentation

Proposed System

The major concept of this system is: (1) a fruit-as-feature semantic SfM is proposed, which significantly reduces the computation and outputs a meaningful map of fruit landmarks; (2) these fruit landmarks are then used to identify double tracked fruits, and eliminate frame-to-frame tracking noise caused by illumination shifts; (3) a new double counting problem introduced by collecting two separate image sequences from two opposite sides of the tree row is solved, which makes this pipeline applicable to a wider range of fruit counting tasks; (4) a more consistent and robust Kalman Filter scheme for frame-to-frame tracking is designed; and (5) a thorough comparison against both actual field counts and the counts estimated by a benchmark algorithm using much more expensive sensors is conducted.

Architecture

BLOCK DIAGRAM