DUAL-HAND DETECTION FOR HUMAN-ROBOT INTERACTION BY A PARALLEL NETWORK BASED ON HAND DETECTION AND BODY POSE ESTIMATION

Abstract

In this study, a parallel network based on hand detection and body pose estimation is proposed to detect and distinguish human’s right and left hands. The network is employed to human-robot interaction (HRI) based on hand gestures. This method fully uses hand feature information and hand information in the human body structure. One channel in the network uses a ResNet-Inception-Single Shot MultiBox Detector to extract hand feature information for human’s hand detection. The other channel estimates human body pose first and then estimates the positions of the left and right hands using the forward kinematic tree of the human skeleton structure. Thereafter, the results of the two channels are fused. In the fusion module, the human body structure can be utilized to correct hand detection results and distinguish between the right and left hands. Experimental results verify that the parallel deep neural network can effectively improve the accuracy of hand detection and distinguish between the right and left hands effectively. This method is also used for the hand gesture based interaction between astronauts and an astronaut assistant robot. Our method can be suitably used in this HRI system.

Existing System

Region-based Convolution Neural Network (R-CNN), Fast RCNN, Faster RCNN

Proposed System

The main contributions of our work as follows: (1)a RI-SSD network that changes the structure of SSD using ResNet and Inception network is proposed to increase the accuracy of hand detection. (2) An improved body pose estimation method based on body structure forward kinematic (FK) tree is proposed to estimate the keypoints of left and right hands. (3) A fusion module is proposed to fuse the results of hand detection and dual-hand position estimation for dual hand detection. (4) Our dual-hand detection method can be used in static and dynamic hand gesture recognition systems. The method is also used in a space HRI system.

Architecture

BLOCK DIAGRAM