2019 · 40 citations · 55 references
ComputingComputer ScienceRobotics EngineeringComputer VisionMachine VisionSensorsDepth EstimationGesture RecognitionGrasp DetectionMobile Human-computer InteractionMotion CaptureMultimodal InteractionGesture RecognizersHand GesturesHand InteractionHand Pose EstimationHand TrackingMergingOnline CalibrationVision-based Gesture Recognition
We present HandSee, a novel sensing technique that can capture the state and movement of the user's hands touching or gripping a smartphone. We place a right angle prism mirror on the front camera to achieve a stereo vision of the scene above the touchscreen surface. We develop a pipeline to extract the depth image of hands from a monocular RGB image, which consists of three components: a stereo matching algorithm to estimate the pixel-wise depth of the scene, a CNN-based online calibration algorithm to detect hand skin, and a merging algorithm that outputs the depth image of the hands. Building on the output, a substantial set of valuable interaction information, such as fingers' 3D location, gripping posture, and finger identity can be recognized concurrently. Due to this unique sensing ability, HandSee enables a variety of novel interaction techniques and expands the design space for full hand interaction on smartphones.
55
Gary Bradski · Medical Entomology and Zoology · 2000 · 3.1K citations
Jaime Lien, Nicholas Gillian, Mustafa Emre Karagozler et al. · ACM Transactions on Graphics · 2016 · 938 citations
Communications in Computer and Information Science
Daniel W. Lewis, Trevor Martin · 2014 · 736 citations