Detection and tracking of pianist hands and fingers

Abstract

Current MIDI recording and transmitting technology allows teachers to teach piano playing remotely (or off-line): a teacher plays a MIDI-keyboard at one place and a student observes the played piano keys on another MIDI-keyboard at another place. What this technology does not allow is to see how the piano keys are played, namely: which hand and finger was used to play a key. In this paper we present a video recognition tool that makes it possible to provide this information. A video-camera is mounted on top of the piano keyboard and video recognition techniques are then used to calibrate piano image with MIDI sound, then to detect and track pianist hands and then to annotate the fingers that play the piano. The result of the obtained video annotation of piano playing can then be shown on a computer screen for further perusal by a piano teacher or a student.