Interactive tangible user interface for music learning

Abstract

This paper introduced muSurface, an interactive surface with ability to interact with tangible musical symbols. Children can explore musical symbols by placing the symbols on the virtual musical staffs displayed on the playing surface, and then the system plays the resulting melody. The system senses musical symbols on the surface using the rear diffused illumination technique, and displays the processed output back via a projector and speakers. The main underlying techniques are image classification based on K-nearest neighbor classifier using 9 features derived from musical symbols. The distance transform also used to find locations of musical notes in order to determine the pitches of the notes.