Multivariate sensor fusion by a neural network model

Abstract

This paper describes a hierarchically organized technical system performing auditory-visual sound source localization and camera control. Desired applications are in the field of mobile robotics or multimedia. The measurement set-up uses four microphones and one video camera. Starting points are functionality and signal processing in the auditory and visual pathways of the central nervous system in mammals, performed with the help of neural networks. Sound and vision estimates of an intentional target are fused in order to control a virtual fovea within the vision system. For this purpose, optical signals of the CCD of the video camera are collected in macropixels which determine the grid of foveal attention control. Pre-processed sound signals are interpreted as spike-train coded action potentials to be accumulated in the neurons soma. Spike signals which arrive approximately synchronously activate an output action potential. This enables the system to perform a correlative input selection as to be used in echo cancellation, for instance. A respective technical system is designed and implemented on an industrial mobile robot. Experimental results of the behavior of the overall system are presented.