Multi-modal tracking of faces for video communications

TLDR

Face detection and tracking are essential for efficient video compression and transmission. The system employs a supervisory architecture that cyclically activates visual processes—blink detection, color‑histogram matching, and cross‑correlation—guided by confidence factors, fuses observations via covariance estimation into a Kalman‑filter‑based recursive estimator, and drives a PD controller for a pan/tilt/zoom camera, with state transitions triggered by process‑detected events. The resulting system delivers robust, precise tracking at approximately 20 images per second on a 150 MHz workstation.

Abstract

Visual processes to detect and track faces for video compression and transmission. The system is based on an architecture in which a supervisor selects and activates visual processes in cyclic manner. Control of visual processes is made possible by a confidence factor which accompanies each observation. Fusion of results into a unified estimation for tracking is made possible by estimating a covariance matrix with each observation. Visual processes for face tracking are described using blink detection, normalised color histogram matching, and cross correlation (SSD and NCC). Ensembles of visual processes are organised into processing states so as to provide robust tracking. Transition between states is determined by events detected by processes. The result of face detection is fed into recursive estimator (Kalman filter). The output from the estimator drives a PD controller for a pan/tilt/zoom camera. The resulting system provides robust and precise tracking which operates continuously at approximately 20 images per second on a 150 megahertz computer workstation.