SIFDriveNet: Speed and Image Fusion for Driving Behavior Classification Network

Abstract

Driving behavior classification is an important direction in the field of social transportation systems and advanced driving assistance system (ADAS), which has attracted more and more attention in recent years. An accurate driving behavior classification algorithm plays a great role in traffic safety, energy saving, and other fields. In this article, we propose a novel vehicle speed and image fusion for driving behavior classification network (SIFDriveNet), which classifies driver behaviors into normal driving, aggressive driving, and drowsy driving. Our method has the following key advantages. First, in the research of driving behavior classification, we are the first to introduce a 2-D image with rich roadside information and convert speeds into a 2-D spectrogram expressing the time–frequency characteristics of speeds through short-time Fourier transform (STFT) while unifying the data space of image information and speed information. Second, we propose a tensor fusion method based on weight decomposition to fully fuse the vectors of the two modalities. This method maps the tensor outer product results to the low-dimensional space through weight decomposition and has a low computational cost while maintaining the fusion effect of the tensor outer product. In addition, we evaluated our model on the public UAH-DriveSet and compared it with the most advanced model. Experimental results show that our model has a better performance, and F1-score is 97.9% on all roads. Especially on the secondary road, our F1-score is 99.4%. Also, our model has strong generalization, and we have reached 99.3% F1 in distracted driving multimodal dataset. In addition, the inference speed reaches 411 FPS, enabling real-time needs. The code is available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/alu222/SIFDriveNet</uri> .