A Deep Reinforcement Learning Method For Multimodal Data Fusion in Action Recognition

Abstract

At present, in the research of multimodal human action recognition, the weighted fusion method with fixed weight is widely applied in the decision level fusion of most models. In this way, the weight is usually obtained from the original experience or traversal search, which is inaccurate or has a large amount of calculation, and ignores the different representation ability of various modal data for various classes of action information. With the help of the powerful decision-making ability of deep reinforcement learning, we propose a multimodal decision-making fusion weight allocation network based on deep reinforcement learning. This letter mainly discusses the design of the model, which involves the modeling of reinforcement learning problem in action recognition, the design of neural network and the selection of problem-solving scheme. Experimental results on NTU RGB + D and HMDB51 datasets show the effectiveness of the proposed method.