What Makes Training Multi-Modal Classification Networks Hard?

TLDR

End‑to‑end training of multi‑modal networks is presumed to outperform uni‑modal models because they receive richer input information. This study investigates why multi‑modal networks sometimes underperform, attributing the drop to overfitting from increased capacity and to mismatched generalization rates across modalities. To address these issues, the authors introduce Gradient‑Blending, an adaptive blending strategy that weights modalities according to their overfitting behavior during joint training. Across several video classification benchmarks, the authors find that uni‑modal models can outperform multi‑modal ones, but Gradient‑Blending consistently outperforms baselines and achieves state‑of‑the‑art accuracy on human action, ego‑centric action, and acoustic event detection tasks.

Abstract

Consider end-to-end training of a multi-modal vs. a uni-modal network on a task with multiple input modalities: the multi-modal network receives more information, so it should match or outperform its uni-modal counterpart. In our experiments, however, we observe the opposite: the best uni-modal network can outperform the multi-modal network. This observation is consistent across different combinations of modalities and on different tasks and benchmarks for video classifications. This paper identifies two main causes for this performance drop: first, multi-modal networks are often prone to overfitting due to increased capacity. Second, different modalities overfit and generalize at different rates, so training them jointly with a single optimization strategy is sub-optimal. We address these two problems with a technique we call Gradient-Blending, which computes an optimal blending of modalities based on their overfitting behaviors. We demonstrate that Gradient Blending outperforms widely-used baselines for avoiding overfitting and achieves state-of-the-art accuracy on various tasks including human action recognition, ego-centric action recognition, and acoustic event detection.