Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning

TLDR

The multilayer convolutional sparse coding (ML‑CSC) model interprets CNNs as cascades of convolutional sparse layers, with the forward pass equivalent to a pursuit algorithm, yet a deeper understanding of the model remains lacking. This work aims to develop a sound pursuit algorithm for the ML‑CSC model by adopting a projection approach. The proposed method bridges matrix factorization, sparse dictionary learning, and sparse autoencoders, employing a projection‑based pursuit and analyzing these connections in detail. We establish tighter stability bounds for the pursuit, show that filter training is essential for nontrivial signals, derive an online dictionary‑learning algorithm, and demonstrate competitive unsupervised performance across several applications.

Abstract

The recently proposed multilayer convolutional sparse coding (ML-CSC) model, consisting of a cascade of convolutional sparse layers, provides a new interpretation of convolutional neural networks (CNNs). Under this framework, the forward pass in a CNN is equivalent to a pursuit algorithm aiming to estimate the nested sparse representation vectors from a given input signal. Despite having served as a pivotal connection between CNNs and sparse modeling, a deeper understanding of the ML-CSC is still lacking. In this paper, we propose a sound pursuit algorithm for the ML-CSC model by adopting a projection approach. We provide new and improved bounds on the stability of the solution of such pursuit and we analyze different practical alternatives to implement this in practice. We show that the training of the filters is essential to allow for nontrivial signals in the model, and we derive an online algorithm to learn the dictionaries from real data, effectively resulting in cascaded sparse convolutional layers. Last, but not least, we demonstrate the applicability of the ML-CSC model for several applications in an unsupervised setting, providing competitive results. Our work represents a bridge between matrix factorization, sparse dictionary learning, and sparse autoencoders, and we analyze these connections in detail.