Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks

TLDR

Factorization Machines extend linear regression by modeling second‑order feature interactions, but treating all interactions equally can introduce noise from uninformative features and degrade performance. This study aims to enhance FM by distinguishing the importance of individual feature interactions. The authors introduce Attentional Factorization Machine, which assigns interaction weights learned through a neural attention network. Experiments on two real‑world datasets demonstrate that AFM improves regression accuracy by 8.6% over FM and consistently outperforms Wide&Deep and DeepCross with a simpler structure and fewer parameters, and the implementation is publicly available.

Abstract

Factorization Machines (FMs) are a supervised learning approach that enhances the linear regression model by incorporating the second-order feature interactions. Despite effectiveness, FM can be hindered by its modelling of all feature interactions with the same weight, as not all feature interactions are equally useful and predictive. For example, the interactions with useless features may even introduce noises and adversely degrade the performance. In this work, we improve FM by discriminating the importance of different feature interactions. We propose a novel model named Attentional Factorization Machine (AFM), which learns the importance of each feature interaction from data via a neural attention network. Extensive experiments on two real-world datasets demonstrate the effectiveness of AFM. Empirically, it is shown on regression task AFM betters FM with a 8.6% relative improvement, and consistently outperforms the state-of-the-art deep learning methods Wide&Deep [Cheng et al., 2016] and DeepCross [Shan et al., 2016] with a much simpler structure and fewer model parameters. Our implementation of AFM is publicly available at: https://github.com/hexiangnan/attentional_factorization_machine