ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training

Abstract

This paper presents a new sequence-tosequence pre-training model called Prophet-Net, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism.