MAUVE: Measuring the Gap Between Neural Text and Human Text using\n Divergence Frontiers

Abstract

As major progress is made in open-ended text generation, measuring how close\nmachine-generated text is to human language remains a critical open problem. We\nintroduce MAUVE, a comparison measure for open-ended text generation, which\ndirectly compares the learnt distribution from a text generation model to the\ndistribution of human-written text using divergence frontiers. MAUVE scales up\nto modern text generation models by computing information divergences in a\nquantized embedding space. Through an extensive empirical study on three\nopen-ended generation tasks, we find that MAUVE identifies known properties of\ngenerated text, scales naturally with model size, and correlates with human\njudgments, with fewer restrictions than existing distributional evaluation\nmetrics.\n