Generating (Factual?) Narrative Summaries of RCTs: Experiments with\n Neural Multi-Document Summarization

Abstract

We consider the problem of automatically generating a narrative biomedical\nevidence summary from multiple trial reports. We evaluate modern neural models\nfor abstractive summarization of relevant article abstracts from systematic\nreviews previously conducted by members of the Cochrane collaboration, using\nthe authors conclusions section of the review abstract as our target. We enlist\nmedical professionals to evaluate generated summaries, and we find that modern\nsummarization systems yield consistently fluent and relevant synopses, but that\nthey are not always factual. We propose new approaches that capitalize on\ndomain-specific models to inform summarization, e.g., by explicitly demarcating\nsnippets of inputs that convey key findings, and emphasizing the reports of\nlarge and high-quality trials. We find that these strategies modestly improve\nthe factual accuracy of generated summaries. Finally, we propose a new method\nfor automatically evaluating the factuality of generated narrative evidence\nsyntheses using models that infer the directionality of reported findings.\n