Reward Modeling for Mitigating Toxicity in Transformer-based Language\n Models

Abstract

Transformer-based language models are able to generate fluent text and be\nefficiently adapted across various natural language generation tasks. However,\nlanguage models that are pretrained on large unlabeled web text corpora have\nbeen shown to suffer from degenerating toxic content and social bias behaviors,\nconsequently hindering their safe deployment. Various detoxification methods\nwere proposed to mitigate the language model's toxicity; however, these methods\nstruggled to detoxify language models when conditioned on prompts that contain\nspecific social identities related to gender, race, or religion. In this study,\nwe propose Reinforce-Detoxify; A reinforcement learning-based method for\nmitigating toxicity in language models. We address the challenge of safety in\nlanguage models and propose a new reward model that is able to detect toxic\ncontent and mitigate unintended bias towards social identities in toxicity\nprediction. The experiments demonstrate that the Reinforce-Detoxify method for\nlanguage model detoxification outperforms existing detoxification approaches in\nautomatic evaluation metrics, indicating the ability of our approach in\nlanguage model detoxification and less prone to unintended bias toward social\nidentities in generated content.\n