ChatGPT’s inconsistent moral advice influences users’ judgment

TLDR

ChatGPT offers conversational search, Q&A, and advice, yet transparency alone does not ensure responsible AI use. The study aims to show that consistent moral advice from ChatGPT could improve users’ moral judgment and calls for better bot design and digital literacy training. ChatGPT’s inconsistent moral advice still influences users’ judgments, leading to corruption rather than improvement, as users underestimate its impact.

Abstract

ChatGPT is not only fun to chat with, but it also searches information, answers questions, and gives advice. With consistent moral advice, it can improve the moral judgment and decisions of users. Unfortunately, ChatGPT’s advice is not consistent. Nonetheless, it does influence users’ moral judgment, we find in an experiment, even if they know they are advised by a chatting bot, and they underestimate how much they are influenced. Thus, ChatGPT corrupts rather than improves its users’ moral judgment. While these findings call for better design of ChatGPT and similar bots, we also propose training to improve users’ digital literacy as a remedy. Transparency, however, is not sufficient to enable the responsible use of AI.