The use of <scp>LLMs</scp> to annotate data in management research: Foundational guidelines and warnings

Abstract

Abstract Research Summary The emergence of large language models (LLMs) offers new opportunities for AI integration in research, particularly for data annotation and text classification. However, researchers lack guidance on implementation best practices, as the benefits and risks of these tools remain poorly understood. We develop a foundational framework for effective LLM implementation in management research, providing structured guidance on key decisions throughout the research process. We illustrate this framework through an empirical application: classifying sustainability claims in crowdfunding projects. While LLMs can match or exceed traditional methods' performance at lower cost, we find that variations in prompt design can significantly affect results and downstream analyses. We develop procedures for sensitivity analysis and provide detailed documentation to help researchers implement robustness while maintaining methodological integrity. Managerial Summary Large language models (LLMs) offer powerful new tools for business research, especially for analyzing and categorizing text data. However, managers and researchers lack clear guidance on how to use these tools effectively and reliably. This study creates a practical framework for implementing LLMs in management research, covering key decisions from model selection to result validation. Using a real‐world example of analyzing sustainability claims in crowdfunding campaigns, we demonstrate that LLMs can match traditional methods while being faster and cheaper. However, small changes in how you instruct the AI can significantly alter results and business conclusions. We provide systematic procedures for testing result reliability and offer practical tools to help managers implement AI‐powered analysis while maintaining rigorous standards and avoiding misleading findings.