2023 · 15 citations · 16 references
ComputingComputer ScienceData ScienceMachine LearningNatural Language ProcessingText AnalyticsLanguage ModelingLanguage ModelsNatural Language UnderstandingText GenerationWord EmbeddingsActorsCarbon Footprint CalculationEmpirical EvaluationFine-tuningLanguage Model Fine-tuningLanguage Model PretrainingLarge Language ModelsLarge-scale Natural Language Processing
Despite the popularity of the pre-train then fine-tune paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of fine-tuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of fine-tuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their fine-tuning energy efficiency.
16
Random search for hyper-parameter optimization
James Bergstra, Yoshua Bengio · 2012 · 7.9K citations
Aligning artificial intelligence with climate change mitigation
Lynn H. Kaack, Priya L. Donti, Emma Strubell et al. · Nature Climate Change · 2022 · 461 citations · Full text
Sustainable AI: Environmental Implications, Challenges and Opportunities
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta et al. · arXiv (Cornell University) · 2021 · 352 citations · Full text