Sensors · 2025 · 48 citations · 67 references
ComputingComputer EngineeringComputer ScienceAutomationEdge ComputingNatural Language ProcessingContext-aware ComputingData CompressionEdge Artificial IntelligenceHardware AccelerationLanguage ModelingMobile Edge ComputingCompressionCompression TechniquesIndustrial Automation SystemsKnowledge DistillationLarge-scale Natural Language ProcessingModel CompressionPower ConsumptionResource-constrained Devices
Large Language Models (LLMs), like GPT and BERT, have significantly advanced Natural Language Processing (NLP), enabling high performance on complex tasks. However, their size and computational needs make LLMs unsuitable for deployment on resource-constrained devices, where efficiency, speed, and low power consumption are critical. Tiny Language Models (TLMs), also known as BabyLMs, offer compact alternatives by using advanced compression and optimization techniques to function effectively on devices such as smartphones, Internet of Things (IoT) systems, and embedded platforms. This paper provides a comprehensive survey of TLM architectures and methodologies, including key techniques such as knowledge distillation, quantization, and pruning. Additionally, it explores potential and emerging applications of TLMs in automation and control, covering areas such as edge computing, IoT, industrial automation, and healthcare. The survey discusses challenges unique to TLMs, such as trade-offs between model size and accuracy, limited generalization, and ethical considerations in deployment. Future research directions are also proposed, focusing on hybrid compression techniques, application-specific adaptations, and context-aware TLMs optimized for hardware-specific constraints. This paper aims to serve as a foundational resource for advancing TLMs capabilities across diverse real-world applications.
67
Sinno Jialin Pan, Qiang Yang · IEEE Transactions on Knowledge and Data Engineering · 2009 · 23.2K citations
Xiang Li, Wenhai Wang, Xiaolin Hu et al. · 2019 · 3K citations
A Survey on Multi-Task Learning
Yu Zhang, Qiang Yang · IEEE Transactions on Knowledge and Data Engineering · 2021 · 2.2K citations