Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · 2022 · 21 citations · 13 references
ComputingComputer ScienceDistributed SystemsMachine LearningRecommender SystemsDeep LearningMeta-learningOptimization For Machine LearningRepresentation LearningDeep Learning For RecommendationEmbedding LearningMeta-learning For RecommendationPartition ProcedureReinforcement Learning TransferScalability BottlenecksShardingSharding Strategies
Embedding learning is an important technique in deep recommendation models to map categorical features to dense vectors. However, the embedding tables often demand an extremely large number of parameters, which become the storage and efficiency bottlenecks. Distributed training solutions have been adopted to partition the embedding tables into multiple devices. However, the embedding tables can easily lead to imbalances if not carefully partitioned. This is a significant design challenge of distributed systems named embedding table sharding, i.e., how we should partition the embedding tables to balance the costs across devices, which is a non-trivial task because 1) it is hard to efficiently and precisely measure the cost, and 2) the partition problem is known to be NP-hard. In this work, we introduce our novel practice in Meta, namely AutoShard, which uses a neural cost model to directly predict the multi-table costs and leverages deep reinforcement learning to solve the partition problem. Experimental results on an open-sourced large-scale synthetic dataset and Meta's production dataset demonstrate the superiority of AutoShard over the heuristics. Moreover, the learned policy of AutoShard can transfer to sharding tasks with various numbers of tables and different ratios of the unseen tables without any fine-tuning. Furthermore, AutoShard can efficiently shard hundreds of tables in seconds. The effectiveness, transferability, and efficiency of AutoShard make it desirable for production use. Our algorithms have been deployed in Meta production environment. A prototype is available at https://github.com/daochenzha/autoshard
13
Mastering the game of Go without human knowledge
David Silver, Julian Schrittwieser, Karen Simonyan et al. · Nature · 2017 · 9.2K citations
Engineering, Educational Psychology, Cognitive Psychology +7
Deep Neural Networks for YouTube Recommendations
Paul Covington, Jay Adams, Emre Sargin · 2016 · 3.4K citations · Full text
Wide & Deep Learning for Recommender Systems
Heng-Tze Cheng, Levent Koç, Jeremiah Harmsen et al. · 2016 · 3.3K citations · Full text
Facial Expression Analysis under Partial Occlusion
ACM Computing Surveys · 2018 · 1.2K citations · Full text