Mathematics · 2022 · 48 citations · 44 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningOptimizationDeep LearningGlobal OptimizationHeuristic SearchMetaheuristic OptimizationNeural NetworksOptimization For Machine LearningGlobal Search AlgorithmsHyperparameter OptimizationHyperparameter TuningOptimization AlgorithmsParticle Swarm OptimizationSearch Efficiency
Deep learning has been widely used in different fields such as computer vision and speech processing. The performance of deep learning algorithms is greatly affected by their hyperparameters. For complex machine learning models such as deep neural networks, it is difficult to determine their hyperparameters. In addition, existing hyperparameter optimization algorithms easily converge to a local optimal solution. This paper proposes a method for hyperparameter optimization that combines the Sparrow Search Algorithm and Particle Swarm Optimization, called the Hybrid Sparrow Search Algorithm. This method takes advantages of avoiding the local optimal solution in the Sparrow Search Algorithm and the search efficiency of Particle Swarm Optimization to achieve global optimization. Experiments verified the proposed algorithm in simple and complex networks. The results show that the Hybrid Sparrow Search Algorithm has the strong global search capability to avoid local optimal solutions and satisfactory search efficiency in both low and high-dimensional spaces. The proposed method provides a new solution for hyperparameter optimization problems in deep learning models.
44
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
Random search for hyper-parameter optimization
James Bergstra, Yoshua Bengio · 2012 · 7.9K citations
Cuckoo search: recent advances and applications
Xin‐She Yang, Suash Deb · Neural Computing and Applications · 2013 · 1.1K citations · Full text