Hyperparameter Setting of LSTM-based Language Model using Grey Wolf Optimizer

Abstract

Hyperparameters is one of the most essential part of deep learning, because they can give big impact to the performance of the model. Recent works show that if the hyperparameters of a Long Short-Term Memory (LSTM) are carefully adjusted or optimized, its performance can achieve the same performance as the more complex LSTM model. Previously, several methods such as grid search and random search is introduced to solve this hyperparameters optimization problem, but it is still not effective enough. Hence, it opens opportunities for meta-heuristic nature-inspired approach like Swarm Intelligence (SI) method to solve this hyperparameter optimization problem. The main advantage of this method is the behaviour of the algorithm that has exploring-exploiting process in order to find the global optima solution in the search space. Algorithm such as Grey Wolf Optimizer (GWO) are one of the SI algorithms that have a promising performance in optimization problem in various field. The algorithm has balanced exploring-exploiting process, that can make the optimization are more effective. Therefore, in this paper the GWO is exploited to optimize the LSTM hyperparameters for a language modeling task. Evaluation for the Penn Tree Bank dataset shows that GWO is capable of giving an optimum hyperparameters of the LSTM.