Evolutionary optimization of long short-term memory neural network language model

Recurrent neural network language models (RNN-LMs) are recently proven to produce better performance than conventional N-gram based language models in various speech recognition tasks. Especially, long short-term memory recurrent neural network language models (LSTM-LMs) give superior performance for its ability to better modeling word history information. However, LSTM-LMs have complex network structure and training configurations, which are meta-parameters that need to be well tuned to achieve the state-of-the-art performance. The tuning is usually performed manually by humans, but it is not easy because it requires expert knowledge and intensive effort with many trials. In this study, we apply covariance matrix adaptation evolution strategy (CMA-ES) and automate the tuning process. CMA-ES is one of the most efficient global optimization techniques that has demonstrated superior performance in various benchmark tasks. In the experiments, the meta-parameters subject to the tuning included unit types at e...