Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding

TLDR

Slot filling is a key challenge in spoken language understanding. The study investigates applying recurrent neural networks to the slot‑filling task. We implemented and compared Elman‑ and Jordan‑type recurrent networks and their variants on the Theano toolkit, evaluated them on the ATIS benchmark, and benchmarked against a CRF baseline. Both RNN types substantially outperform the CRF baseline, with a bi‑directional Jordan network achieving a 14 % relative error reduction.

Abstract

One of the key problems in spoken language understanding (SLU) is the task of slot filling. In light of the recent success of applying deep neural network technologies in domain detection and intent identification, we carried out an in-depth investigation on the use of recurrent neural networks for the more difficult task of slot filling involving sequence discrimination. In this work, we implemented and compared several important recurrent-neural-network architectures, including the Elman-type and Jordan-type recurrent networks and their variants. To make the results easy to reproduce and compare, we implemented these networks on the common Theano neural network toolkit, and evaluated them on the ATIS benchmark. We also compared our results to a conditional random fields (CRF) baseline. Our results show that on this task, both types of recurrent networks outperform the CRF baseline substantially, and a bi-directional Jordantype network that takes into account both past and future dependencies among slots works best, outperforming a CRFbased baseline by 14% in relative error reduction.