Fast Online Training with Frequency-Adaptive Learning Rates for Chinese Word Segmentation and New Word Detection

TLDR

Training Chinese word segmentation on large-scale datasets is costly, and adding high-dimensional features further slows training. The study proposes a joint model for Chinese word segmentation and new word detection, and introduces an adaptive online gradient descent method that uses feature frequency to enable very fast training on large-scale, high-dimensional data. The method employs high-dimensional word-based and enriched edge features in a joint model, and uses adaptive online gradient descent driven by feature frequency to train parameters rapidly. The adaptive online gradient descent method is an order of magnitude faster than existing training methods while achieving equal or higher accuracies, and it is a general-purpose optimization technique applicable beyond Chinese word segmentation.

Abstract

We present a joint model for Chinese word segmentation and new word detection. We present high dimensional new features, including word-based features and enriched edge (label-transition) features, for the joint modeling. As we know, training a word segmentation system on large-scale datasets is already costly. In our case, adding high dimensional new features will further slow down the training speed. To solve this problem, we propose a new training method, adaptive online gradient descent based on feature frequency information, for very fast online training of the parameters, even given large-scale datasets with high dimensional features. Compared with existing training methods, our training method is an order magnitude faster in terms of training time, and can achieve equal or even higher accuracies. The proposed fast training method is a general purpose optimization method, and it is not limited in the specific task discussed in this paper.