A stochastic Japanese morphological analyzer using a forward-DP backward-<i>A*</i> N-best search algorithm

Masaaki Nagata

1994 · 90 citations · 9 references

DOIFull text

Open access

Concepts

TL;DR

Japanese text lacks delimiters, making segmentation challenging, so a delimiter‑free method is needed. The study introduces a novel method for segmenting sentences into words and assigning parts of speech. The method employs a statistical language model and an efficient two‑pass N‑best search algorithm that does not require delimiters between words. The analyzer achieved 95.1 % recall and 94.6 % precision on the ATR Corpus.

Abstract

We present a novel method for segmenting the input sentence into words and assigning parts of speech to the words. It consists of a statistical language model and an efficient two-pass N-best search algorithm. The algorithm does not require delimiters between words. Thus it is suitable for written Japanese. The proposed Japanese morphological analyzer achieved 95.1% recall and 94.6% precision for open text when it was trained and tested on the ATR Corpus.

References

9