2000 · 17 citations · 3 references
This paper presents an algorithm to extract sentences from paragraph by detecting the true sentence breaking spaces, by applying the statistical part-of-speech (POS) tagging technique to the space classification problem. The algorithm considers 2 consequent strings with a space in between each time for determining the space as whether a true sentence breaking space or not. We divided the ORCHID Thai POS tagged corpus into 10 portions for cross-validation test. The evaluation result shows that the average accuracy of space classification and break-space detection are 85.26% and 79.82% respectively and the average of false-break rate is 8.75%. Our approach also shows a significant improvement to the traditional statistical POS tagging technique. The average of POS tagging error rate reduction is as high as 11.3%.
3