Publication | Closed Access
Estimating and comparing entropies across written natural languages using PPM compression
27
Citations
0
References
2003
Year
Unknown Venue
Translation StudiesSyntactic ParsingEngineeringPpm CompressionSummary FormLarge Language ModelCorpus LinguisticsNatural LanguagesText MiningNatural Language ProcessingApplied LinguisticsSyntaxData ScienceComputational LinguisticsLanguage EngineeringGrammarUtilized Ppmd+Language StudiesMachine TranslationNatural LanguageMachine TranslationsGrammar InductionText NormalizationText ProcessingLinguistics
Summary form only given. The measurement of the entropy of written English is extended to include the following written natural languages: Arabic, Chinese, French, Japanese, Korean, Russian, and Spanish. It was observed that translations of the same document have approximately the same size when compressed even though they have widely varying uncompressed sizes. In the experiment, an efficient compression algorithm was used. It utilized PPMD+, PPMZ, and BZIP2 to compress the given texts and compare the resulting sizes. Similar experiments with machine translations were also performed. Based on the findings, it suggests that compression can be used as a tool to find poor translations. The results of these experiments, while preliminary, support the hypothesis that translation preserves information content. This analysis opens new horizons for future research concerning the relationship between compression and translation.