International Journal of Artificial Intelligence Tools · 2004 · 90 citations · 15 references
ComputingComputer ScienceComputational LinguisticsInformation RetrievalNatural Language ProcessingCorpus AnalysisInformation ExtractionKeyword AnalysisKeyword ExtractionTerm ExtractionText MiningAutomatic Term ExtractionCo-occurrence AnalysisCo-occurrence ModelingCorpus-based Term ExtractionSingle-word Term ExtractionStatistical Keyword Extraction
The study introduces a novel keyword extraction algorithm that operates on a single document without requiring a corpus. The algorithm first extracts frequent terms, then builds co‑occurrence distributions between each term and the frequent terms based on sentence co‑occurrence; a term is deemed a keyword if its co‑occurrence distribution is biased toward a subset of frequent terms, with bias measured by a chi‑square statistic. The algorithm achieves performance comparable to tf‑idf while not relying on a corpus.
We present a new keyword extraction algorithm that applies to a single document without using a corpus. Frequent terms are extracted first, then a set of co-occurrences between each term and the frequent terms, i.e., occurrences in the same sentences, is generated. Co-occurrence distribution shows importance of a term in the document as follows. If the probability distribution of co-occurrence between term a and the frequent terms is biased to a particular subset of frequent terms, then term a is likely to be a keyword. The degree of bias of a distribution is measured by the χ 2 -measure. Our algorithm shows comparable performance to tfidf without using a corpus.
15
An algorithm for suffix stripping
Martin Porter · Program electronic library and information systems · 1980 · 8.1K citations
Word association norms, mutual information, and lexicography
Kenneth Church, Patrick Hanks · 1990 · 3.7K citations
Accurate methods for the statistics of surprise and coincidence
Ted Dunning · 1993 · 2.7K citations