Knowledge-Acquisition Interfaces for Domain Experts: An Empirical Evaluation of Protégé-2000

Abstract

Application experts need to be able to maintain, populate, and verify existing knowledge bases in order to use knowledge-based tools to perform their daily tasks. Protg-2000 is a tool that enables experts to perform the knowledgeacquisition and maintenance process. In this paper, we describe a controlled study that we performed to measure both the efficiency and quality of knowledge acquisition by domain experts and to evaluate how well the domain experts can retain their proficiency in using a knowledge-acquisition tool. We adopted the techniques that are used to evaluate usability of traditional software products to evaluating knowledge-acquisition process. Using Protg-2000 and its domainspecific extensions, the domain experts with no previous experience with knowledge-based systems were able to perform complex knowledge-acquisition tasks efficiently and correctly. Our data show that subject-matter experts were able to use both tools after only 1-2 hours of training, and entered knowledge correctly overall 94% of the time. The custom-developed user interface greatly enhanced the ability of experts to identify known errors in a knowledge base (93% vs. 48% detection). The experts also were able to retain their skills over 1-2 week period of time without using the system.