Web Scale Discovery: What and Why?

Abstract

Abstract Web scale discovery services for the library environment have the capacity to more easily connect researchers with the library's vast information repository. This includes locally held and hosted content such as physical holdings, digital collections, and local institutional repositories. Perhaps more significantly, web scale discovery also accesses a huge array of remotely hosted content often purchased or licensed by the library, such as publisher and aggregator content for tens of thousands of full-text journals, additional content from abstracting and indexing resources, and content from open access repositories. This chapter defines web scale discovery and highlights a few key concepts essential for understanding these services. For anyone who has worked a reference interview and heard a student utter, I couldn't find an article in the library web scale discovery services hold tremendous potential. Extensive research on user expectations in the discovery arena, and the tools used by those seeking information--tools often disassociated from the library and often overlooking much of what the library holds and licenses--provide ample rationale for why web scale discovery is important for the library environment. ********** What Is Web Scale Discovery? Connecting users with the information they seek is one of the central pillars of our profession. Succinctly put, Web scale discovery can be considered as deep discovery within a vast ocean of content. The mechanics behind Web scale discovery are not necessarily new, though a commercial application of this approach within the library environment--efficiently and, it's hoped, effectively--is very new. While there are various approaches to Web scale discovery, this issue of Library Technology Reports will focus on what today appears to be the most common approach, which, at its heart, involves huge, centralized, preaggregated indexes searched by the end user. Expanded further, Web scale discovery is--or certainly holds the potential to be--the evolution that libraries have long sought for information discovery. As information professionals, we all have at least a general awareness of the evolution of discovery tools within the library context. Such tools initially were print-based, such as bound handwritten catalogs, the card catalog, and works such as Poole's Index to Periodical Literature and the Reader's Guide to Periodical Literature. For the past several generations, such tools gradually transferred into the automated, electronic realm, with an obvious example the development and evolution of the online integrated library system (ILS) with a front-end catalog accessible to librarians and end users. These catalogs were initially available within the local library's physical building, often through a menu-driven, text-based interface. With the development in the 1990s of the Web's physical infrastructure and the empowerment of many to access this new environment, library vendors created HTTP Web-based online catalogs. Other evolutions included early pioneers in broad online information systems, including Dialog and LexisNexis. The 1990s ushered in growth in publisher-based electronic journal content, e-text and e-book content, abstracting and indexing databases, and full-text content aggregators looking to pull related information together within an easily accessible and searchable electronic medium. Many of these services or products were initially provided on CD-ROM or through text-based, menu-driven networked systems, which all eventually evolved into information search and delivery through a Web-based environment. During this time--we'll say the mid-1990s--many of us witnessed (as we still witness) the confusion suffered by many end users faced with the choice of myriad information systems with their myriad interfaces and specialized bodies of content. Many of us may remember, from around 1998, a new search engine developed by two Stanford graduate students. …