A trace-based analysis of duplicate suppression in HTTP

Abstract

Many HTTP resources (pages, graphics, etc.) are exact duplicates of other resources with different URLs. If an HTTP cache contains a duplicate of a requested resource, and could detect this, it could avoid substantial network costs by returning the cached duplicate in place of the requested URL. Previous studies have shown that there is substantial duplication of content in both HTTP and FTP, and several protocols have been proposed to support efficient and safe duplicate suppression in HTTP. We use traces covering millions of HTTP requests to quantify the potential benefit of an HTTP duplicate-suppression extension. In particular, we show that the benefits vary depending on content-type, and that a small fraction of Web servers account for most of the duplicated resources.