The Anatomy of a Large-Scale Hypertextual Web Search Engine
Abstract
In this paper, we present Google, a prototype of a large-scale search engine
which makes heavy use of the structure present in hypertext. Google is designed
to crawl and index the Web efficiently and produce much more satisfying search
results than existing systems. The prototype with a full text and hyperlink
database of at least 24 million pages is available.
|
|
To engineer a search engine is a challenging task. Search engines index tens to
hundreds of millions of web pages involving a comparable number of distinct
terms. They answer tens of millions of queries every day. Despite the importance
of large-scale search engines on the web, very little academic research has been
done on them. Furthermore, due to rapid advance in technology and web
proliferation, creating a web search engine today is very different from three
years ago. This paper provides an in-depth description of our large-scale web
search engine -- the first such detailed public description we know of to date.
Apart from the problems of scaling traditional search techniques to data of this
magnitude, there are new technical challenges involved with using the additional
information present in hypertext to produce better search results. This paper
addresses this question of how to build a practical large-scale system which can
exploit the additional information present in hypertext. Also we look at the
problem of how to effectively deal with uncontrolled hypertext collections where
anyone can publish anything they want.
Keywords: World Wide Web, Search Engines, Information Retrieval, PageRank,
Google
Introduction
(Note: There are two versions of this paper -- a longer full version and a
shorter printed version. The full version is available on the web and the
conference CD-ROM.)
The web creates new challenges for information retrieval. The amount of
information on the web is growing rapidly, as well as the number of new users
inexperienced in the art of web research. People are likely to surf the web
using its link graph, often starting with high quality human maintained indices
such as Yahoo! or with search engines. Human maintained lists cover popular
topics effectively but are subjective, expensive to build and maintain, slow to
improve, and cannot cover all esoteric topics.
|
|
|
|
| |
