TY  - JOUR
AU  - Cho, Junghoo
AU  - Garcia-Molina, Hector
AU  - Haveliwala, Taher
AU  - Lam, Wang
AU  - Paepcke, Andreas
AU  - Raghavan, Sriram
AU  - Wesley, Gary
T1  - Stanford WebBase components and applications
JO  - ACM Transactions on Internet Technology
PY  - 2006/05
VL  - 6
IS  - 2
SP  - 153
EP  - 186
UR  - http://doi.acm.org/10.1145/1149121.1149124
DO  - 10.1145/1149121.1149124
KW  - webbase
KW  - science
KW  - analysis
KW  - archive
KW  - search
KW  - stream
KW  - research
KW  - crawling
KW  - web
L1  - 
SN  - 
N1  - 
N1  - 
AB  - We describe the design and performance of WebBase, a tool for Web research. The system includes a highly customizable crawler, a repository for collected Web pages, an indexer for both text and link-related page features, and a high-speed content distribution facility. The distribution module enables researchers world-wide to retrieve pages from WebBase, and stream them across the Internet at high speed. The advantage for the researchers is that they need not all crawl the Web before beginning their research. WebBase has been used by scores of research and teaching organizations world-wide, mostly for investigations into Web topology and linguistic content analysis. After describing the system's architecture, we explain our engineering decisions for each of the WebBase components, and present respective performance measurements.
ER  -