The Web Lab:
A Joint Project of Cornell University and the Internet Archive

• Home
  • Overview
  • Publications
• Researchers' Area
  • Tools
  • Documentation
  • Status
  • Registration
  • Login
  • Change password
• About the Web Lab
  • Contributors
  • Contact Us

The Web Laboratory: Documentation

Data base schema

• Database schema. The main database and the smaller collections (such as the Amazon collection) use the same schema.

Data base access tools

• GetPages. This tool is used to retrieve a subset of the data in a Web Lab database, by specifying which fields to included in the data, and restrictions which are used to select which data to retrieve.
• GetCrawls. This tool shows you contents of the database's 'Crawl' table.

Internet Archive Crawler

• WebBack Crawler script This is a tool that can crawl both the live web and the Internet Archive. After crawling, the script can analyze different aspects of the crawl, as well as output metadata to a CSV file.

Analysis tools

• HeritrixWebLab. This is a customized version of the Heritrix open-source, web crawler.
• VizTool. A simple tool for interactively exploring the web graph of a selected set of seed pages.

File formats

• ARC/DAT. Raw data as received from the Internet Archive is in the ARC and DAT formats.
• TSV files. The WebLab TSV (Tab-Separated Values) file format is used by the tools that save some subset of the data from the WebLab database to a file.