Publication
The Viuva Negra crawler
| Summary: | This report discusses architectural aspects of web crawlers and details the design, implementation and evaluation of the Viuva Negra (VN) crawler. VN has been used for 4 years, feeding a search engine and an archive of the Portuguese web. In our experiments it crawled over 2 million documents per day, correspondent to 63 GB of data. We describe hazardous situations to crawling found on the web and the adopted solutions to mitigate their effects. The gathered information was integrated in a web warehouse that provides support for its automatic processing by text mining applications. |
|---|---|
| Main Authors: | Gomes, Daniel |
| Other Authors: | Silva, Mário J. |
| Subject: | Crawler design tumba! web partitioning,experiments harvesting Tomba |
| Year: | 2006 |
| Country: | Portugal |
| Document type: | report |
| Access type: | open access |
| Associated institution: | Universidade de Lisboa |
| Language: | Portuguese |
| Origin: | Repositório da Universidade de Lisboa |
| Summary: | This report discusses architectural aspects of web crawlers and details the design, implementation and evaluation of the Viuva Negra (VN) crawler. VN has been used for 4 years, feeding a search engine and an archive of the Portuguese web. In our experiments it crawled over 2 million documents per day, correspondent to 63 GB of data. We describe hazardous situations to crawling found on the web and the adopted solutions to mitigate their effects. The gathered information was integrated in a web warehouse that provides support for its automatic processing by text mining applications. |
|---|
Funded activities
No funded projects found.