The Crawling Experiment

I wanted to see how efficient it is to crawl a dynamic domain of 65,000 pages without parsing a single sitemap file. By relying strictly on HTML parsing and breadth-first search (BFS) links extraction, we managed to audit the entire directory structure in record time.

Key Findings

1. Link Density: Dynamic internal link systems provide 98% coverage compared to XML sitemaps.
2. Speed Metrics: A parallelized worker setup hit a consistent 80.95 requests per second (RPS).
3. Orphan Pages: We identified about 1,200 orphan pages that were present in search consoles but completely unlinked from the main layout.

$ crawl --domain "amalalexander.com" --depth 5 --workers 10
[INFO] Crawl started...
[STATS] Crawled 65,000 pages, rps=80.95, errors=12

By removing the reliance on sitemaps, we audit what a search bot actually discovers by crawling link relationships.