ICEcrawl
- give it a url for clean markdown of one page, or turn it loose to crawl an entire domain.
- fastapi in front of a redis job queue and a headless browser, with ssrf guards and robots-aware rate limiting.
- claude does schema extraction when structured data is wanted instead of text.
- ships an mcp server, so an agent can call it as a tool directly.
- 115 tests, one docker compose file, no third-party api to pay for.