spider-rs/spider
Get web data for AI agents and LLMs - fast, efficient, and reliable with Rust
SpiderSuite (web security crawler) releases, wiki and roadmap
Appears on
Quick read
Latest capture 2026-10-18 09:43
Archive status reflects the last catalog sync, not a maintenance assessment. Check recent releases, issue responses, and fit for your project with the repository shortlisting checklist.
0 paths
Agent instructions and tool configuration found in this repository.
No config files detected.
1 observed capture since 2026-10-18. Observed captures are shown by default.
Stars from first capture 0
Observed captures only
All tracked data
Observed snapshots
Observed snapshots
Nearest indexed repositories by embedding similarity.
Get web data for AI agents and LLMs - fast, efficient, and reliable with Rust
A python library to recursively crawl every sitemap.xml for a website. Also handles robots.txt and other well-knowns.
Scrapy, a fast high-level web crawling & scraping framework for Python.
SuiteCRM - Open source CRM for the world
A personal search engine: Create a searchable library from your personal documents, interests, and more!
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.