Sign in
← Back to search
github Not archived

AIMLPM/markcrawl

Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.

1 awesome list

Quick read

Stars
3
Forks
0
Open issues
10
Commits
234

Activity and growth

Latest capture 2026-09-20 03:06

Stars · last 7 days
No history
Commits · last 7 days
No history
Stars since tracking
+1
Stored snapshots
10

Classification

Topics

#ai-agents #anthropic-claude #data-extraction #gemini #ingestion-pipeline #llm #markdown-extraction #openai #pgvector #python #rag #sitemap-crawler #structured-data #supabase #vector-database #webcrawler

Metadata

Language
Python
License
MIT
Default branch
main
Created
2025-09-07
First commit
2026-04-04
Last pushed
2026-08-21
GitHub updated
2026-07-05
Last synced
2026-09-20 03:06
Stack scanned
2026-09-20 03:06
Archived
No

Archive status reflects the last catalog sync, not a maintenance assessment. Check recent releases, issue responses, and fit for your project with the repository shortlisting checklist.

Sign in to like AIMLPM/markcrawl
README View on GitHub

AI development signals

1 path

Agent instructions and tool configuration found in this repository.

Claude Code

View paths

Growth history

Tracked growth

10 observed captures since 2026-05-22. Observed captures are shown by default.

Stars from first capture +1

Chart data

Observed captures only

Time horizon

All tracked data

Custom date range

Stars history

Observed snapshots

Commits history

Observed snapshots

Similar repositories

Nearest indexed repositories by embedding similarity.

any4ai/AnyCrawl

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

3,481 stars
TypeScript 1 awesome list

firecrawl/firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

192,247 stars
TypeScript 2 awesome lists

study8677/antigravity-workspace-template

Give Claude Code, Cursor, Codex CLI a ChatGPT for your codebase. Multi-agent knowledge engine, grounded Q&A with file paths and line numbers. Works in any AI IDE.

1,294 stars
Python 3 awesome lists

Pyx-Corp/spectrawl

The unified web layer for AI agents. Search (8 engines), stealth browse, auth, and act on 24 platforms. One npm install, self-hosted.

28 stars
JavaScript 1 awesome list

0xMassi/webclaw

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

2,380 stars
Rust 2 awesome lists

amantus-ai/llm-codes

Transform developer documentation to clean Markdown

350 stars
TypeScript 0 awesome lists