Sign in
← Back to search
github Not archived

huggingface/datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

2 awesome lists

Quick read

Stars
3,327
Forks
299
Open issues
103
Commits
738

Activity and growth

Latest capture 2026-09-09 03:07

Stars · last 7 days
No history
Commits · last 7 days
No history
Stars since tracking
+261
Stored snapshots
7

Classification

Technology stack

Flask pytest

Metadata

Language
Python
License
Apache-2.0
Default branch
main
Created
2023-06-14
First commit
2023-06-14
Last pushed
2026-08-13
GitHub updated
2026-09-08
Last synced
2026-09-09 03:07
Stack scanned
2026-09-09 03:07
Archived
No

Archive status reflects the last catalog sync, not a maintenance assessment. Check recent releases, issue responses, and fit for your project with the repository shortlisting checklist.

Sign in to like huggingface/datatrove
README View on GitHub

AI development signals

1 path

Agent instructions and tool configuration found in this repository.

Agent instructions

View paths

Growth history

Tracked growth

7 observed captures since 2026-05-25. Observed captures are shown by default.

Stars from first capture +261

Chart data

Observed captures only

Time horizon

All tracked data

Custom date range

Stars history

Observed snapshots

Commits history

Observed snapshots

Similar repositories

Nearest indexed repositories by embedding similarity.

slothflowlabs/duckle

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

1,508 stars
Rust 3 awesome lists

RUC-NLPIR/FlashRAG

⚡FlashRAG: A Python Toolkit for Efficient RAG Research (WWW2025 Resource)

3,566 stars
Python 1 awesome list

PKU-YuanGroup/Helios

Helios: Real Real-Time Long Video Generation Model

2,146 stars
Python 1 awesome list

dotflow-io/dotflow

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

10 stars
Python 1 awesome list

PacktPublishing/LLM-Engineers-Handbook

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

5,317 stars
Python 1 awesome list

melih-unsal/DemoGPT

🤖 Create agentic apps in a second with your prompts. Everything you need to create an LLM Agent - tools, prompts, frameworks, and models - all in one place.

1,906 stars
Python 3 awesome lists