← All awesome lists Search these repos

Awesome List

Awesome Data Engineering

A curated list of data engineering tools for software developers

igorbarinov/awesome-data-engineering #awesome #awesome-list

Open GitHub

List stars: 9,173
README repos: 148
Indexed repos: 142
List commits: 438
Forks: 1,651
Open issues: 49

Tracked list growth

GitHub stars and default-branch commits for igorbarinov/awesome-data-engineering.

Latest scan 2026-10-17 10:49

Likes history

GitHub stars

Commits history

Default branch commits

Indexed repositories

22 repos matching these filters.

Package manager: cargo

Latest repo push 2026-10-17

Browse

Filter this list

Search within Awesome Data Engineering or narrow by ecosystem and project health.

Clear 1 refinement

Search repositories

Search mode

Keyword Semantic

Tune results

The controls most people need first.

Language

Freshness

Sort

Direction

More filters Topics, generated tags, stack, files, age, archive status, and growth.

Ecosystem

GitHub topic

Generated tag

Framework or stack

Package manager

Files

Has file

Choose a suggestion or use commas to require multiple files.

Health

Minimum stars

Repository age

Uses known first-commit dates.

Archive status

AI development signals

Momentum

Unmaintained for

Commit velocity

Star growth

Reset filters

Highlighted

Open highlighted repo slot

Put your repository first

Promote a GitHub repo at the top of Awesome repository list views for 7 days.

protocolbuffers/protobuf

Protocol Buffers - Google's data interchange format

Stack

C++ Bundler Cargo CMake

GitHub topics

#marshalling #protobuf #protobuf-runtime #protoc #protocol-buffers #protocol-compiler

Updated: 2026-10-17
Lists: 1 list mention
First commit: 2008-07-10
License: NOASSERTION
Issues: 369 open

72,115

stars

Forks: 16,314
Commits: 24,430 commits
Star growth, last 7 days: +12 0.0%
Commit velocity, last 7 days: +104 +0.4%

Website GitHub

pathwaycom/pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Stack

Python Axum FastAPI LangChain pytest Cargo npm PEP 517

GitHub topics

#batch-processing #data-analytics #data-pipelines #data-processing #dataflow #etl

Updated: 2026-10-16
Lists: 5 list mentions
First commit: 2023-07-21
License: NOASSERTION
Issues: 37 open

62,113

stars

Forks: 1,683
Commits: 2,312 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

influxdata/influxdb

Scalable datastore for metrics, events, and real-time analytics

AI dev

Stack

Rust Axum Cargo

GitHub topics

#database #go #influxdb #metrics #monitoring #react

Updated: 2026-10-15
Lists: 3 list mentions
First commit: 2013-04-11
License: Apache-2.0
Issues: 2,179 open

31,766

stars

Forks: 3,722
Commits: 49,927 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

scylladb/scylladb

NoSQL data store using the Seastar framework, compatible with Apache Cassandra and Amazon DynamoDB

AI dev

Stack

C++ pytest Starlette Cargo CMake uv

GitHub topics

#c-plus-plus #cassandra #cpp #database #nosql #scylla

Updated: 2026-10-16
Lists: 1 list mention
First commit: 2014-08-10
License: NOASSERTION
Issues: 3,825 open

15,790

stars

Forks: 1,526
Commits: 57,503 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

manticoresoftware/manticoresearch

Open-source search database for full-text, vector, and hybrid search with real-time indexing and SQL.

Stack

C++ Cargo CMake Go modules

GitHub topics

#api #bm25 #cpp #database #full-text-search #hacktoberfest

Updated: 2026-10-16
Lists: 4 list mentions
First commit: 2003-11-19
License: GPL-3.0
Issues: 652 open

12,057

stars

Forks: 647
Commits: 14,097 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

cocoindex-io/cocoindex

Incremental engine for long horizon agents 🌟 Star if you like it!

AI dev

Stack

Rust Astro Axum FastAPI pytest Cargo npm PEP 517

GitHub topics

#agentic-data-framework #ai #ai-agents #change-data-capture #codebase-intelligence #context-engineering

Updated: 2026-10-16
Lists: 3 list mentions
First commit: 2025-03-03
License: Apache-2.0
Issues: 103 open

11,710

stars

Forks: 919
Commits: 2,168 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

xberg-io/xberg

Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

AI dev

Stack

Rust Android Astro Axum pytest Bundler Cargo Composer

GitHub topics

#bun #csharp #document-intelligence #elixir #ffi #golang

Updated: 2026-10-16
Lists: 3 list mentions
First commit: 2025-01-31
License: MIT
Issues: 14 open

9,422

stars

Forks: 594
Commits: 10,366 commits
Star growth, last 7 days: +22 +0.2%
Commit velocity, last 7 days: +108 +1.1%

Website GitHub

kreuzberg-dev/kreuzberg

A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

Stack

Rust Android Axum pytest Shelf Bundler Cargo Composer

GitHub topics

#bun #csharp #document-intelligence #elixir #ffi #golang

Updated: 2026-06-23
Lists: 3 list mentions
First commit: 2025-01-31
License: NOASSERTION
Issues: 13 open

8,536

stars

Forks: 502
Commits: 6,280 commits
Star growth, last 7 days: No 7-day history
Commit velocity, last 7 days: No 7-day history

Website GitHub

treeverse/lakeFS

lakeFS - Data version control for your data lake | Git for data

Stack

Go Android Cobra Gin gRPC Go Bundler Cargo Go modules

GitHub topics

#apache-spark #apache-sparksql #aws-s3 #azure-blob-storage #azure-storage #data-engineering

Updated: 2026-10-07
Lists: 2 list mentions
First commit: 2019-09-12
License: NOASSERTION
Issues: 450 open

5,557

stars

Forks: 485
Commits: 6,569 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

zombodb/zombodb

Making Postgres and Elasticsearch work together like it's 2023

Archived

Stack

PLpgSQL Cargo

GitHub topics

#elasticsearch #postgres #postgresql #sql #text-search

Updated: 2025-03-01
Lists: 2 list mentions
First commit: 2015-07-17
License: NOASSERTION
Issues: 17 open

4,719

stars

Forks: 212
Commits: 1,653 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

apache/gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

AI dev

Stack

Java pytest Starlette Cargo Gradle PEP 517

GitHub topics

#ai-catalog #data-catalog #datalake #federated-query #lakehouse #metadata

Updated: 2026-10-16
Lists: 1 list mention
First commit: 2023-04-23
License: Apache-2.0
Issues: 1,099 open

3,249

stars

Forks: 957
Commits: 4,671 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

slothflowlabs/duckle

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

Stack

Rust Axum React Tauri Vite Cargo npm PEP 517

GitHub topics

#cdc #connectors #data-engineering #data-integration #data-orchestration #data-pipeline

Updated: 2026-10-14
Lists: 3 list mentions
First commit: 2026-05-21
License: Apache-2.0
Issues: 73 open

1,508

stars

Forks: 120
Commits: 1,751 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

filodb/FiloDB

Distributed Prometheus time series database

Stack

Scala Cargo

GitHub topics

#database #distributed #kafka #metrics #prometheus #reactive

Updated: 2026-10-13
Lists: 1 list mention
First commit: 2015-01-14
License: Apache-2.0
Issues: 72 open

1,468

stars

Forks: 239
Commits: 3,147 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

GitHub

ModernRelay/omnigraph

Object-storage native graph database with branching and typed ontology

AI dev

Stack

Rust Axum Cargo

GitHub topics

#apache-arrow #context-graph #datafusion #graph-database #knowledge-graph #lakehouse

Updated: 2026-10-16
Lists: 1 list mention
First commit: 2026-04-10
License: MIT
Issues: 117 open

1,320

stars

Forks: 249
Commits: 1,327 commits
Star growth, last 7 days: +48 +3.8%
Commit velocity, last 7 days: +63 +5.0%

Website GitHub

reductstore/reductstore

High-performance, time-indexed object storage for robotics and industrial IoT

AI dev

Stack

Rust Axum pytest Cargo pip

GitHub topics

#ai #big-data #bigdata #blob-storage #edge-computing #etl

Updated: 2026-10-16
Lists: 1 list mention
First commit: 2021-12-23
License: Apache-2.0
Issues: 15 open

373

stars

Forks: 48
Commits: 1,639 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

supernovae-st/nika

Intent as Code | the workflow language for AI. One file, 4 verbs, one Rust binary. Local-first, any model, AGPL-3.0. 🦋

AI dev

Stack

Rust Cargo

GitHub topics

#agents #ai #ai-agents #ai-workflows #automation #cli

Updated: 2026-10-15
Lists: 5 list mentions
First commit: 2026-04-13
License: AGPL-3.0
Issues: 152 open

stars

Forks: 5
Commits: 2,525 commits
Star growth, last 7 days: -1 -1.1%
Commit velocity, last 7 days: +3 +0.1%

Website GitHub

flex-analytics/flexviz

Interactive data exploration that is extremely fast and agent-native

AI dev

Stack

Python FastAPI pytest Starlette Cargo PEP 517 uv

GitHub topics

#agent #ai-agents #big-data #dashboard #data-exploration #data-visualization

Updated: 2026-10-13
Lists: 1 list mention
First commit: 2026-08-21
License: Apache-2.0
Issues: 44 open

stars

Forks: 2
Commits: 583 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

chrislusf/seaweedfs

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

Stack

Go Cobra Gin gRPC Go Cargo Go modules Maven

Updated: 2026-10-16
Lists: 3 list mentions
First commit: 2011-11-30
License: Apache-2.0
Issues: Disabled

stars

Forks: 7
Commits: 12,815 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

MoonMao42/ReceiptBI

Turn CSV, Excel, and read-only databases into verified, editable business reports.

Stack

Python Electron FastAPI Next.js pytest Cargo npm PEP 517

GitHub topics

#business-intelligence #chatbi #csv #data-analysis #data-visualization #desktop-app

Updated: 2026-10-16
Lists: 1 list mention
First commit: 2025-12-12
License: MIT
Issues: 0 open

stars

Forks: 0
Commits: 316 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

GitHub

faucet-hq/faucet-stream

The fast, config-driven way to move data in Rust — a native CLI and an embeddable Rust ETL library

AI dev

Stack

Rust Axum Cargo

GitHub topics

#cdc #cli #connectors #data-engineering #data-integration #data-pipeline

Updated: 2026-10-16
Lists: 2 list mentions
First commit: 2026-03-23
License: NOASSERTION
Issues: 39 open

stars

Forks: 8
Commits: 509 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

wordbricks/onequery

One interface for your whole data stack, with built-in safeguards and a simpler workflow for your team. Written in Rust. Docker not needed.

AI dev

Stack

TypeScript Astro React Tailwind CSS Vite Bun Cargo npm

GitHub topics

#agent #ai #ai-agents #anthropic #chatgpt #claude

Updated: 2026-07-30
Lists: 1 list mention
First commit: 2026-03-30
License: Apache-2.0
Issues: 5 open

stars

Forks: 2
Commits: 310 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

Website GitHub

godalida/koala-diff

Blazingly fast data comparison tool for Python, powered by Rust. Compare massive CSV/Parquet datasets instantly.

Stack

Python pytest Cargo PEP 517

GitHub topics

#csv #data-engineering #data-quality #diff #high-performance #parquet

Updated: 2026-02-25
Lists: 1 list mention
First commit: 2026-02-13
License: MIT
Issues: 0 open

stars

Forks: 0
Commits: 33 commits
Star growth, last 7 days: 0 0.0%
Commit velocity, last 7 days: 0 0.0%

GitHub

Activity

Default branch: master
Last pushed: 2026-09-07
GitHub updated: 2026-10-17
Created: 2015-06-18
First commit: 2015-06-18
Last scanned: 2026-10-17 10:49
Watchers: 252

Indexed repo mix

Repo stars: 875,040
Repo forks: 139,169
Active: 120
Archived: 22

Languages

Python (39) Go (21) Java (20) C++ (10) JavaScript (9) Rust (9) Scala (7) C (5) TypeScript (5) Shell (3) Kotlin (2) Ruby (2)

Awesome Data Engineering

Tracked list growth

Likes history

Commits history

Indexed repositories

Filter this list

Put your repository first

How it works

Pricing