Awesome Data Engineering

Apache Superset is a Data Visualization and Data Exploration Platform

The Prometheus monitoring system and time series database.

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

An orchestration platform for the development, production, and observation of data assets.

Incremental engine for long horizon agents 🌟 Star if you like it!

Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

lakeFS - Data version control for your data lake | Git for data

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

Open-source backend for Kepler.gl. Self-hosted alternative to CARTO and Foursquare Studio.

No description.

One database for user-user (follows), user-item (likes), and item-item (related items) interactions — precomputed at write time, served as simple lookups

Trust metadata for AI agents — a stamp costs ~15 tokens, re-verifying costs 15,000. Agents stamp what they verify; the next agent runs 'akf check' and builds on it. pip install akf

One interface for your whole data stack, with built-in safeguards and a simpler workflow for your team. Written in Rust. Docker not needed.

No description.