Awesome Data Engineering

Apache Superset is a Data Visualization and Data Exploration Platform

The Prometheus monitoring system and time series database.

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Data Apps & Dashboards for Python. No JavaScript Required.

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

An orchestration platform for the development, production, and observation of data assets.

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Apache Druid: a high performance real-time analytics database.

Open-source search database for full-text, vector, and hybrid search with real-time indexing and SQL.

Incremental engine for long horizon agents 🌟 Star if you like it!

Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

lakeFS - Data version control for your data lake | Git for data

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Node.js client for Apache Kafka 0.8 and later.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

Python helpers for building dashboards using Flask and React

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

HyperDex is a scalable, searchable key-value store

A distributed system designed to ingest and process time series data

Distributed Transactional In-Memory Database (全球首个支持分布式事务的MongoDB)

Open-source backend for Kepler.gl. Self-hosted alternative to CARTO and Foursquare Studio.

An experimental embedded SQL engine in C++20. Query Parquet, CSV, JSON, Arrow, Avro, SQLite, and Excel files directly with SQL, in-process. Early-stage.

Accumulo backed time series database

No description.

One database for user-user (follows), user-item (likes), and item-item (related items) interactions — precomputed at write time, served as simple lookups

Data Quality and Observability platform for the whole data lifecycle, from profiling new data sources to full automation with Data Observability. Configure data quality checks from the UI or in YAML files, let DQOps run the data quality checks daily to detect data quality issues.

PDF extraction that audits its own output — and certifies any other extractor's, catching pages they silently dropped. Verify signed manifests offline: free, MIT, no account. 0.903 on opendataloader-bench, #2 of 8 engines. 7-tool MCP server.

A kafka logger for winston

Data policy IN, dynamic view OUT: PACE is the Policy As Code Engine. It helps you to programatically create and apply a data policy to a processing platform like Databricks, Snowflake or BigQuery (or plain 'ol Postgres, even!) with definitions imported from Collibra, Datahub, ODD and the like.

Turn CSV, Excel, and read-only databases into verified, editable business reports.

Reverse ETL for the code-first data stack

Trust metadata for AI agents — a stamp costs ~15 tokens, re-verifying costs 15,000. Agents stamp what they verify; the next agent runs 'akf check' and builds on it. pip install akf

One interface for your whole data stack, with built-in safeguards and a simpler workflow for your team. Written in Rust. Docker not needed.

High performant ETL engine written in TypeScript

Docker microservice for saving/restoring volume data to S3

MCP server for real-time news with bias scoring, live stock/ETF/crypto data, AI options pricing, balanced news synthesis, and meme search. 10 tools, 5000+ sources, free tier.

No description.

Open, audit-grade agentic data quality framework with portable industry packs

No description.

No description.

Enrich any CSV with company data (revenue, employees, credit score) from 250M+ companies. Free ZoomInfo/Clearbit alternative. Zero dependencies.