Awesome Data Engineering

Apache Superset is a Data Visualization and Data Exploration Platform

Protocol Buffers - Google's data interchange format

The Prometheus monitoring system and time series database.

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.

Scalable datastore for metrics, events, and real-time analytics

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Data Apps & Dashboards for Python. No JavaScript Required.

Analyzes resource usage and performance characteristics of running containers.

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

The lightweight, fault-tolerant database built on SQLite. Designed to keep your data highly available with minimal effort.

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

An orchestration platform for the development, production, and observation of data assets.

A lightweight, lightning-fast, in-process vector database

NoSQL data store using the Seastar framework, compatible with Apache Cassandra and Amazon DynamoDB

An open-source graph database

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Apache Druid: a high performance real-time analytics database.

Open-source search database for full-text, vector, and hybrid search with real-time indexing and SQL.

CMAK is a tool for managing Apache Kafka clusters

Incremental engine for long horizon agents 🌟 Star if you like it!

Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

Dockerfile for Apache Kafka

A fast compressor/decompressor

Simple, resilient multi-host containers networking and more.

Java binary serialization and cloning: fast, efficient, automatic

Generic command line non-JVM Apache Kafka producer and consumer

lakeFS - Data version control for your data lake | Git for data

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

A scalable, distributed Time Series Database.

Making Postgres and Elasticsearch work together like it's 2023

Privacy and Security focused Segment-alternative, in Golang and React

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Get your data in RAM. Get compute close to data. Enjoy the performance.

Utils for streaming large files (S3, HDFS, gzip, bz2...)

DEPRECATED: Data collection and processing made easy.

Container data volume manager for your Dockerized application

A distributed, fault-tolerant graph database

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

JDBC importer for Elasticsearch *** THIS REPOSITORY WILL BE DELETED WITHOUT ANY FURTHER NOTICE AFTER AUG 1 2026 ***

High-performance time-series aggregation for PostgreSQL

Node.js client for Apache Kafka 0.8 and later.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Python helpers for building dashboards using Flask and React