Awesome Data Engineering

Python Stream Processing. A Faust fork

Secor is a service implementing Kafka log persistence

A large-scale entity and relation database supporting aggregation of properties

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Fast scalable time series database

Lightweight data stream processing engine for IoT edge

🦎 A lightweight, multi-protocol gateway for event-driven applications and AI agents. Expose and govern Kafka, MQTT, APIs, and MCP through one high-performance engine with shared routing, security, schema, and observability.

Python interface to Hive and Presto. 🐝

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

What's in your data? Extract schema, statistics and entities from datasets

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

Distributed Prometheus time series database

HyperDex is a scalable, searchable key-value store

Object-storage native graph database with branching and typed ontology

a full featured file system for online data storage

Apache Cassandra® Cluster Manager (CCM) – easily create and destroy clusters on localhost

The Apache Kafka C/C++ library

Pinball is a scalable workflow manager

Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster

Pinterest MySQL Management Tools

A pure python HDFS client

The Heroic Time Series Database

Time-series database

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

See gitlab: https://gitlab.com/Project-FiFo/DalmatinerDB/dalmatinerdb

Package golang service into minimal docker containers.

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption

A distributed system designed to ingest and process time series data

IonDB, a key-value datastore for resource constrained systems.

Distributed Transactional In-Memory Database (全球首个支持分布式事务的MongoDB)

Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

KafkaT-ool

Open-source backend for Kepler.gl. Self-hosted alternative to CARTO and Foursquare Studio.

An experimental embedded SQL engine in C++20. Query Parquet, CSV, JSON, Arrow, Avro, SQLite, and Excel files directly with SQL, in-process. Early-stage.

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.

Docker composition tool with idempotency features for deploying apps composed of multiple containers.

Accumulo backed time series database

High-performance, time-indexed object storage for robotics and industrial IoT

A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.

Kroxylicious, the snappy open source proxy for Apache Kafka®

No description.

Mirror of Apache Hivemall (incubating)

Kyoto Tycoon key-value store (and the underlying Kyoto Cabinet library)

Docker image for Logstash 1.4

One database for user-user (follows), user-item (likes), and item-item (related items) interactions — precomputed at write time, served as simple lookups

A lightweight tool for easy deployment and rollback of dockerized applications.