Awesome Data Engineering

Apache Superset is a Data Visualization and Data Exploration Platform

Protocol Buffers - Google's data interchange format

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.

Data Apps & Dashboards for Python. No JavaScript Required.

An orchestration platform for the development, production, and observation of data assets.

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

lakeFS - Data version control for your data lake | Git for data

Privacy and Security focused Segment-alternative, in Golang and React

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Get your data in RAM. Get compute close to data. Enjoy the performance.

Utils for streaming large files (S3, HDFS, gzip, bz2...)

Container data volume manager for your Dockerized application

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

High-performance time-series aggregation for PostgreSQL

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Python helpers for building dashboards using Flask and React

Python Stream Processing. A Faust fork

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Lightweight data stream processing engine for IoT edge

Python interface to Hive and Presto. 🐝

What's in your data? Extract schema, statistics and entities from datasets

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

Apache Cassandra® Cluster Manager (CCM) – easily create and destroy clusters on localhost

The Apache Kafka C/C++ library

Pinball is a scalable workflow manager

Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster

A pure python HDFS client

The Heroic Time Series Database

A distributed system designed to ingest and process time series data

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match

An experimental embedded SQL engine in C++20. Query Parquet, CSV, JSON, Arrow, Avro, SQLite, and Excel files directly with SQL, in-process. Early-stage.

Accumulo backed time series database

High-performance, time-indexed object storage for robotics and industrial IoT

Kroxylicious, the snappy open source proxy for Apache Kafka®

No description.

Data Quality and Observability platform for the whole data lifecycle, from profiling new data sources to full automation with Data Observability. Configure data quality checks from the UI or in YAML files, let DQOps run the data quality checks daily to detect data quality issues.

Discover, mask, and verify sensitive data in SQL databases — an auditable scan → mask → validate workflow for safe database copies.

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

Trust metadata for AI agents — a stamp costs ~15 tokens, re-verifying costs 15,000. Agents stamp what they verify; the next agent runs 'akf check' and builds on it. pip install akf

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

No description.

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.