Awesome Data Engineering

The Prometheus monitoring system and time series database.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

An open-source graph database

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Simple, resilient multi-host containers networking and more.

lakeFS - Data version control for your data lake | Git for data

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Lightweight data stream processing engine for IoT edge

Open-source backend for Kepler.gl. Self-hosted alternative to CARTO and Foursquare Studio.

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.