Awesome Data Engineering

Protocol Buffers - Google's data interchange format

The Prometheus monitoring system and time series database.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.

Analyzes resource usage and performance characteristics of running containers.

The lightweight, fault-tolerant database built on SQLite. Designed to keep your data highly available with minimal effort.

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

An open-source graph database

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Open-source search database for full-text, vector, and hybrid search with real-time indexing and SQL.

Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

Simple, resilient multi-host containers networking and more.

lakeFS - Data version control for your data lake | Git for data

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

Privacy and Security focused Segment-alternative, in Golang and React

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Lightweight data stream processing engine for IoT edge

Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption

Open-source backend for Kepler.gl. Self-hosted alternative to CARTO and Foursquare Studio.

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.

Accumulo backed time series database

One database for user-user (follows), user-item (likes), and item-item (related items) interactions — precomputed at write time, served as simple lookups

A lightweight Snowflake emulator built with Go and DuckDB for local development and testing

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

Data engineering and machine learning toolkit for Go.