Awesome Data Engineering

Connecting Apache Spark with different data stores [DEPRECATED]

Data Quality and Observability platform for the whole data lifecycle, from profiling new data sources to full automation with Data Observability. Configure data quality checks from the UI or in YAML files, let DQOps run the data quality checks daily to detect data quality issues.

Discover, mask, and verify sensitive data in SQL databases — an auditable scan → mask → validate workflow for safe database copies.

Scriptella is an open source ETL (Extract-Transform-Load) and script execution tool written in Java.

INACTIVE: A PostgreSQL extension to produce messages to Apache Kafka.

Source-available semantic and context layer, rule engine, and semantic sidecar for agentic AI and governed analytics. Compiles declarative YAML models into optimized SQL, KPIs, and semantic context across 8 SQL dialects.

Intent as Code | the workflow language for AI. One file, 4 verbs, one Rust binary. Local-first, any model, AGPL-3.0. 🦋

PDF extraction that audits its own output — and certifies any other extractor's, catching pages they silently dropped. Verify signed manifests offline: free, MIT, no account. 0.903 on opendataloader-bench, #2 of 8 engines. 7-tool MCP server.

Interactive data exploration that is extremely fast and agent-native

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

Run scripts, explore your genome, and get personalized explanations — all in natural dialogue.

Forecast what will happen, including outcomes of your decisions.

A lightweight Snowflake emulator built with Go and DuckDB for local development and testing

A kafka logger for winston

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

Data policy IN, dynamic view OUT: PACE is the Policy As Code Engine. It helps you to programatically create and apply a data policy to a processing platform like Databricks, Snowflake or BigQuery (or plain 'ol Postgres, even!) with definitions imported from Collibra, Datahub, ODD and the like.

Turn CSV, Excel, and read-only databases into verified, editable business reports.

Reverse ETL for the code-first data stack

Live import all your Google Sheets to your data warehouse

Trust metadata for AI agents — a stamp costs ~15 tokens, re-verifying costs 15,000. Agents stamp what they verify; the next agent runs 'akf check' and builds on it. pip install akf

The fast, config-driven way to move data in Rust — a native CLI and an embeddable Rust ETL library

Declarative data quality engine. Define checks in YAML, run anywhere.

One interface for your whole data stack, with built-in safeguards and a simpler workflow for your team. Written in Rust. Docker not needed.

High performant ETL engine written in TypeScript

Docker microservice for saving/restoring volume data to S3

The GA Release of SnackFS

MCP server for real-time news with bias scoring, live stock/ETF/crypto data, AI options pricing, balanced news synthesis, and meme search. 10 tools, 5000+ sources, free tier.

A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

No description.

Data engineering and machine learning toolkit for Go.

Blazingly fast data comparison tool for Python, powered by Rust. Compare massive CSV/Parquet datasets instantly.

A general-purpose data analysis engine radically changing the way batch and stream data is processed

Cold storage that queries like hot storage. High-ratio JSONL compressor with block-level random access. Query cold storage in seconds — no restore, no full decompression. 5–13% ratio on JSONL logs, 25–57% smaller than gzip. 1-hour window via DuckDB in under 6s. Free for personal and open-source use.

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.

Open, audit-grade agentic data quality framework with portable industry packs

Data ingestion engine — Singer taps to Parquet in cloud buckets

No description.

Free sample: 42-table synthetic business dataset with double-entry accounting. AdventureWorks alternative. CSV + SQL + Parquet + SQLite. AU tax compliance (ATO/GST).

No description.

Pipeline de ingestão e busca semântica de documentos para RAG — com rate limiting, circuit breaker e fila resiliente.

Enrich any CSV with company data (revenue, employees, credit score) from 250M+ companies. Free ZoomInfo/Clearbit alternative. Zero dependencies.