Awesome Data Engineering

Apache Superset is a Data Visualization and Data Exploration Platform

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Data Apps & Dashboards for Python. No JavaScript Required.

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

An orchestration platform for the development, production, and observation of data assets.

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Utils for streaming large files (S3, HDFS, gzip, bz2...)

Container data volume manager for your Dockerized application

Python Stream Processing. A Faust fork

Python interface to Hive and Presto. 🐝

What's in your data? Extract schema, statistics and entities from datasets

a full featured file system for online data storage

Apache Cassandra® Cluster Manager (CCM) – easily create and destroy clusters on localhost

Pinterest MySQL Management Tools

A pure python HDFS client

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match

No description.

Discover, mask, and verify sensitive data in SQL databases — an auditable scan → mask → validate workflow for safe database copies.

Source-available semantic and context layer, rule engine, and semantic sidecar for agentic AI and governed analytics. Compiles declarative YAML models into optimized SQL, KPIs, and semantic context across 8 SQL dialects.

PDF extraction that audits its own output — and certifies any other extractor's, catching pages they silently dropped. Verify signed manifests offline: free, MIT, no account. 0.903 on opendataloader-bench, #2 of 8 engines. 7-tool MCP server.

Interactive data exploration that is extremely fast and agent-native

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

Run scripts, explore your genome, and get personalized explanations — all in natural dialogue.

Forecast what will happen, including outcomes of your decisions.

Turn CSV, Excel, and read-only databases into verified, editable business reports.

Reverse ETL for the code-first data stack

Trust metadata for AI agents — a stamp costs ~15 tokens, re-verifying costs 15,000. Agents stamp what they verify; the next agent runs 'akf check' and builds on it. pip install akf

Declarative data quality engine. Define checks in YAML, run anywhere.

A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

No description.

Blazingly fast data comparison tool for Python, powered by Rust. Compare massive CSV/Parquet datasets instantly.

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.

Open, audit-grade agentic data quality framework with portable industry packs

Data ingestion engine — Singer taps to Parquet in cloud buckets

Pipeline de ingestão e busca semântica de documentos para RAG — com rate limiting, circuit breaker e fila resiliente.