Awesome Data Engineering

Apache Superset is a Data Visualization and Data Exploration Platform

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Data Apps & Dashboards for Python. No JavaScript Required.

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

An orchestration platform for the development, production, and observation of data assets.

A lightweight, lightning-fast, in-process vector database

NoSQL data store using the Seastar framework, compatible with Apache Cassandra and Amazon DynamoDB

Incremental engine for long horizon agents 🌟 Star if you like it!

Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

lakeFS - Data version control for your data lake | Git for data

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Utils for streaming large files (S3, HDFS, gzip, bz2...)

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

High-performance time-series aggregation for PostgreSQL

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

Python Stream Processing. A Faust fork

Python interface to Hive and Presto. 🐝

What's in your data? Extract schema, statistics and entities from datasets

a full featured file system for online data storage

Apache Cassandra® Cluster Manager (CCM) – easily create and destroy clusters on localhost

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

High-performance, time-indexed object storage for robotics and industrial IoT

No description.

Discover, mask, and verify sensitive data in SQL databases — an auditable scan → mask → validate workflow for safe database copies.

Source-available semantic and context layer, rule engine, and semantic sidecar for agentic AI and governed analytics. Compiles declarative YAML models into optimized SQL, KPIs, and semantic context across 8 SQL dialects.

PDF extraction that audits its own output — and certifies any other extractor's, catching pages they silently dropped. Verify signed manifests offline: free, MIT, no account. 0.903 on opendataloader-bench, #2 of 8 engines. 7-tool MCP server.

Interactive data exploration that is extremely fast and agent-native

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

Forecast what will happen, including outcomes of your decisions.

Turn CSV, Excel, and read-only databases into verified, editable business reports.

Reverse ETL for the code-first data stack

Trust metadata for AI agents — a stamp costs ~15 tokens, re-verifying costs 15,000. Agents stamp what they verify; the next agent runs 'akf check' and builds on it. pip install akf

Declarative data quality engine. Define checks in YAML, run anywhere.

A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

Blazingly fast data comparison tool for Python, powered by Rust. Compare massive CSV/Parquet datasets instantly.

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.

Open, audit-grade agentic data quality framework with portable industry packs

Data ingestion engine — Singer taps to Parquet in cloud buckets

Pipeline de ingestão e busca semântica de documentos para RAG — com rate limiting, circuit breaker e fila resiliente.