Awesome Data Engineering

Apache Superset is a Data Visualization and Data Exploration Platform

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

An orchestration platform for the development, production, and observation of data assets.

NoSQL data store using the Seastar framework, compatible with Apache Cassandra and Amazon DynamoDB

Incremental engine for long horizon agents 🌟 Star if you like it!

Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

lakeFS - Data version control for your data lake | Git for data

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

a full featured file system for online data storage

Source-available semantic and context layer, rule engine, and semantic sidecar for agentic AI and governed analytics. Compiles declarative YAML models into optimized SQL, KPIs, and semantic context across 8 SQL dialects.

Interactive data exploration that is extremely fast and agent-native

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

Forecast what will happen, including outcomes of your decisions.

Turn CSV, Excel, and read-only databases into verified, editable business reports.

Declarative data quality engine. Define checks in YAML, run anywhere.

A detective for your data. Zero-config data quality monitoring — works with dbt, Postgres, BigQuery, Snowflake. No YAML.

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs. The first merge library where every operation is mathematically guaranteed to converge.