Awesome Observability

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

Open-source observability for your GenAI or LLM application, based on OpenTelemetry

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

eBPF Observability - Distributed Tracing and Profiling

Nvidia GPU exporter for prometheus using nvidia-smi binary OR using NVML

Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊

OrcaReplay — Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.

Plug-and-play homelab dashboard in one container — GPU, local-AI VRAM, Docker, systemd, host health. Built-in read-only MCP server so AI agents can explore it too.

Intent as Code | the workflow language for AI. One file, 4 verbs, one Rust binary. Local-first, any model, AGPL-3.0. 🦋

Observability for AI coding assistants — cost tracking, efficiency scoring, and anti-pattern detection for Claude Code, Cursor, Copilot, and more. Local-first; New Relic optional.

Lightweight cost observability for every API call in your stack. Track LLM tokens, database ops, and external services from your terminal.

Free, read-only GPU waste scanner for Kubernetes inference clusters, by Paralleliq. Finds idle GPUs, oversized GPU tiers and wasted capacity in vLLM deployments.

Diff your AI agent's behavior between two runs. See exactly which tool calls, args, costs and outputs changed when you swap models or edit prompts.

Local-first observability dashboard for AI agents. MCP-native. Look at every span your agents emit.

Unified observability gateway for AI agents — one MCP server for Prometheus, Loki, and any backend, with cross-signal anomaly detection and a built-in Web UI.

Local replay debugger for Browser Use failures with screenshots, model I/O, failed-step timelines, and public-safe HTML exports.

Local-first DevTools for AI calls. See every LLM call your app makes: prompt, cost, tokens, latency, tool calls.

Chat-driven Prometheus visualization with one-click Grafana export. Self-hosted, single image, bring your own LLM.