What is Ingero?
Ingero is an eBPF-based agent that provides production-safe, kernel-level causal tracing for GPU workloads. It answers the question: "Why is my GPU training/inference slow right now?" It is designed for developers and operators debugging GPU performance issues.
How to use Ingero?
Install the single Ingero binary, then start the MCP server with ingero mcp --db ~/.ingero/ingero.db for stdio transport (Claude Code, Cursor) or ingero mcp --db ~/.ingero/ingero.db --http :8090 for HTTPS. No code changes are required. Use the provided MCP tools to diagnose performance.
Key features of Ingero
- eBPF-based causal tracing across 4 layers (CUDA API, host kernel, system context)
- Production-safe with <2% overhead and zero code changes required
- Python source line attribution (3.10/3.11/3.12) via DWARF
- SQLite storage with 10 GB rolling cap
- Kubernetes support (DaemonSet, Helm chart, pod metadata)
- Single binary, no runtime dependencies
Use cases of Ingero
- Identifying the root cause of slow GPU training or inference
- Debugging CUDA kernel launch latencies and memory bottlenecks
- Analyzing system-level events (CPU scheduling, OOM, memory pressure) affecting GPU workloads
- Running synthetic demos to test instrumentation without a GPU or root access
评论