Aegis
An OpenAI-compatible reverse proxy that sits in front of your LLM providers and adds what a raw API call doesn't have: cost-aware routing, automatic failover, rate limits that survive a restart, prompt experimentation, and a regression suite that tells you exactly what broke.
Thirteen capabilities, each graded honestly below — not a page claiming uniform completeness. 58 automated tests. Try it live at the bottom of this page.
The Problem
A direct call to an LLM provider is a single point of failure with no cost discipline. One 429 or 503 halts every downstream workflow. Every prompt defaults to the most expensive model regardless of how simple the question is. Nobody can say which prompt version is actually better, or whether last week's change quietly made outputs worse.
The Gateway
Aegis sits between your app and every provider. It routes on complexity, fails over on outage, enforces budgets per team, runs prompt variants against each other with real statistics, and keeps a golden-case suite that catches regressions the moment they happen — all wired into one request path, not five disconnected demos.
Two independent paths, one control plane
A chat completion runs the full pipeline on the left. The four applied spokes on the right are separate endpoints — they don't route through it. Click any box for what it actually does.
Click any box for exactly what it does — this is the real request flow, not an illustration of it.
13 capabilities, graded the way I'd want a vendor to grade themselves
REAL means wired, tested, and live-verified. REAL (heuristic) means the same, with a specific, named simplification instead of a hidden one.
Golden-case harness diffs pass/fail against the last run — flags exactly which case regressed.
Slack alerts log-only unless a webhook is configured.
Routes simple prompts to cheap models, complex ones to reasoning-tier models, by real content analysis.
Full per-hop span waterfall, fetchable by trace ID.
OTel-shaped, not OTel-protocol-compatible.
Real Python AST parsing; CI opens an actual correction PR on drift.
Only catches drift in already-documented functions.
Weighted confidence score from 3 independent checks, wired into every arbitrated request.
Critics are pattern heuristics, not separate model calls.
Real BM25 scoring plus real ingestion of markdown files with genuine line-number citations.
“Dense” score is lexical trigram overlap, not embeddings.
Real computed latency, cost, and savings on every response.
Plain-English → SQL translation, structurally incapable of emitting destructive statements, plus AST-ish execution guardrails.
Rules-based translator, not an LLM call.
Live-wired weighted variant selection with a real two-sample z-test for significance.
No multiple-testing correction across many variants yet.
Exact parameter math; benchmark accuracy is a real function of rank, verified strictly increasing.
Estimates a training run — doesn't perform one.
Real CLOSED/OPEN/HALF_OPEN state machine and token-bucket limiter, now aware of per-team limits.
Single-process — a fleet needs Redis for shared counters.
Percentage rollout with moving-average auto-rollback — verified tripping live.
Every failed trace is auto-mined into a growing golden eval dataset.
No dedup/clustering of similar failures yet.
What makes this service-shaped, not just clever
This is an engineering demonstration, not a product for sale or rent — the multi-tenant primitives below are real and tested, with nothing behind a paywall.
SHA-256 hashed at rest, shown once on issuance, revocable — verified end to end, including rejecting a revoked key.
Configurable requests/min, tokens/min, and spend budget per team — auth is opt-in, off by default, so the demo below needs no key.
Cumulative spend is SQLite-backed and survives a process restart — proven by actually killing the process and checking the number again.
Aegis AI Reliability Engine
An open-architecture AI Gateway and Control Plane featuring circuit-breaker failovers, semantic cost routing, AST-based SQL guardrails, and hybrid retrieval with citation grounding.