You Are Renting a Moving Truck to Carry One Bag of Groceries
Almost everyone overpays for inference the exact same way. They fire every request at the biggest model they own, then act shocked when the bill reads like a car payment. The other camp does the reverse. They route everything to the cheap model and quietly ship worse answers on the requests that actually mattered.
Both are wrong. Both are lazy. And both are fixable in an afternoon.
I spent most of last year cutting inference cost on a vLLM endpoint doing tens of thousands of requests a day. There was no clever prompt that saved the day. It was three boring ideas stacked on top of each other: route on complexity, cache what repeats, batch what runs at the same time. That code lived behind an NDA. This is the same idea rebuilt in the open, small enough to read in one sitting.
How It Actually Works
A FastAPI service. Not a notebook you run once and forget in a tab.
• Routing: it scores every prompt on length, code markers, and reasoning language, then sends it to the small tier or the big tier. "What are your hours?" does not need your smartest model. It stops pretending otherwise.
• Caching: a TTL cache keyed on the prompt, model, and max_tokens hands back repeat questions instantly. You pay for an answer once, not every time someone asks it again.
• Batching: the batch endpoint groups cache-misses by tier and fires each tier as a single call, so the fixed overhead spreads across every request in the group. That is the real engine behind vLLM's continuous batching win.
• Hardening: optional API-key auth, request size limits, a readiness probe, clean error handling, and a non-root Docker image with a health check. Boring on purpose.
The Part Nobody Wants to Talk About
The routing logic was never the hard part. Making the benchmark honest was. Any repo that promises you a suspiciously round "94% savings" is selling something. So I put the workload mix and the cost gap between tiers in plain sight in the eval script, and the whole thing runs against a deterministic mock. Zero API keys, zero cloud spend, and you can check my math instead of trusting it.
That one constraint, reproducible with nothing external, forced a cleaner design than I would have shipped otherwise. Point it at a real vLLM or any OpenAI-compatible backend with an env var when you are ready for real traffic.
The Number
On a 500-request workload (25% genuinely hard, 25% repeat FAQs, 50% simple templated stuff), stacking routing, caching, and batching cuts total cost by 73% and p95 latency by 73%. Same answer quality on every request that actually needed the big brain.
Then I ran the same router against real weights, because a mock will tell you whatever you designed it to say. Two real Qwen models as the tiers, real wall-clock latency, real per-token pricing. The saving fell to 15%, because the long analytical answers that correctly stay on the big model dominate token spend. And the 0.5B tier scored 79% on exactly checkable prompts (it thinks 45 plus 55 is 90), so the cost win comes out of your quality budget when your tiers are that far apart. Both findings are in the README, because they are more useful than the flattering version.
CI runs the suite on Python 3.10, 3.11, and 3.13, plus a Docker build that boots the container and hits the live endpoint. 28 tests, all green. Clone it and watch the numbers fall out on your own machine.