Best LLM Gateways In 2026: Routing, Caching, and Cost Control Compared

The best LLM gateway in 2026 is the one that matches the shape of your traffic, not the one with the lowest published latency overhead. Chat-shaped traffic wants semantic caching and a wide model catalog. Agent-shaped traffic wants per-step routing and context discipline, because caching barely fires on it. Get that call wrong, and you will install a gateway, watch your bill keep climbing, and conclude that gateways do not work.

Here is the number that should reframe the whole decision. In April 2026, Anthropic more than doubled its own published cost estimate for Claude Code, from $6 to $13 per developer per active day, with the 90th-percentile ceiling moving from $12 to $30. A company spokesperson told Business Insider there was no pricing or product change. The primary model had moved from Sonnet 3.7 to Opus 4.7, and the newer model simply consumed more tokens per task.

Per-token prices fell. The bill went up 2.17x. That gap is what an LLM gateway does and does not control, and it is the subject of this piece.

What follows is the mechanism of each cost lever with its published ceiling, a nine-tool comparison with real numbers, and a selection guide organised by the requirement you actually have.

What does an LLM gateway actually control?

An LLM gateway is a reverse proxy built for model traffic. Your application calls one OpenAI-compatible endpoint, and the gateway handles provider authentication, model selection, retries, failover, caching, rate limits, spend attribution and logging. You stop writing provider-specific code and you get one place to see what everything costs.

That is the standard description, and it is accurate. It is also incomplete in a way that matters, because it describes the plumbing rather than the economics.

Three levers, three different ceilings

Cost control at the LLM gateway layer breaks into three mechanisms, and only two of them live inside the gateway.

Route changes the price per token by sending a request to a cheaper model. Its ceiling is the price gap between tiers, which in 2026 runs to roughly 50x between Claude Haiku 4.5 at $1 per million input tokens and Claude Fable 5.1 at $10, or wider once you include open-weight models.

Cache stops you paying twice for a prefix you already sent. Its ceiling is set by the share of your spend that is repeated input, and it is a hard ceiling, because no provider discounts output tokens.

Reduce cuts the number of tokens that exist at all: fewer tool schemas declared, less context re-sent, fewer turns, shorter reasoning chains. Nothing at the gateway layer can do this, because a proxy sees a finished request body. It cannot know that 6,610 of those tokens are a browser toolset you are not using in this session.

Installing a gateway and turning on caching and routing is the right first move, and it is where most published guidance ends. It is also where the ceiling arrives. The rest of this piece is about what sits above it.

The gateway doesn't decide how many tokens you send

The load-bearing point: a gateway is a request-time system. By the time a request reaches it, the token count is already fixed: the system prompt is written, the tool schemas are attached, the file contents are in the messages array, and the reasoning budget is set.

The gateway can send that request somewhere cheaper. It cannot make it smaller.

Why do you need LLM gateways in 2026?

Because token volume per task rose faster than price per token fell. Two documented mechanisms drive it, and both are invisible on a pricing page.

Same product, same vendor, no price change. The published cost estimate still moved 2.17x.

Better models spend more tokens on the same task

The Claude Code repricing is the clearest public example. Anthropic's guidance now reads: across enterprise deployments, roughly $13 per developer per active day and $150 to $250 per developer per month, with 90% of users below $30 per active day. Before 16 April 2026 the same page said $6 and $12. The change was attributed to Opus 4.7 replacing Sonnet 3.7 as the primary model.

For a 100-developer team at 20 active days a month, that is a move from about $12,000 to about $26,000 a month for identical work, with no line item to point at.

The tokenizer changed underneath you

Anthropic's pricing documentation states it plainly: Claude 4.7 and later models use a newer tokenizer that "produces approximately 30% more tokens for the same text." Sonnet 4.6 and earlier use the previous one.

Read that against the headline price. Sonnet 5 lists at $2 per million input and $10 per million output, cheaper than Sonnet 4.6 at $3 and $15. Anthropic confirmed on 10 August 2026 that the $2/$10 rate is now permanent and the increase to $3/$15 scheduled for 1 September will not happen. Good news, and widely reported.

Less widely reported: the same English sentence becomes roughly 30% more billable tokens on Sonnet 5 than it was on Sonnet 4.6. A 33% price cut against a 30% token increase is close to a wash, and on code and structured data, where the tokenizer shift bites hardest, it can go the wrong way.

The practical consequence for anyone comparing gateways on cost: per-token price is not a stable unit of measurement. Cost per completed task is. A gateway that reports the first and not the second is telling you about the market rather than about your workload.

Token tax

Tool schemas are re-sent on every single turn; they are large, and almost nobody counts them. This is the clearest case in the whole category where a small change at the client beats anything the gateway can do.

Anthropic publishes the exact token counts. Assembling them into one table appears not to have been done in public before, so here it is.

Computed from Anthropic's published tool-use token tables. The full stack costs 622,050 tokens of pure schema across a 50-turn session.

What a full toolset costs before you type anything

On Claude Opus 5, from Anthropic's tool use pricing tables:

Across a 50-turn session that is 622,050 tokens of schema, before a single line of your code, your prompt or a tool result. Uncached, on Opus 5, that is $3.11 per session. Read from cache it is $0.31.

Two things fall out of this table:

First, caching is not optional on agentic workloads, it is structural. The 10x difference between $3.11 and $0.31 is entirely the cache.

Second, and more usefully: the computer and browser toolsets are 11,130 of those 12,441 tokens, 89% of the schema tax. If your agent is editing files and running tests, it does not need either. Removing tools you are not using in this session is a one-line config change that beats most gateway tuning, and no gateway can make that decision for you because it cannot tell which tools your task needs.

Anthropic also documents that disabling the zoom member of the computer toolset removes about 410 tokens, and that the browser toolset's four optional members add about 880. That level of granularity exists because the tokens are real money at scale.

The 9 best LLM gateways in 2026

Ranked by how much of the cost problem each one actually addresses, with the honest limit on each.

1. ML.ai Inference

Best for: teams whose spend is dominated by a repeating set of tasks, and who want the savings to compound rather than plateau after the first configuration.

Every other gateway on this list routes between models somebody else trained. ML.ai does that across 40+ models, and then does something structurally different with what it learns: it shadows your frontier traffic with no user impact, clusters it into recurring tasks, uses the frontier answers as gold labels to fine-tune a small task model, and shifts traffic across only once that model clears your own evals. A drift monitor triggers retuning when the distribution moves.

That loop is why it leads this list, and the argument for it does not rest on our own numbers.

In April 2026, Guo, Wu and Yiu published RouteNLP, which builds this architecture and measures it. Their enterprise partner was spending over $200,000 a month on inference "despite over 70% of queries being routine tasks well within the capability of smaller models." RouteNLP cut cost 40% to 85% across six tasks, and 62% on simulated production traffic, while holding 96% to 100% quality on structured tasks.

The ablation is the number that matters here. Removing the distillation co-optimisation loop increases cost by 28%. Static routing leaves that on the table by construction.

There is a related structural finding in the literature. A 2026 paper on cost-efficient inference notes that routing "is inherently binary (SLM or LLM), and when the LLM is selected, the system incurs the full inference cost." Routing alone cannot get below the frontier price on the requests that genuinely need the frontier. Improving what the small model can handle is the only lever that moves that boundary.

Published ML.ai figures for a four-step task. The two cheap steps fall 61%. The two hard steps fall 28%.

On the coding side, ML.ai Code applies the same routing per step rather than per request. Classifying intent, extracting fields, drafting the change and verifying it are four different jobs, and most agents send all four to the same frontier model. ML.ai's published per-task figures put that at $0.70 against $0.43 routed, a 39% cut. The distribution inside that total is the interesting part: the classify and extract steps fall 61%, the draft and verify steps only 28%. Teams that try to optimise drafting first are working on the step with the least headroom.

It ships as a VS Code and Cursor extension, four agents (Explore, General, Architect, Plan) of which only General can write, a native diff before every permission prompt, MCP server support, and a shell command classifier that parses rather than pattern-matches, so `curl url` passes and `curl -o ~/.zshrc url` does not. Nothing reaches disk before you approve it, measured at 16ms from approval to write.

The numbers, with their caveats stated. ML.ai publishes 86% on SWE-bench Verified against 58% for a leading frontier model single-shot, using retries and test-driven verification in place of one expensive guess.

Read the footnote on that. It is a 50-instance slice of the public set; the full 500-instance run and SWE-bench Pro are still in progress, and 50 instances carry a wide confidence interval. The 30% to 45% cost range is a stated design target ahead of general availability, not an average across customers. The one named customer result is NeoSapien's consumer voice assistant: 42% lower monthly AI bill, 28% faster responses, 21 days to full rollout, zero quality regressions.

Where it is not the answer: If you need a gateway running in front of production traffic this afternoon with no shadow period, take LiteLLM or Cloudflare. Where traffic is genuinely one-off with no recurring task structure, there is nothing to distil and the loop has nothing to learn from. If you need 383 models to experiment against, OpenRouter has a wider catalog. And if your requirement is a security control plane your CISO has already approved, the Palo Alto answer below is the shorter procurement path.

Try the editor agent from the VS Code Marketplace listing, or start a 30-day pilot on one workload.

2. LiteLLM

Best for: self-hosting a unified endpoint across 100+ providers without paying anyone a platform fee.

LiteLLM is the category default and deserves to be. One OpenAI-compatible endpoint, virtual keys, budgets, spend tracking, fallbacks and load balancing, all open source and all free if you run it yourself. If you are not sure what you need yet, start here and find out.

The constraint is the Python runtime. LiteLLM's own high-availability docs note that deployments above 1,000 requests per second across 10+ instances can hit PostgreSQL connection exhaustion and deadlocks on shared key and team records. Semantic caching requires Redis Stack with the RediSearch module, which many managed Redis offerings do not ship by default. The Rust rewrite, benchmarked in July 2026 at 0.7ms p99 and 21.8MB peak memory, addresses the runtime directly and is the thing to watch.

3. Prisma AIRS AI Gateway (formerly Portkey)

Best for: enterprises where the gateway has to satisfy a security review before it satisfies an engineer.

Palo Alto Networks completed its acquisition of Portkey on 29 May 2026, confirmed in its Form 10-Q. Portkey now serves as the AI Gateway inside Prisma AIRS, which reached general availability about six weeks after close.

The capability set was already strong: conditional routing, automatic fallbacks, circuit breakers, simple and semantic caching, budget limits, guardrails including PII redaction and jailbreak detection. What the acquisition adds is agent identity authentication and runtime inspection of agentic traffic, sitting inside a platform most large enterprises have already procured.

The context for why this happened is worth knowing. Palo Alto's own next-generation firewall telemetry shows MCP activity rising from 11% in late 2025 to 41.4% by mid-2026, with monthly AI transaction volume growing twelve-fold over the same six months and some individual sessions moving hundreds of megabytes of enterprise data outbound. If you are connecting MCP servers to production agents, that is the number to take to your security team.

The trade-off is independence. Portkey is now a security product with a gateway inside it, and pricing follows enterprise procurement rather than a public page.

4. OpenRouter

Best for: evaluating many models quickly against one API key, and for teams who want breadth over control.

383+ models across 60+ providers on an OpenAI-compatible endpoint, no token markup, a 5.5% fee on credit purchases, and roughly 25 to 28 genuinely free models rate-limited to 20 requests per minute and 50 or 1,000 requests per day depending on whether you have ever bought $10 in credits.

For model evaluation this is the fastest path that exists. For production it is thin on governance: routing is manual or automatic model selection rather than policy-driven, caching is whatever the provider does, and the 5.5% credit fee is $550 a month on $10,000 of spend. At scale, bring your own keys elsewhere or self-host.

5. Cloudflare AI Gateway

Best for: teams already on Cloudflare who want caching, rate limiting and analytics with no infrastructure to run.

Free on standard and Workers Paid plans, running on Cloudflare's edge, with unified billing added in 2026 so third-party model usage can land on your Cloudflare invoice. Setup is a base URL change and there is no service to operate.

The limits are the usual managed-service ones: routing rules are still maturing, prompts and completions are processed inside Cloudflare's infrastructure which raises data-residency questions in regulated sectors, and Logpush to external storage bills $0.05 per million records beyond 10 million a month. For agentic workloads that need deep per-step logs, the abstraction gets in the way.

6. Bifrost (Maxim AI)

Best for: high-RPS production traffic where tail latency and memory are the binding constraint, self-hosted.

Written in Go, so no GIL and no asyncio overhead. The performance claims are vendor-run and contested, as covered above, but the architectural argument is sound: a compiled runtime with goroutines behaves better under sustained concurrent load than a Python proxy, and the failure mode LiteLLM users report is exactly memory exhaustion under concurrency.

The genuinely differentiated feature is native semantic caching with dual-layer hit matching and no Redis Stack requirement, plus an MCP Gateway with Code Mode in the free OSS tier. Content-safety guardrails sit behind the paid plan, which is the reverse of Portkey's packaging.

7. Kong AI Gateway

Best for: organisations already running a Kong API mesh.

Kong brings LLM routing into a mature API management platform: plugin ecosystem, SSO, PII redaction, the governance apparatus enterprises already know how to operate. If Kong is your API layer, adding AI traffic to it is the low-friction move and the audit story is already written.

If Kong is not already in your stack, the operational weight is difficult to justify for LLM routing alone. Of the nine options here it is the heaviest, and it assumes infrastructure underneath it.

8. Helicone

Best for: teams whose first problem is visibility rather than routing.

One-line integration, per-request logging, cost and latency breakdowns, user-level attribution, open source with Docker, Kubernetes and bare-metal deployment. Free to 10,000 requests, Pro from $79 a month. Cross-provider caching, where a response cached from one provider serves a request routed to another, is a genuinely useful feature that few others offer.

Routing is latency and health-aware balancing without conditional routing on custom metadata. If you need policy-driven routing, pair it with something else or pick differently.

9. Vercel AI Gateway

Best for: Next.js teams who want consolidated model access and billing without leaving the platform.

Native to the Vercel AI SDK, consolidated billing, failover across providers, minimal setup if your application already lives there. Governance and routing sophistication are thinner than the dedicated options, and you are inside the Vercel platform for better and worse. For a team shipping a Next.js product, that trade is often correct.

Which LLM gateway should you actually pick?

Match the requirement, not the ranking.

The two-gateway pattern

The teams getting the most out of this stack frequently run two things rather than one, and the split is by traffic shape.

High-volume, low-variance traffic (classification, extraction, support answers, RAG endpoints) goes to a gateway with aggressive semantic caching and tier routing. Cache hit rates of 40% are achievable and every hit is free.

Low-volume, high-variance traffic (coding agents, research agents, anything reading a private repository) goes to an agent runtime that controls context and routes per step, because semantic caching will not fire on it and the money is in input volume rather than input price.

Running one configuration across both means tuning for the average of two workloads that have nothing in common.

What should you do in your first two weeks?

A sequence, in the order that produces information fastest.

Days 1 to 2: run a token cost audit. You need three numbers per workload: total input tokens, cached input tokens, and output tokens. Cache hit rate is the second divided by the first. If input is over 80% of your cost, caching and context reduction are your levers. If output is over 40%, you have a reasoning-effort and turn-count problem that no cache will touch.

Day 3: audit your tool schemas. Count the tools you declare against the tools your agent actually calls in a typical session. Anthropic's token counting endpoint gives you the exact number in advance. Removing an unused browser toolset is 6,610 tokens off every turn and takes ten minutes.

Days 4 to 5: turn on prefix caching properly. Order your prompt so the stable parts come first: system prompt, then tool definitions, then few-shot examples, then the variable content. A cache breakpoint after a section that changes every request does nothing.

Week 2: pilot two gateways on real traffic. Not a benchmark. Your traffic, for five days, measuring cost per completed task and the retry rate. If a gateway cannot report cost per completed task, that is your answer about how seriously it treats cost.

Then decide about routing. Model routing is the right third move, not the first, because a routing layer sitting on top of an uncached, over-tooled workload optimises the wrong number and hides the real one.

Start with the number that tells you which problem you have

You now have the mechanism behind each cost lever and the ceiling on each.

The first step is small. Pull three numbers for one workload: input tokens, cached input tokens, output tokens. That ratio tells you whether you have a caching problem, a routing problem or a context problem, and it takes an afternoon. The third one is the one people usually discover they have been solving with tools built for the first two.

If the workload turns out to be a repeating set of tasks running on a frontier model, that is the case ML.ai Inference was built for: shadow it for 30 days against your own evals, agree the targets in writing first, and owe nothing if we miss them. To see it in your editor before talking to anyone, ML.ai Code installs from the Marketplace and the first question you ask can be one that changes nothing.

Frequently asked questions

What is an LLM gateway?

An LLM gateway is a proxy that sits between your application and one or more model providers, giving you a single OpenAI-compatible endpoint. It handles provider authentication, model routing, failover, caching, rate limiting, spend tracking and logging, so your application code does not have to. Every major option in 2026 passes provider token rates through without markup and makes money on a platform fee, a subscription, or an enterprise contract.

Do LLM gateways add meaningful latency?

Rarely. Published overhead ranges from microseconds to a few milliseconds against inference calls that take one to five seconds. The numbers are also all vendor-published and mutually inconsistent, with two vendors differing by roughly 400x on the same gateway. Latency becomes a genuine constraint only at sustained high RPS with streaming, where tail latency and memory exhaustion matter more than median overhead.

Does semantic caching work for coding agents?

Mostly no. Semantic caches embed the entire messages array, which for a coding agent contains repository context, diffs and tool results that are unique to each request. Prefix caching does work well on the same traffic, because system prompts and tool schemas genuinely repeat. The two get marketed as one feature and behave completely differently.

How much can prompt caching realistically save?

Up to 75% of a typical agentic session, and no more. Anthropic prices cache reads at 0.1x base input, so a perfectly cached 1M-input, 40k-output session on Opus 5 falls from $6.00 to $1.50. At that floor, output is 67% of what remains and no provider discounts output tokens on any model.

Can an LLM gateway reduce the number of tokens I send?

No. A gateway sees a finished request body, so the token count is fixed before it arrives. Reducing token volume means changing what the client sends: fewer tool schemas, less re-sent context, fewer turns, lower reasoning effort. This is why cost work that stops at the gateway hits a ceiling, and why the same workload can cost twice as much on two setups running the identical model through the identical gateway.

Is self-hosting a gateway cheaper than a managed one?

Usually yes on paper and often no in practice. Self-hosting removes the platform fee entirely, which is $550 a month on $10,000 of OpenRouter spend. It adds Redis, Postgres, monitoring, on-call and the operational work behind documented failure modes like connection exhaustion above 1,000 RPS. Below roughly $5,000 a month in model spend, the platform fee is cheaper than the engineering time.

What is the difference between an LLM gateway and an inference layer?

A gateway routes requests to models somebody else trained. An inference layer can also change which models exist, by distilling your recurring traffic into task-tuned small models and shifting work onto them once they clear your evals. The distinction matters because routing alone cannot go below frontier price on requests that need the frontier; improving the cheap tier is the only thing that moves that boundary.