AI Gateway: Cost Control, Failover, and New Risk

The last time an AI budget cap bit me, it did not look like a budget problem at all. Our CI/CD pipelines started failing, cryptically, the way pipelines do when something upstream has quietly changed shape. Developers were blocked. Nobody connected the failures to spend straight away, because nothing in the failure text said "spend." We had crossed our OpenAI budget limit, and the first thing that told us was a red build.

There was no single culprit either. It was aggregate growth. Adoption simply outran a cap nobody had revisited since it was set, and because there was no per-team attribution anywhere in the stack, the obvious follow-up question, who is burning this, had no answer. We could see the total. That was all we could see.

That is a cost failure presenting as an availability failure, and it is the sort of thing an AI gateway exists to prevent. It happens because LLM calls typically leave your systems the same way they arrived: as raw SDK calls to a provider, with a key in an environment variable and no layer in between that knows anything about them. This post is about building that layer, what it buys you, and the uncomfortable thing it becomes once you have it.

The Substation at the Edge of Town

Power plants do not connect to your house. They connect to a substation, and the substation connects to everything else. Several generating stations feed in on high-voltage lines. Inside, transformers step each feed down and meter it before it goes anywhere. A metering house logs load per circuit. Breakers sit on the individual outgoing lines, so a fault on one street does not take the district with it. And because every feed lands in one place, an operator can switch between them: plant B picks up when plant A drops, and nobody in the city notices.

An AI gateway is that substation for model traffic. Providers are the plants, your applications are the streets, and the gateway is the one structure that everything passes through. Funnelling everything through one point is exactly what makes metering, policy, and switching possible.

It is also what makes a fault inside that one point dim the whole skyline. Hold that thought.

What an AI Gateway Actually Does

The gateway is the general API gateway routing pattern specialized for a workload that bills by the token and fails in unusually creative ways. Concretely it handles credential management, so applications never hold provider keys; routing and failover, so a provider outage becomes a latency blip; per-team rate limits and budgets, so one team cannot spend the organisation into a wall; caching, so identical prompts stop costing money twice; observability, so tokens and spend are attributable; and guardrails, so PII and prompt injection get inspected in one place rather than in fourteen.

The reason this stopped being optional is money. The FinOps Foundation's State of FinOps 2026 found 98% of organizations now manage AI spend, up from 63% a year earlier, and 73% reported AI costs exceeding original projections. That second number is my incident, industrialised.

Where the credible options sit depends less on features than on who you already are. LiteLLM is the self-hosted default when owning the data path matters. Portkey sells governance as SaaS. Kong AI Gateway makes sense when you already run Kong and this is one more plugin rather than one more platform. Cloudflare AI Gateway fits teams already living in that ecosystem. Envoy AI Gateway reached v1.0 on 23 June 2026 with a unified provider API and token-aware traffic management, which is the natural pick if your platform is already CNCF-shaped.

Self-hosted versus managed is a genuine tradeoff rather than a solved question. Self-hosting keeps prompts and completions inside your boundary and hands you the patching, the scaling, and the pager. Managed hands all of that to a vendor and puts your traffic through their infrastructure. Pick based on which of those two liabilities your organisation is actually equipped to carry.

Setting Up an AI Gateway

The examples below use LiteLLM's proxy config, because it is the most portable illustration of the shape. The concepts transfer. Every step runs on a workstation with Docker installed, so you can watch each control work before you argue for it in a design review. You will need an OpenAI key and an Anthropic key, and the whole thing costs cents to exercise.

Step 1: Put every provider behind one endpoint

Start by declaring the models you allow and the providers behind them. Applications get logical names like chat-default, never gpt-4o and never a provider SDK.

# config.yaml
model_list:
  - model_name: chat-default
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: chat-backup           # the fallback target for Step 4
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: embed-default
    litellm_params:
      model: openai/text-embedding-3-large
      api_key: os.environ/OPENAI_API_KEY

The proxy needs Postgres for virtual keys and spend tracking, and Redis for the cache in Step 5, so bring up all three together:

# docker-compose.yml
services:
  gateway:
    image: ghcr.io/berriai/litellm:main-v1.83.14-stable   # a tag, pinned by digest in Step 6
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports: ["4000:4000"]
    volumes: ["./config.yaml:/app/config.yaml:ro"]
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      LITELLM_MASTER_KEY: sk-local-master-key       # local only, never a real deployment
      DATABASE_URL: postgresql://litellm:litellm@db:5432/litellm
      REDIS_URL: redis://cache:6379
    depends_on: [db, cache]
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
  cache:
    image: redis:7

That image tag is one of LiteLLM's -stable releases rather than main-latest, which is the version you want in front of anything you care about. Check the current one before you copy this, because the tag you pin should be a release that exists at the time you are reading.

Put your two provider keys in a .env file beside the compose file, where Docker Compose picks them up automatically:

# .env
OPENAI_API_KEY=sk-proj-your-openai-key-here
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key-here

Add that file to .gitignore before you paste anything real into it. These are the only two places a provider key appears in the whole setup, which is the point of Step 2.

Then start it and wait for the endpoint to answer. A bare curl in the first half minute returns curl: (52) Empty reply from server, because the port is open before the proxy is ready to serve on it:

docker compose up -d

# the proxy binds port 4000 immediately but runs database migrations first,
# so give it up to a minute before it answers
until curl -sf http://localhost:4000/health/readiness; do sleep 3; done

Step 2: Move the keys out of your applications

Provider keys now live only in the gateway's environment, which the compose file above already arranged. Applications hold a gateway key instead, which you can rotate, scope, and revoke without touching a provider console or redeploying anything. Add this block to config.yaml and restart with docker compose up -d:

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL   # required for virtual keys and spend tracking

Because the surface stays OpenAI-compatible, adopting it from .NET is a base URL and a key, not a rewrite. A console app is enough to prove it. The official OpenAI package is the one you want, pointed somewhere other than OpenAI:

dotnet new console -n GatewayDemo -o GatewayDemo
cd GatewayDemo
dotnet add package OpenAI

Replace the generated Program.cs with this. Top-level statements mean there is no class or Main to write:

using OpenAI;
using OpenAI.Chat;
using System.ClientModel;

var client = new OpenAIClient(
    new ApiKeyCredential("sk-local-master-key"),                    // gateway key, not a provider key
    new OpenAIClientOptions { Endpoint = new Uri("http://localhost:4000/v1") });

// Call sites use the logical model name and never learn who served it.
var chat = client.GetChatClient("chat-default");
var reply = await chat.CompleteChatAsync("Summarise this incident report.");
Console.WriteLine(reply.Value.Content[0].Text);

Then run it against the gateway you started in Step 1:

dotnet run

The completion comes back the way it always did, and the request has been metered, attributed, and routed on its way through:

Hello! How can I assist you today?

The Endpoint property is doing all the work. Without it the same client talks to OpenAI directly, and every control in this article is bypassed. Note also what is missing from that snippet: no provider name, no provider key, no provider SDK. Swapping who serves chat-default is a gateway config change, and the application never learns it happened.

Step 3: Give every team its own budget and its own breaker

This is the step that answers the opening. One organisation-wide ceiling means the first team to breach it takes down everyone's pipelines. Per-key budgets mean the noisy team hits its own limit and nobody else notices.

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-local-master-key" \
  -H "Content-Type: application/json" \
  -d '{
        "key_alias": "team-payments",
        "models": ["chat-default", "embed-default"],
        "max_budget": 400,
        "budget_duration": "30d",
        "rpm_limit": 120,
        "metadata": { "cost_centre": "PAY-114" }
      }'

That returns a virtual key. To watch the breaker trip rather than take my word for it, mint one with a budget small enough to exhaust, then spend it:

# max_budget in dollars, deliberately tiny
KEY=$(curl -s -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-local-master-key" \
  -H "Content-Type: application/json" \
  -d '{"key_alias":"budget-demo","max_budget":0.01,"budget_duration":"30d"}' \
  | python -c "import sys,json; print(json.load(sys.stdin)['key'])")

# repeat until it stops answering
for i in $(seq 1 20); do
  curl -s -o /dev/null -w "%{http_code}\n" http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    -d '{"model":"chat-default","messages":[{"role":"user","content":"hi"}]}'
done

The 200s turn into 400s once the key exceeds its budget, and every other key keeps working. That is the whole argument in one terminal window: the noisy team hits its own ceiling and nobody else notices. Give CI its own key with its own budget too. A build pipeline that dies because a product team ran an experiment is an org-design failure wearing a stack trace.

Step 4: Configure fallback across providers

Routing is the switching gear. Declare the fallback order and the conditions, and a provider incident becomes extra latency rather than an outage.

router_settings:
  routing_strategy: usage-based-routing-v2
  num_retries: 2
  timeout: 30
  fallbacks:
    - chat-default: ["chat-backup"]
  allowed_fails: 3
  cooldown_time: 60      # seconds a failing deployment sits out

You can simulate the provider outage without waiting for a real one. Point the primary at a key that will be rejected, recreate the container so it picks the change up, and send a request to chat-default:

OPENAI_API_KEY=sk-invalid docker compose up -d --force-recreate gateway
curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key" -H "Content-Type: application/json" \
  -d '{"model":"chat-default","messages":[{"role":"user","content":"hi"}]}' | head -c 400

The completion still arrives, served by Anthropic, and the calling application sees an ordinary success:

{"model":"chat-default","object":"chat.completion",
 "choices":[{"finish_reason":"stop","message":{"role":"assistant",
   "content":"Hello! How can I assist you today?"}}]}

The model field still reads chat-default, because that is the logical name the application asked for, and docker compose logs gateway shows the retry and the switch behind it. Run docker compose up -d --force-recreate gateway again afterwards to restore the real key. A fallback you have never exercised is a fallback you do not have, so break the primary key on purpose once and watch this path before an outage does it for you.

Step 5: Make spend attributable before you need it

Emit spend and token counts per key and per team, tagged with the cost centre from Step 3, so the question "who is burning this" is answered by a dashboard rather than by a forensic exercise.

litellm_settings:
  success_callback: ["prometheus", "otel"]
  failure_callback: ["prometheus", "otel"]
  cache: true
  cache_params:
    type: redis
    ttl: 3600

After the requests you sent in Steps 3 and 4, the attribution is already queryable:

curl -s http://localhost:4000/spend/keys -H "Authorization: Bearer sk-local-master-key"
curl -sL http://localhost:4000/metrics/ -H "Authorization: Bearer sk-local-master-key"

Mind the trailing slash on /metrics/, because without it the proxy answers with a 307 and an empty body, which looks exactly like a broken exporter. The spend counters appear there once real requests have succeeded, so an idle gateway shows only process and in-flight metrics.

Send the same prompt twice and compare latency to confirm the Redis cache is answering the second one. Then alert on trajectory, not on the limit. Fire at 70% of a monthly budget with two weeks left, because an alert that arrives at 100% is a notification that something has already broken.

Step 6: Treat the gateway as the critical path it now is

Pin the version, run from a pinned image digest, and put real health checks on it. Every application in your estate now depends on this pod. Resolve the digest of the image you just ran and pin to that rather than to a moving tag:

docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/berriai/litellm:main-v1.83.14-stable

The rest of this step is where the workstation stops being the point, because probes and rollouts belong to whatever runs it in anger.

# deployment.yaml (excerpt)
image: ghcr.io/berriai/litellm@sha256:<digest>   # digest, not :latest
livenessProbe:
  httpGet: { path: /health/liveliness, port: 4000 }
  periodSeconds: 10
  failureThreshold: 6        # tolerate slow event loops, do not kill on a blip
readinessProbe:
  httpGet: { path: /health/readiness, port: 4000 }

When you are done experimenting, docker compose down -v removes the containers and the spend database with them.

Where the Substation Analogy Breaks

Here is the honest part. A substation is passive iron and copper. It can be knocked out by lightning or a digger, but it cannot be socially engineered, and nobody can poison it through its dependency tree. Your gateway is live, patchable, exploitable software, and 2026 has demonstrated that difference three separate times.

In March, LiteLLM's PyPI packages v1.82.7 and v1.82.8 were published with a credential-stealing payload, live for roughly 40 minutes, exfiltrating to a non-official domain. The one component holding every provider key in your organisation was, briefly, a credential harvester. Users on pinned Docker images were unaffected, which is Step 6 earning its place. In February, a regression in Redis cache eviction closed httpx clients still in use by the proxy, high severity, lasting about six days. And in a separate incident, a synchronous database reconnect froze the asyncio event loop for 30 to 120 seconds, starving the liveness endpoint until Kubernetes killed the pods. A database hiccup became a full proxy outage.

Every plant upstream was still generating fine. The skyline went dark anyway.

Failure shape	Without a gateway	With a gateway
Provider outage	Every app calling that provider is down	Fails over, latency blip
Runaway spend	Discovered when something else breaks	Per-team breaker trips, blast radius is one team
Key rotation	Redeploy every service	One config change
Cost attribution	One invoice, no split	Per key, per team, per cost centre
Compromised gateway build	Not applicable	Every provider key in the estate is exposed
Gateway pod restart loop	Not applicable	Every AI feature in the estate is down

That bottom-right quadrant is new risk you are choosing to accept in exchange for everything above it. Accept it deliberately.

One Substation Per District

I have lived this exact shape at a different layer. We run a shared Keycloak instance serving multiple applications across company domains. It has not failed yet. That is the wrong tense to be comfortable with, because when it does fail, or simply during scheduled downtime, every application loses user and application authentication simultaneously, which is devastating when some of those applications are mission-critical. The resolution we landed on was to decentralise: one Keycloak instance per domain rather than one instance for everyone.

The same decision applies here, and it is the one worth making before your gateway is load-bearing rather than after.

• A shared bottleneck concentrates capability and risk in equal measure. You do not get one without the other.
• Scope the blast radius to a domain, not to the company. One gateway tier per domain gives you the metering and switching without a single company-wide chokepoint.
• Per-team budgets are an availability control, not a finance control. That is what my broken pipelines were really telling me.
• Pin your versions and run from digests. Forty minutes of a poisoned package is enough.
• Your gateway needs the operational rigour you give your payment path, because that is what it now is.

Put the substation in. Meter every feed, break every outgoing line, and route around the plants that go down. Then look at how many districts are hanging off it, and ask whether that number should be one.

So: if your AI gateway went down for an hour tomorrow, how many teams would notice, and how many would still be able to ship? And if you cannot answer today's question of who is burning your token budget, what exactly are you going to say when the pipelines start failing?