LLM Gateway
An LLM gateway is a proxy that sits in front of one or more large language model providers to route requests, log usage, cache responses, and enforce rate limits.
Last updated:
An LLM gateway is the single entry point applications call instead of hitting each model provider directly. It centralizes credentials, routes traffic to the right model, logs requests and token counts, caches repeat responses, and applies rate limits and retries. Teams adopt one to avoid scattering API keys and provider-specific code across every service.
A gateway is about traffic management, not spend management. It can tell you a request happened and which model served it, but it does not, by itself, attribute that token spend to a user or project, enforce a dollar budget on the request path, or apply governance controls like PII scrubbing and prompt-injection defense.
Where AI Token FinOps fits
AI Token FinOps is the layer that turns gateway traffic into cost control: per-session and per-project attribution, hard token budgets enforced before an overrun reaches the provider invoice, and forecasting. Behest is the control center that adds this on top of routing — and can operate as your full AI backend if you choose, though that is an option, not a requirement. If you only need routing and logging, a gateway is enough; if you need cost and governance controls, that is FinOps territory.