Benchmarks

AI Gateway Benchmarks: Full Governance at Near-Zero Latency Overhead

What one fully governed AI request costs in added latency: policy, pricing, budget, guardrails, accounting, and the call itself, timed end to end under sustained load.

Start free trialTalk to the founders
≤ 13ms
p95 for a fully governed request, end to end, on the cloud gateway
Model allow-list block
Refused before provider egress
≤ 1 ms
Edge ≤ 1 ms
Guardrail block
Refused before provider egress
≤ 3 ms
Edge ≤ 1 ms
Exact-cache hit
Provider call skipped
≤ 5 ms
Edge ≤ 3 ms
Governed request
Auth, policy, pricing, budget, the call itself
≤ 13 ms
Edge ≤ 10 ms
Governed + guardrails
Secrets, personal data, and org rules
≤ 13 ms
Edge ≤ 10 ms
Exact-cache miss
Governed request plus cache lookup and write
≤ 19 ms
Edge ≤ 16 ms

Server-measured p95, against a deterministic backend with no live provider. Thick bars: cloud gateway on managed credits. Thin bars: edge gateway on One Google Cloud n2-standard-4 VM (4 vCPU, 16 GB), at 300 requests per second sustained. Two edge gateway replicas behind HAProxy, with the edge's Redis, its execution engine, and the test backend on the same VM. Bars scale 0–21 ms.

0
provider calls on a blocked request
100%
of provider spend skipped on a cache hit
$0.00
overspend in a concurrent budget race
The problem

Why we publish this

Every AI gateway claims to be fast. Few say what their number includes.

  • Vendor benchmarks usually run every policy off
  • Provider response time, which no gateway controls, gets blended into the headline number
  • Guardrails, budget enforcement, and caching are rarely measured on their own
How it works

How we measure

Sustained load against a test backend that adds no delay, so each figure is Cloptima's own time, start to finish.

  • Every request runs a production policy: model allow-list, token caps, rate limits, and budgets
  • Cloud requests also reserve and settle a prepaid wallet
  • Edge: One Google Cloud n2-standard-4 VM (4 vCPU, 16 GB), at 300 requests per second sustained
  • We publish p95: the highest across our test sessions, not the best
Day one

Speed and guarantees

Governed, guardrail, and cache figures are timed. Block and budget results are guarantees: zero provider calls and zero overspend, on every run.

Architecture

Governance and execution in one number

Portkey, Bifrost, LiteLLM, and similar tools are proxy engines. Cloptima also enforces policy, pricing, budgets, guardrails, and accounting on every request, and the figures above include all of it.

Developer quickstart

Keep your SDK, change the base URL

The gateway speaks the OpenAI and Anthropic protocols. Point your existing client at it and add attribution headers.

bash
# Measure end-to-end governed latency including auth, policy allowlist, atomic budget & outbox
curl -w "\nConnect: %{time_connect}s | TTFB: %{time_starttransfer}s | Total: %{time_total}s\n" \
  -i https://api.cloptima.ai/v1/ai/chat/completions \
  -H "Authorization: Bearer $CLOPTIMA_PAT" \
  -H "X-Cloptima-Team: benchmarks" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-5.6-luna", "messages": [{"role": "user", "content": "ping"}]}'

# Published benchmark: p95 up to 13ms end to end, governance and execution included
Capabilities

What each scenario includes

CapabilityWhat it includesResult
Governed requestVirtual-key auth, attribution, policy checks, live pricing, atomic budget and wallet enforcement, and the call itself, plus a reconciled ledger record.p95 up to 13ms (10ms on edge)
GuardrailsThe governed request with input and output guardrails redacting benign traffic: secrets, personal data (emails, SSNs, phone and card numbers), internal hosts, and project codenames.p95 up to 13ms (10ms on edge)
Exact response cacheGoverned cache hits versus misses on identical requests.Hit: p95 up to 5ms (3ms on edge), with no provider spend. Miss: p95 up to 19ms (16ms on edge).
Guardrail block enforcementA request that matches a guardrail block rule.Blocked in p95 up to 3ms (1ms on edge). Zero provider calls on every run.
Model / provider allow-list enforcementA request for a model or provider outside the bound policy.Blocked in p95 up to 1ms on cloud and edge. Zero provider calls on every run.
Atomic budget enforcementA concurrent burst of requests racing one budget ceiling at the same instant.Zero overspend on cloud and edge. Every admitted request reconciles exactly against the ledger.
Get started

Explore the gateway

See the features behind these numbers.

Comparisons

How the Cloptima gateway compares

Cloptima vs LiteLLM

OpenAI-compatible gateway routing vs. full FinOps control plane, attribution, and hot-path team budget limits.

Cloptima vs Portkey

Routing and guardrails vs. pre-flight budget enforcement, finance ledger, and p95 up to 13ms end-to-end latency.

Cloptima vs Cloudflare AI Gateway

Edge proxying vs. enterprise attribution, team quota enforcement, and cost reporting.

Cloptima vs Helicone

LLM observability logs vs. active request-path budget controls, response caching, and unit economics.

FAQ

Questions teams ask

Put your first governed call through in five minutes

Point your SDK at the gateway, bind a policy, and watch the first call land on the ledger.

Start free trialTalk to the founders
No credit card required
5-minute setup
Free trial