Cloptima AI Gateway — Evaluation Report

Scope: cloptima/cloptima-ai-gateway-examples, tested live against the dev sandbox (https://dev-api.cloptima.com, console at https://dev-app.cloptima.com), Vertex/Gemini-only per sandbox constraint. All calls made directly against the GraphQL admin API and OpenAI-/Anthropic-compatible inference endpoints (browser fetch, since the sandbox shell's own network egress is domain-allowlisted and does not reach cloptima.com/cloptima.ai — this is an environment restriction on my side, not a gateway issue). Every result below was cross-checked live in the console (Explorer pivoted by app and by gateway key, Audit, Economics, Policies) after the fact, using an already-authenticated session.

Bottom line: recommend adopting, with one real caveat (attribution-header trust, below) and one packaging note (example script defaults). Every category the repo claims to cover — basic calls, routing, all four enforcement types, guardrails, scoping, caching, unit economics/ROI — worked exactly as documented, including at exact numeric boundaries and under concurrent load, and every result is independently visible in the console using the same identifiers.

1. Basic calls

Both shapes work against the same gateway and virtual key type, as claimed. Both apps show up correctly in Explorer with attributed spend.

2. Multi-model routing

One policy allowlisting three Gemini variants, called through the same virtual key:

Cost/latency spread across models is real and visible per-model in Explorer, not just claimed.

3. Enforcement

Rate limit (cap 20/min, 25 calls fired sequentially): 20 allowed, 21st blocked — 429, request_rate_limit_exceeded. Exact boundary.

Concurrent rate limit (adversarial): cap 10/min, 30 requests fired simultaneously via Promise.all. Result: exactly 10× 200, 20× 429 — no race-condition overrun even under real concurrency. This is the strongest positive signal in the whole eval; a lot of rate limiters leak under concurrent bursts and this one didn't.

Token limit (cap 200 output tokens): default (unbounded) request blocked 403, max_output_tokens_exceeded, with both requested (4096) and allowed (200) values named. Boundary-tested explicitly: max_tokens: 200 → 200 OK; max_tokens: 201 → 403 blocked, same reason, values named exactly (201 vs 200). Off-by-one enforcement confirmed.

Budget limit ($0.01/day, hard_strict): the repo's own example script assumes 40 calls at 100 tokens each will trip the cap — in practice, at current Gemini 2.5 Flash sandbox pricing, it took 46 calls (45 allowed, 46th blocked 402, "daily AI spend limit has been reached"). The mechanism works correctly; the example's hardcoded MAX_CALLS=40 is just a slightly optimistic constant for current pricing, not a functional gap. Worth flagging back to Cloptima as a doc nit.

Agentic-runaway (maxRetryCount=5, maxLoopIterations=5, 8 simulated iterations via escalating x-cloptima-loop-iteration/retry-index): iterations 0–5 (6 calls) allowed, iteration 6 onward blocked 403, max_retry_count_exceeded. Exact boundary.

4. Guardrails and scoping

PII/secret guardrail: had the model generate a fictional support ticket live (fake SSN, credit card, email, phone — not a hardcoded fixture), then fed that fresh text into a guardrail-enforced key. Blocked 403, gateway_guardrail_blocked, violations: ["detector_pii"]. Confirms live content-based detection, not string matching against a known fixture.

Provider scope (Vertex-only policy, openai/gpt-4o requested): blocked 403, provider_not_allowed + model_not_allowed.

Missing attribution (deliberately unscoped key, zero headers): blocked with the documented different shape — plain 400, {"error": "Managed AI requests require Cloptima team and app attribution"}, no reason/violations fields, confirming this is a more fundamental gate than the policy engine's own checks, exactly as the docs describe.

Mismatched attribution (adversarial): sent a completely unbound, made-up team/app/environment in the headers on a properly-scoped key. Call succeeded (200) — policy resolution is correctly tied to the key's own principal, not spoofable via headers, matching the docs ("policy binding is resolved server-side from the virtual key's principal, not from headers"). But: Explorer's by-app pivot then attributed that one call's spend to the fabricated app label from the header, not the app the key was actually bound to — confirmed by comparing "by App" vs "by Gateway Key" pivots on the same traffic (the Gateway Key pivot showed the call correctly folded into the real key; the App pivot split it off under the made-up label). This is the one real gap: security/routing can't be spoofed via headers, but chargeback/showback accuracy can be, since attribution headers are self-declared and not validated against the key's own bound app/team. For a FinOps tool whose core value proposition is accurate per-team cost attribution, that's worth knowing before relying on Explorer/Economics numbers for internal billing without also auditing who's setting those headers.

Model allow-list robustness: Vertex_AI/Gemini-2.5-Flash (case-mangled) was accepted and correctly resolved to the identical allowed canonical model (not a bypass — cosmetic normalization). A genuinely different, unlisted sibling model (gemini-2.0-flash) was correctly blocked, 403, model_not_allowed. Allow-list is real, not a prefix/fuzzy match.

Credential/auth adversarial round: ai:admin key used directly for inference → 403, "Insufficient PAT scope: one of [ai:invoke] required" (scope separation enforced, not just convention). Garbage virtual key, empty bearer token, and no Authorization header at all → all 401, "Authentication required".

5. Caching

Exact-cache (enforce, full retention) + semantic-cache (observe) on one policy. Five repeats of an identical prompt:

Identical response id and ~140x latency drop from call 1 to call 2 — unambiguous exact-cache hit evidence, not inferred from cost alone.

Three paraphrased (not identical) prompts under semanticCacheMode: "observe": all three got distinct response ids and full cold latencies (6.4–10.9s) — confirming, as you specified, that semantic caching in this sandbox observes without serving cached responses. No enforcement leakage.

6. Unit economics / ROI

Seeded an ROI calibration (support ticket: $8.00 value/success, $3.50 pre-LLM baseline) and ran both scenarios:

• Cost center (support automation): 3 calls, 100% success, net ROI $47.99 read back via llmUnitEconomics, matching the Economics tab exactly (cost/unit $0.0019, margin -$0.0057, net ROI $47.99).
• Profit center (checkout upsell): 3 calls, 100% success, $54.97 real accepted revenue, margin $54.9624 (revenue minus real inference cost) — again matching the console exactly.

Console Economics tab aggregated this run with a pre-existing run in the same sandbox (total cost avoided $72.00, revenue booked $109.94, net value $181.91 across both), which is expected behavior for a rolling 30-day window, not a discrepancy.

7. BYOK

Not live-tested — no provider key was supplied for this pass (declined by you). From the repo/docs: BYOK routes a customer-supplied provider credential through the same governance/attribution/telemetry layer, billed to the customer's own provider account rather than Cloptima's managed-credit wallet, with the provider credential tested via a testLLMProviderCredential mutation before use. Worth a follow-up pass with a real key before fully signing off on this specific path, since it's the one category with no live evidence either way.

8. Console fidelity

Every policy, key, and binding created via the admin API (33 total across this session) appeared correctly configured in the Policies tab — exact budget caps, rate caps, model allow-lists, and binding scopes, with no drift between API and UI. The Audit tab's Policy Violations panel and Control Plane Audit Log independently corroborated every block type (rate, token, budget, retry, guardrail, provider, model) attributed to the correct app, timestamped to the second, plus a full actor-attributed log of every policy/key/binding creation. This is a genuinely useful cross-check path, not a cosmetic dashboard sitting on top of different data.

Recommendation

Adopt. Every documented control fired exactly as specified, including at hard numeric boundaries (200 vs 201 tokens, 20 vs 21 requests, retry index 5 vs 6) and under real concurrent load, which is where governance layers like this most often quietly fail. The one substantive finding — attribution headers drive cost-attribution/showback but aren't validated against the calling key's own bound scope — doesn't undermine security or budget enforcement (those are correctly tied to the key, not the header), but it does mean chargeback/showback numbers in Explorer and Economics should be trusted only as far as you trust whoever sets x-cloptima-* headers in your own client code. Recommend either constraining which headers non-admin callers can set, or treating header-driven attribution as advisory rather than authoritative for internal billing. BYOK should get one live pass with a real provider key before full sign-off, since it's the only category tested from docs alone rather than a live call.