Keep Requests Working When a Provider Fails

• Retries with backoff for short failures
• Fallback to another provider for longer ones
• Deadlines for a whole request and for the first words of an answer
• Budgets that count every attempt
• A test that proves failover works before you need it

You need an owner or admin role and a policy bound to your traffic. Every setting here is yours to choose, and each has a sensible default.

Fallback conditions also accept familiar status codes as shorthand: 401, 402, 403, 404, 408, 429, 5xx, and timeout.

• Each fallback provider uses its own credential, or Cloptima credits where they apply
• A provider with no usable credential is left out of the plan, so it can never be tried without one
• Your policy lists decide which providers are allowed
• Where a budget needs a price, a provider with no known price is left out

The guide on asking for a model by name shows how providers are chosen and ordered.

1. 1

   Open the policy

   Go to AI → Policies and open the policy that governs the traffic.

2. 2

   Open the retries and fallback settings

   Find the section for retries, backoff, and deadlines.

3. 3

   Set attempts

   Choose how many tries each provider gets and how many attempts a request may use in total.

4. 4

   Set backoff

   Choose none, or exponential with jitter, and the shortest and longest pause.

5. 5

   Set deadlines

   Choose the total time a request may take and how long to wait for the first output.

6. 6

   Save

   The settings apply to new requests at once.

resource "cloptima_llm_gateway_policy" "support_production" {
  name                           = "support-production"
  mode                           = "enforce"
  allowed_providers              = ["anthropic", "bedrock", "vertex_ai"]
  routing_fallback_routes        = ["bedrock/anthropic.claude-opus-5-5", "vertex_ai/claude-opus-5-5"]
  routing_fallback_conditions    = ["rate_limit", "provider_unavailable", "model_unavailable"]
  routing_max_attempts_per_route = 2
  routing_max_total_attempts     = 5
  routing_backoff                = "exponential_jitter"
  routing_request_timeout_ms     = 30000
  routing_first_output_timeout_ms = 8000
}

• Up to seven fallback routes
• Each route needs a credential for its provider
• The policy is authoritative: a route that the policy's allowed and denied lists exclude is not saved and is never used

Switching providers halfway would stitch one reply from two models. Fallback happens only while nothing has been sent. Streaming replies stay whole, and the first output timeout is what gives a stuck call a chance to fail over before it starts.

A budget reserves the worst case for a request: the most expensive attempts your settings allow, up to your total attempts. Reliability never quietly spends past a limit.

1. 1

   Create a test policy

   Allow two providers that serve the same model.

2. 2

   Bind a bad key to the first provider

   Add a credential with a wrong key for the primary provider, and a good one for the second.

3. 3

   Ask by model name

   Send a request with the plain model name.

4. 4

   Check the answer and the Explorer

   The request succeeds. Group by Provider and the spend shows on the second provider.

5. 5

   Clean up

   Fix or remove the bad credential.

1. 1

   Start with defaults

   Turn on fallback by asking for models by name.

2. 2

   Add backoff and deadlines

   Set a request timeout your users accept.

3. 3

   Add explicit fallbacks

   Only where you need a specific order.

4. 4

   Watch Provider spend

   A sudden shift in the Explorer's Provider grouping means failover is happening.