Set Request and Token Rate Limits

• A request limit and a token limit per minute
• Numbers based on your own traffic, not guesses
• A separate allowance for every caller
• A readable response and a retry pattern for developers

You need an owner or admin role and a policy bound to your traffic.

1. 1

   Find a busy hour

   In AI → Explorer, group by App and choose 24h. Use Cost Trend to spot the busiest hour, and Spend Breakdown for the day's requests and tokens.

2. 2

   Turn it into a per-minute number

   Divide the requests in that hour by 60, then double the result for bursts.

3. 3

   Size the token limit

   Multiply that per-minute request peak by the average tokens a request uses.

4. 4

   Add headroom

   Round up. A limit that sits at your peak blocks real work.

1. 1

   Open the policy

   Go to AI → Policies and open the policy that governs your traffic.

2. 2

   Open Throughput limits

   On Advanced, find the Throughput limits section.

3. 3

   Enter the limits

   Fill in Request rate limit / min and Token rate limit / min. Leave a field empty for no limit.

4. 4

   Set Enforcement mode to Enforce

   Rate limits take effect on an enforcing policy. Switch one app's policy first, then widen.

5. 5

   Save

   The limits apply to new requests at once.

• Retry only on HTTP 429, never on a 402 or 403
• Wait, then retry, and back off further each time
• Add a little random jitter so many workers do not retry together
• Queue batch work instead of sending it all at once

import random
import time
from openai import OpenAI, APIStatusError

client = OpenAI(base_url="https://api.cloptima.ai/v1/ai", api_key="clop_vk_...")

def ask(prompt, attempts=4):
    for attempt in range(attempts):
        try:
            return client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": prompt}],
            )
        except APIStatusError as error:
            if error.status_code != 429 or attempt == attempts - 1:
                raise
            time.sleep(2 ** attempt + random.random())  # back off, with jitter

1. 1

   Set a low limit

   On a test policy, set Request rate limit / min to 10 and Mode to Enforce. Bind it to a test key.

2. 2

   Send a burst

   Send 30 requests in under a minute.

3. 3

   Read the result

   The first requests succeed. Later ones return 429 with request_rate_limit_exceeded.

for i in $(seq 1 30); do
  curl -s -o /dev/null -w "%{http_code}\n" https://api.cloptima.ai/v1/ai/chat/completions \
    -H "Authorization: Bearer $CLOPTIMA_VIRTUAL_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "gpt-4o-mini", "max_tokens": 16, "messages": [{"role": "user", "content": "Say hello"}]}'
done

1. 1

   Start with the Request Simulator

   Check that the right policy applies to the app's key.

2. 2

   Enforce on one app

   Set the limits on a policy bound to a single app and switch it to Enforce.

3. 3

   Watch the Blocked column

   In the Explorer, a few 429s at your peak mean the limit is close. Raise it if real work is waiting.

4. 4

   Widen the binding

   Bind the policy to more apps.