Rate limits and spending limits

Two kinds of limit apply to API calls. Rate limits slow you down for a moment and clear by themselves. Spending limits stop calls until you raise them.

Rate limits#

• Per key, per minute. Each API key may start a limited number of requests in a minute.
• Per account, at once. An account may have a limited number of calls running at the same time, across all its keys and the chat.
• A busy model. A model that is serving as much as it can answers 429 for a moment too.

A call over a rate limit is refused before it runs and is not charged. The answer is 429 with the type rate_limit_error, and a Retry-After header with the seconds to wait. When a limit refused it, error.code says which:

Spread work across time rather than in bursts, and keep the number of parallel calls modest. Streamed calls count as running until the stream ends.

Back off when told to#

Honour Retry-After; where it is absent, wait longer after each failure and add a little randomness so many clients do not retry together. OpenAI's clients do this for you: they retry 429 and the temporary failures a few times, and you can raise the number.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://sven.swedishembedded.com/v1",
    api_key=os.environ["SVEN_API_KEY"],
    max_retries=5,
)

To make a retry free of the risk of a second charge, add an Idempotency-Key. See Safe retries.

Spending limits#

A spending limit refuses a call with 402, never 429, because waiting does not clear it:

• A key's lifetime spending limit (key_spend_limit). Raise or remove it on the API keys page. See Authentication and API keys.
• The account's monthly budget (monthly_budget). Raise it under Billing, or wait for the next month.
• The account's credit itself, which has no code. Add credit.

Each of these is checked before the call runs, against the most the call could cost. See How calls are charged.