Rate limits

Understand how Prem enforces limits and how to handle a 429.

Rate limits restrict the number of requests that a client can send to the API in a given period. This page explains the mechanism and the error handling. For the numbers of each tier, see Tiers.

The purpose of rate limits

We apply rate limits for three reasons. They protect the API against abuse, because a malicious actor can send too many requests and make the service unavailable. They make access fair for all users, because one organization cannot make the API slow for the other users. They hold the total load on our infrastructure at a level that gives a constant response time.

The operation of rate limits

We measure rate limits in three ways:

• RPS (requests per second), for every endpoint
• TPM (tokens per minute), for the chat completion endpoints
• Audio seconds per minute, for the transcription and translation endpoints

You can get to a limit on any of the three. The first limit that you get to stops the request. As an example, take the Base tier: 5 inference requests per second, and 38,000 tokens per minute. Five small requests in one second fill your request bucket, although those requests use very few tokens.

These points are important too:

• The request limit applies to your organization. All users and API keys in an organization draw on one request bucket.
• The token limit and the audio limit apply to each member. Prem tracks these two buckets for the member that owns the API key, not for the organization as a whole.
• Each endpoint has its own request allowance. Requests to one endpoint do not consume the allowance of another.
• Rate limits change with your tier. See Tiers for the capacity of each tier.

The token bucket

Each limit is a bucket with a capacity and a refill rate:

• The capacity is the largest burst that you can send at one time.
• Each request takes 1 token from the request bucket. Each inference request takes its total token count from the token bucket.
• The bucket refills at a constant rate. This lets you send short bursts, but it stops continuous overuse.

Identifier scope

The rate limiter uses the most specific identifier that is available:

• Organization ID: the primary scope. Your API key supplies it.
• User ID: the rate limiter uses this if there is no organization context
• IP Address: the rate limiter uses this only for unauthenticated requests

So all requests from the API keys of one organization draw on the same request bucket.

When you get to a limit

An empty request bucket returns a 429 Too Many Requests status with a Retry-After header in seconds:

{
  "status": 429,
  "data": null,
  "error": "Rate limit exceeded, try again in 1 seconds",
  "message": null,
  "env": "production",
  "log": null,
  "validator": null,
  "support_id": "req_support_019a3f7c8d2e7b41a0c6"
}

Record the support_id. Send it to support with a timestamp if you need us to trace a request.

The Retry-After value on this response is always 1, because every bucket refills at one request per second or faster. Do not use the value as a fixed back-off budget. Use it as a minimum. Then apply your own exponential back-off if the next attempt is also unsuccessful.

Token and audio limits return a 400, not a 429

An empty token bucket or audio bucket does not return a 429. The API refuses the request with a 400 status and one of these error strings:

Bucket	Error string
Tokens per minute	TOKEN_LIMIT_EXCEEDED
Audio seconds per minute	AUDIO_DURATION_LIMIT_EXCEEDED

The TOKEN_LIMIT_EXCEEDED response holds your current usage in data.usage:

{
  "status": 400,
  "error": "TOKEN_LIMIT_EXCEEDED",
  "data": {
    "usage": {
      "tier": { "code": "TIER_1", "name": "Tier 1" },
      "usage": {
        "tokens": [
          {
            "period_sec": 60,
            "tokens_available": 0,
            "tokens_limit": 250000,
            "tokens_used": 250000,
            "tokens_refill": 250000
          }
        ]
      }
    }
  }
}

A client that watches only for 429 misses both of these errors. Branch on the error string too. Then wait for the start of the next minute before you send the request again.

An empty bucket stays closed until it refills to 20% of your limit. A retry before that point receives the same 400.

Other 429 responses

The rate limiter returns the shape in the example. Not every 429 that you receive looks like it.

Condition	Error string	Retry-After
Your API key reached its concurrent request limit	Concurrent request limit reached. Your limit allows 5 concurrent request(s). Wait for ongoing requests to complete, or contact us for higher limits.	No
A traffic surge at the model that you called	We are experiencing a traffic surge. Try again later.	Yes

A rate limit can also happen upstream of Prem. Those responses have a 429 status with a generic body and no Retry-After header:

{
  "error": {
    "message": "Internal server error",
    "type": "server_error"
  }
}

This body looks like a 500, but the status code is still 429. Automated clients must obey these rules:

• Branch on the HTTP status code, not the response body. Any 429 is a rate limit, whatever the error.type or the message text says.
• Do not expect a Retry-After header. Use your own exponential back-off with jitter when the header is not there.

If you see 429 responses without a Retry-After header, or with an unexpected body, contact us with the support_id and a timestamp.

Error mitigation

Steps to decrease rate limit errors

Be careful when you give users programmatic access, bulk processing features, and automated posting. Enable these features only for trusted customers.

Set a usage limit for each user in a specified period. This protects against automated and high-volume misuse. Add a hard cap or a manual review process for users who go above the limit.

Retry with exponential backoff

One simple way to prevent rate limit errors is to retry requests automatically with a random exponential backoff. With this method, the client sleeps for a short time when a rate limit error occurs. Then the client retries the unsuccessful request. If the request is again unsuccessful, the client increases the sleep time and retries again. This continues until the request is successful or until the client reaches a maximum number of retries. This approach has many benefits:

• Automatic retries let you recover from rate limit errors without crashes or lost data
• With exponential backoff, the first retries occur quickly, and later retries have longer delays
• Random jitter in the delay prevents all retries from occurring at the same time

Unsuccessful requests count against your rate limit. Do not send the same request again and again without a delay.

Example:

This implementation uses the Retry-After header from the API response. It uses exponential backoff when the response has no such header:

import createRvencClient from "@premai/api-sdk";

async function retryWithExponentialBackoff<T>(
  fn: () => Promise<T>,
  maxRetries: number = 6,
  baseDelay: number = 1000
): Promise<T> {
  let retries = 0;

  while (true) {
    try {
      return await fn();
    } catch (error: any) {
      const isRateLimitError = error?.status === 429;

      if (!isRateLimitError || retries >= maxRetries) {
        throw error;
      }

      // Some 429 responses (e.g. from an upstream backend) omit Retry-After.
      // Fall back to exponential backoff instead of retrying immediately.
      const retryAfterHeader = error?.headers?.["retry-after"];
      const parsedRetryAfter = retryAfterHeader ? parseInt(retryAfterHeader, 10) : NaN;
      const delay = Number.isFinite(parsedRetryAfter)
        ? parsedRetryAfter * 1000
        : baseDelay * 2 ** retries;

      retries++;
      console.log(
        `Rate limited (support_id ${error?.support_id || "unknown"}). ` +
        `Retrying in ${Math.round(delay)}ms... (attempt ${retries}/${maxRetries})`
      );

      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
}

// Usage
async function main() {
  const client = await createRvencClient({
    apiKey: process.env.PREM_API_KEY,
    clientKEK: process.env.CLIENT_KEK
  });

  try {
    const response = await retryWithExponentialBackoff(
      () =>
        client.chat.completions.create({
          model: "deepseek-v4.1-flash-1017",
          messages: [
            { role: "system", content: "You are a helpful assistant." },
            { role: "user", content: "What is the capital of France?" },
          ],
        }),
      6,
      1000
    );

    console.log(response.choices[0].message.content);
  } catch (error) {
    console.error("Request failed after retries:", error);
  }
}

main().catch(console.error);

Control how many requests run at one time

Prem limits the number of inference requests that one API key can process at the same time. The limit applies to every inference endpoint, and it is separate from the request and token buckets. If your application sends many requests at one time, put them in a queue. A queue processes the requests in sequence or in controlled batches.

import createRvencClient from "@premai/api-sdk";

class RequestQueue {
  private maxConcurrent: number;
  private running: number = 0;
  private queue: Array<() => void> = [];

  constructor(maxConcurrent: number = 3) {
    this.maxConcurrent = maxConcurrent;
  }

  async execute<T>(requestFn: () => Promise<T>): Promise<T> {
    while (this.running >= this.maxConcurrent) {
      await new Promise<void>((resolve) => this.queue.push(resolve));
    }

    this.running++;
    try {
      return await requestFn();
    } finally {
      this.running--;
      const resolve = this.queue.shift();
      if (resolve) resolve();
    }
  }
}

// Usage
async function main() {
  const client = await createRvencClient({
    apiKey: process.env.PREM_API_KEY,
    clientKEK: process.env.CLIENT_KEK
  });
  const queue = new RequestQueue(3); // max 3 concurrent requests

  const prompts = [
    "Explain photosynthesis",
    "What is machine learning?",
    "Describe the water cycle",
    "Explain gravity",
    "What is DNA?",
  ];

  try {
    const results = await Promise.all(
      prompts.map((prompt) =>
        queue.execute(() =>
          client.chat.completions.create({
            model: "deepseek-v4.1-flash-1017",
            messages: [
              { role: "system", content: "You are a helpful assistant." },
              { role: "user", content: prompt },
            ],
          })
        )
      )
    );

    results.forEach((response, index) => {
      console.log(`Response ${index + 1}:`, response.choices[0].message.content);
    });
  } catch (error) {
    console.error("One or more requests failed:", error);
  }
}

main().catch(console.error);

Tips for developers

• Group related operations to decrease the number of requests.
• Cache frequently accessed data so that you do not request the same data again and again.
• Branch on the status code, not on the shape of the error body.
• Use the Retry-After header when the response has one. Apply your own back-off when the response has none.
• Watch for TOKEN_LIMIT_EXCEEDED and AUDIO_DURATION_LIMIT_EXCEEDED on 400 responses, not only for 429.
• Log the support_id for each unsuccessful request.

Higher rate limits

Your limits increase when you move to a higher tier. See Tiers.

If you need higher limits, contact us for an enterprise plan with custom rate limits.

Related

Tiers

The request, token, and audio limits of each tier.

Errors

Every status code that the API returns, and what to do about it.