Tokens, Pricing and Rate Limits

Count tokens, estimate the cost of a feature in rupees and dollars, and design around rate limits and timeouts.

Est. 25 min Updated Suggest a fix
// Part 01 — Mental model

A Token Is Not a Word

Providers measure model input and output in tokens. A token may be a whole short word, part of a long word, punctuation, or a piece of a non-English script. The same sentence can use a different number of tokens with different tokenizers. Indian-language text, code, JSON punctuation, and repeated conversation history can all have token counts that surprise you if you estimate from word count alone.

The request usually consumes input tokens for instructions, conversation, retrieved context, and tool results. The answer consumes output tokens. These quantities may have different prices. An API response often reports usage; log that observed count rather than relying on an estimate after the feature ships.

Think in measured tokens for each request, not in vague page or word counts.
Input componentExampleCost implication
InstructionsRules repeated on every requestUsually a stable input cost per call
User contentA question, document or support messageVaries with user behaviour
Conversation / retrievalHistory or selected document chunksCan grow unless explicitly bounded
Generated answerThe model completionOutput pricing and latency; set a useful limit
💡 Note
Tokenisation is model-family dependent. Use the tokenizer or usage field associated with the provider and model you actually call; treat a different tokenizer's result as an estimate, not a billing record.
// Part 02 — Inspect

Count Tokens Before You Send a Request

A local tokenizer is useful for prompt sizing, truncation checks, and comparing alternative templates. The following example uses tiktoken as an illustration. Choose the encoding documented for your model, and verify estimates against provider-reported usage because chat framing and provider-specific inputs may add tokens.

bash
python -m pip install tiktoken
count_tokens.py
python
import tiktoken

text = "Help a student understand why their UPI payment is pending."
encoding = tiktoken.get_encoding("cl100k_base")
tokens = encoding.encode(text)

print("token count:", len(tokens))
print("token pieces:", [encoding.decode([token]) for token in tokens])

Encoding names are tokenizer choices, not model names. If a provider offers a tokenizer endpoint or official counting helper for the selected model, prefer that for more accurate preflight checks. For actual billing, reconcile with usage reported by the API and your provider's current pricing page.

🎯 Pro Tip
Measure representative production-like examples, including long prompts, code, mixed English and Telugu, JSON, and empty or malformed inputs. A single short English sentence is not a useful sizing benchmark.
// Part 03 — Budget

Estimate Monthly Cost in Rupees

Cost planning is multiplication, not guesswork. Estimate the number of calls per month and average input and output tokens per call; multiply each token count by its respective price per million tokens. For a rupee estimate, convert the resulting foreign-currency amount using an exchange rate that your finance configuration owns. Keep both prices and the exchange rate in deployment configuration so a pricing change does not require a code edit.

.env.example
bash
# Replace these placeholders with the current values for your selected model.
INPUT_USD_PER_MILLION=0.00
OUTPUT_USD_PER_MILLION=0.00
USD_TO_INR=0.00
MONTHLY_CALLS=10000
AVG_INPUT_TOKENS=1200
AVG_OUTPUT_TOKENS=250
estimate_cost.py
python
import os

calls = int(os.environ["MONTHLY_CALLS"])
input_tokens = int(os.environ["AVG_INPUT_TOKENS"])
output_tokens = int(os.environ["AVG_OUTPUT_TOKENS"])
input_price = float(os.environ["INPUT_USD_PER_MILLION"])
output_price = float(os.environ["OUTPUT_USD_PER_MILLION"])
usd_to_inr = float(os.environ["USD_TO_INR"])

monthly_input_tokens = calls * input_tokens
monthly_output_tokens = calls * output_tokens
usd = (
    monthly_input_tokens / 1_000_000 * input_price
    + monthly_output_tokens / 1_000_000 * output_price
)
print(f"Estimated monthly model cost: ${usd:.2f} / ₹{usd * usd_to_inr:,.2f}")

The zeroes above are deliberately placeholders, not quoted prices. Populate them from the current official pricing page for the chosen model, and keep an audit date with that configuration. Production estimates should also include retries, evaluation calls, embeddings, cached-input rules if applicable, and taxes or other platform costs where relevant. Add a safety margin based on observed variability, not an unexplained multiplier.

  1. Measure prompt and answer token distributions from realistic examples.
  2. Estimate monthly calls by feature and include expected retries and background jobs.
  3. Read current input and output prices for the configured model into deployment config.
  4. Compare the estimate with provider usage and actual invoices after launch; update assumptions.
// Part 04 — Capacity

RPM and TPM Are Different Limits

Rate limits protect shared infrastructure and are commonly expressed as requests per minute (RPM) and tokens per minute (TPM). A service can stay under its request quota while exceeding its token quota with a few very large prompts. Conversely, many tiny calls can exhaust RPM first. Providers can also apply other limits, such as concurrent requests, daily quotas, or model-specific capacity. Read the current account and model limits; do not assume one universal number.

Exact limit dimensions and reset behaviour vary by provider and account.
LimitWhat is countedUseful response
RPMRequests during a time windowQueue work, reuse results, or safely combine small tasks
TPMTokens sent and/or generated during a time windowBound context and output; pace by token budget
ConcurrencyRequests in flight at onceUse a bounded worker pool and connection limits
Quota / budgetAccount-level spend or usage over a longer periodAlerts, budgets, graceful degradation, and owner escalation

A useful scheduler reserves capacity before dispatch: estimate input plus a conservative output allowance, then update the estimate with actual usage. Keep a queue for work that can wait, and give interactive requests priority without allowing background jobs to starve them.

⚠️ Important
A 429 response is not a cue to retry instantly in a tight loop. That creates more load, wastes quota, and can delay recovery. Honour provider retry guidance when available and apply exponential backoff with jitter.
// Part 05 — Efficiency

Batching, Caching and Context Budgets

Batching can reduce per-request overhead when several independent tasks are ready, and some providers offer offline batch facilities with their own limits and latency trade-offs. Batch only requests whose results can be consumed together; one slow item should not hold an urgent interactive response hostage. Preserve an identifier for every input so results can be matched to the right record.

Caching avoids repeating identical work. A deterministic exact-response cache can use a key derived from the model, normalized input, instructions, and relevant settings. Do not cache personalized or sensitive output across users. Set a suitable expiration, invalidate on source changes, and decide whether the quality and freshness trade-off is acceptable. Semantic caching is more complex and requires separate relevance checks.

bounded_work.py
python
from collections.abc import Iterable

MAX_ITEMS = 20

def batches(items: Iterable[str], size: int = MAX_ITEMS):
    if size < 1:
        raise ValueError("size must be positive")
    batch = []
    for item in items:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

for group in batches(["record-a", "record-b", "record-c"], size=2):
    print("submit a provider-supported batch:", group)

Reducing the request can improve quality as well as cost: retain only the conversation turns needed to answer the current question, retrieve a few relevant chunks instead of a whole document, and set an output limit appropriate for the task. Avoid silently dropping crucial policy or user-provided context just to hit a budget.

// Part 06 — Operations

Build a Budget and a Backpressure Plan

Measure input tokens, output tokens, latency, retries, and 429s by feature and configured model. Aggregate these metrics rather than logging raw prompts by default; prompts may contain personal or confidential data. Set alerts for unusual spend and quota exhaustion, and decide what the product does when the provider is unavailable: queue, offer a non-AI fallback, or return a clear retry-later message.

retry_after_rate_limit.py
python
import random
import time
from collections.abc import Callable

# The operation should raise the provider SDK's rate-limit exception on 429.
def with_backoff(operation: Callable[[], str], attempts: int = 4) -> str:
    for attempt in range(attempts):
        try:
            return operation()
        except Exception as error:
            # In a real adapter, catch only the provider's rate-limit and
            # transient server exceptions; let validation/auth errors surface.
            if not is_retryable_provider_error(error) or attempt == attempts - 1:
                raise
            delay = min(30.0, 2 ** attempt) + random.uniform(0, 0.5)
            time.sleep(delay)
    raise RuntimeError("unreachable")

is_retryable_provider_error is intentionally an adapter boundary: implement it with the exceptions from your installed SDK. Do not copy a made-up exception class. Also avoid retrying a request with a non-idempotent side effect unless your application has an idempotency strategy. Every retry consumes time and may consume tokens.

📌 Real World Example
For an order-support feature, show the order status already stored in your own system if model capacity is exhausted. The model can explain the status when available; it should not be the only route for users to see critical account data.
// Part 07 — Pitfalls

Common Cost and Capacity Mistakes

Estimate tokens by counting words

Why: Token boundaries vary across text, code, scripts, and tokenizers.

Fix: Use an appropriate tokenizer for sizing and verify with provider-reported usage.

Use a price copied into source code

Why: Prices and model availability can change, and input/output may differ.

Fix: Read placeholder values from deployment config populated from current official pricing.

Track RPM but ignore TPM

Why: A few long requests can exhaust a token budget even with low request volume.

Fix: Budget both request count and token count, including output allowance.

Retry immediately forever

Why: Retries add load and spend while preventing the queue from recovering.

Fix: Use bounded retries, jitter, provider guidance, and a fallback or queue policy.

// Part 08 — Practice

Plan a UPI Support Assistant Budget

Practice

Estimate, load-test and control a monthly budget

  1. Choose three realistic support prompts and count their input tokens with the tokenizer for your provider/model; compare estimates with API usage when safely available.
  2. Run the cost script with clearly labelled test configuration values, then replace them with prices from the selected model’s current official pricing page before making a real estimate.
  3. Estimate a month for low, expected and peak call volumes. Include input and output separately and state the assumptions.
  4. Design a request queue that limits both concurrent requests and token throughput; decide which work is interactive and which can wait.
  5. Describe an exact-cache key and expiration policy that cannot leak one customer’s answer to another customer.

Hint: Keep all numeric prices and exchange rates in configuration. A good estimate explains its assumptions and gets compared with measured usage.

🎯 Key Takeaways

  • ✓Tokens are model-specific units, not words; measure representative input and output.
  • ✓Use tokenizer counts for sizing and provider-reported usage for reconciliation.
  • ✓Estimate spend with calls × token counts × configured price per million, then convert to INR using a configured exchange rate.
  • ✓Prices in examples are placeholders; verify current official prices and account limits before deployment.
  • ✓RPM, TPM, concurrency, and account quotas are distinct constraints and need separate controls.
  • ✓Batch independent work, cache only safe and valid results, and apply bounded retries with backoff.

Next

What comes next

Now that requests fit your budget, learn to turn model responses into validated data your application can safely use.

Lesson 03 · Structured Outputs with JSON Schema and Pydantic →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...