Tokens, Pricing and Rate Limits
Count tokens, estimate the cost of a feature in rupees and dollars, and design around rate limits and timeouts.
A Token Is Not a Word
Providers measure model input and output in tokens. A token may be a whole short word, part of a long word, punctuation, or a piece of a non-English script. The same sentence can use a different number of tokens with different tokenizers. Indian-language text, code, JSON punctuation, and repeated conversation history can all have token counts that surprise you if you estimate from word count alone.
The request usually consumes input tokens for instructions, conversation, retrieved context, and tool results. The answer consumes output tokens. These quantities may have different prices. An API response often reports usage; log that observed count rather than relying on an estimate after the feature ships.
| Input component | Example | Cost implication |
|---|---|---|
| Instructions | Rules repeated on every request | Usually a stable input cost per call |
| User content | A question, document or support message | Varies with user behaviour |
| Conversation / retrieval | History or selected document chunks | Can grow unless explicitly bounded |
| Generated answer | The model completion | Output pricing and latency; set a useful limit |
Count Tokens Before You Send a Request
A local tokenizer is useful for prompt sizing, truncation checks, and comparing alternative templates. The following example uses tiktoken as an illustration. Choose the encoding documented for your model, and verify estimates against provider-reported usage because chat framing and provider-specific inputs may add tokens.
python -m pip install tiktokenimport tiktoken
text = "Help a student understand why their UPI payment is pending."
encoding = tiktoken.get_encoding("cl100k_base")
tokens = encoding.encode(text)
print("token count:", len(tokens))
print("token pieces:", [encoding.decode([token]) for token in tokens])Encoding names are tokenizer choices, not model names. If a provider offers a tokenizer endpoint or official counting helper for the selected model, prefer that for more accurate preflight checks. For actual billing, reconcile with usage reported by the API and your provider's current pricing page.
Estimate Monthly Cost in Rupees
Cost planning is multiplication, not guesswork. Estimate the number of calls per month and average input and output tokens per call; multiply each token count by its respective price per million tokens. For a rupee estimate, convert the resulting foreign-currency amount using an exchange rate that your finance configuration owns. Keep both prices and the exchange rate in deployment configuration so a pricing change does not require a code edit.
# Replace these placeholders with the current values for your selected model.
INPUT_USD_PER_MILLION=0.00
OUTPUT_USD_PER_MILLION=0.00
USD_TO_INR=0.00
MONTHLY_CALLS=10000
AVG_INPUT_TOKENS=1200
AVG_OUTPUT_TOKENS=250import os
calls = int(os.environ["MONTHLY_CALLS"])
input_tokens = int(os.environ["AVG_INPUT_TOKENS"])
output_tokens = int(os.environ["AVG_OUTPUT_TOKENS"])
input_price = float(os.environ["INPUT_USD_PER_MILLION"])
output_price = float(os.environ["OUTPUT_USD_PER_MILLION"])
usd_to_inr = float(os.environ["USD_TO_INR"])
monthly_input_tokens = calls * input_tokens
monthly_output_tokens = calls * output_tokens
usd = (
monthly_input_tokens / 1_000_000 * input_price
+ monthly_output_tokens / 1_000_000 * output_price
)
print(f"Estimated monthly model cost: ${usd:.2f} / ₹{usd * usd_to_inr:,.2f}")The zeroes above are deliberately placeholders, not quoted prices. Populate them from the current official pricing page for the chosen model, and keep an audit date with that configuration. Production estimates should also include retries, evaluation calls, embeddings, cached-input rules if applicable, and taxes or other platform costs where relevant. Add a safety margin based on observed variability, not an unexplained multiplier.
- Measure prompt and answer token distributions from realistic examples.
- Estimate monthly calls by feature and include expected retries and background jobs.
- Read current input and output prices for the configured model into deployment config.
- Compare the estimate with provider usage and actual invoices after launch; update assumptions.
RPM and TPM Are Different Limits
Rate limits protect shared infrastructure and are commonly expressed as requests per minute (RPM) and tokens per minute (TPM). A service can stay under its request quota while exceeding its token quota with a few very large prompts. Conversely, many tiny calls can exhaust RPM first. Providers can also apply other limits, such as concurrent requests, daily quotas, or model-specific capacity. Read the current account and model limits; do not assume one universal number.
| Limit | What is counted | Useful response |
|---|---|---|
| RPM | Requests during a time window | Queue work, reuse results, or safely combine small tasks |
| TPM | Tokens sent and/or generated during a time window | Bound context and output; pace by token budget |
| Concurrency | Requests in flight at once | Use a bounded worker pool and connection limits |
| Quota / budget | Account-level spend or usage over a longer period | Alerts, budgets, graceful degradation, and owner escalation |
A useful scheduler reserves capacity before dispatch: estimate input plus a conservative output allowance, then update the estimate with actual usage. Keep a queue for work that can wait, and give interactive requests priority without allowing background jobs to starve them.
Batching, Caching and Context Budgets
Batching can reduce per-request overhead when several independent tasks are ready, and some providers offer offline batch facilities with their own limits and latency trade-offs. Batch only requests whose results can be consumed together; one slow item should not hold an urgent interactive response hostage. Preserve an identifier for every input so results can be matched to the right record.
Caching avoids repeating identical work. A deterministic exact-response cache can use a key derived from the model, normalized input, instructions, and relevant settings. Do not cache personalized or sensitive output across users. Set a suitable expiration, invalidate on source changes, and decide whether the quality and freshness trade-off is acceptable. Semantic caching is more complex and requires separate relevance checks.
from collections.abc import Iterable
MAX_ITEMS = 20
def batches(items: Iterable[str], size: int = MAX_ITEMS):
if size < 1:
raise ValueError("size must be positive")
batch = []
for item in items:
batch.append(item)
if len(batch) == size:
yield batch
batch = []
if batch:
yield batch
for group in batches(["record-a", "record-b", "record-c"], size=2):
print("submit a provider-supported batch:", group)Reducing the request can improve quality as well as cost: retain only the conversation turns needed to answer the current question, retrieve a few relevant chunks instead of a whole document, and set an output limit appropriate for the task. Avoid silently dropping crucial policy or user-provided context just to hit a budget.
Build a Budget and a Backpressure Plan
Measure input tokens, output tokens, latency, retries, and 429s by feature and configured model. Aggregate these metrics rather than logging raw prompts by default; prompts may contain personal or confidential data. Set alerts for unusual spend and quota exhaustion, and decide what the product does when the provider is unavailable: queue, offer a non-AI fallback, or return a clear retry-later message.
import random
import time
from collections.abc import Callable
# The operation should raise the provider SDK's rate-limit exception on 429.
def with_backoff(operation: Callable[[], str], attempts: int = 4) -> str:
for attempt in range(attempts):
try:
return operation()
except Exception as error:
# In a real adapter, catch only the provider's rate-limit and
# transient server exceptions; let validation/auth errors surface.
if not is_retryable_provider_error(error) or attempt == attempts - 1:
raise
delay = min(30.0, 2 ** attempt) + random.uniform(0, 0.5)
time.sleep(delay)
raise RuntimeError("unreachable")is_retryable_provider_error is intentionally an adapter boundary: implement it with the exceptions from your installed SDK. Do not copy a made-up exception class. Also avoid retrying a request with a non-idempotent side effect unless your application has an idempotency strategy. Every retry consumes time and may consume tokens.
Common Cost and Capacity Mistakes
Estimate tokens by counting words
Why: Token boundaries vary across text, code, scripts, and tokenizers.
Fix: Use an appropriate tokenizer for sizing and verify with provider-reported usage.
Use a price copied into source code
Why: Prices and model availability can change, and input/output may differ.
Fix: Read placeholder values from deployment config populated from current official pricing.
Track RPM but ignore TPM
Why: A few long requests can exhaust a token budget even with low request volume.
Fix: Budget both request count and token count, including output allowance.
Retry immediately forever
Why: Retries add load and spend while preventing the queue from recovering.
Fix: Use bounded retries, jitter, provider guidance, and a fallback or queue policy.
Plan a UPI Support Assistant Budget
Practice
Estimate, load-test and control a monthly budget
- Choose three realistic support prompts and count their input tokens with the tokenizer for your provider/model; compare estimates with API usage when safely available.
- Run the cost script with clearly labelled test configuration values, then replace them with prices from the selected model’s current official pricing page before making a real estimate.
- Estimate a month for low, expected and peak call volumes. Include input and output separately and state the assumptions.
- Design a request queue that limits both concurrent requests and token throughput; decide which work is interactive and which can wait.
- Describe an exact-cache key and expiration policy that cannot leak one customer’s answer to another customer.
Hint: Keep all numeric prices and exchange rates in configuration. A good estimate explains its assumptions and gets compared with measured usage.
🎯 Key Takeaways
- ✓Tokens are model-specific units, not words; measure representative input and output.
- ✓Use tokenizer counts for sizing and provider-reported usage for reconciliation.
- ✓Estimate spend with calls × token counts × configured price per million, then convert to INR using a configured exchange rate.
- ✓Prices in examples are placeholders; verify current official prices and account limits before deployment.
- ✓RPM, TPM, concurrency, and account quotas are distinct constraints and need separate controls.
- ✓Batch independent work, cache only safe and valid results, and apply bounded retries with backoff.
Next
What comes next
Now that requests fit your budget, learn to turn model responses into validated data your application can safely use.
Lesson 03 · Structured Outputs with JSON Schema and Pydantic →HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
