Multimodal APIs — Vision, Audio and Speech

Send images and documents to vision-capable models, transcribe and generate speech, and design features that combine text, images and audio.

Est. 35 min Updated Suggest a fix
// Part 01 — Concept

A Multimodal Request Has More Than One Kind of Input

A text-only model receives text and returns text. A multimodal feature may combine text with an image, a scanned page, an audio recording, or generated speech. The application still sends a request to a model API, but the content parts and processing constraints differ. You need to know what is sent, how large it is, whether the provider can process it, and what the response means before you build a user-facing workflow.

Think of a campus help desk that accepts a photo of a fee receipt and a spoken question. The image may contain a student name or transaction reference; the audio may capture other people in the room. Make the user intent explicit, request only the necessary media, and avoid treating model output as verified financial data. If a number affects a payment or account, validate it against an authoritative system and ask for confirmation.

ModalityTypical taskImportant boundary
Image inputDescribe a chart or extract visible textImage quality, crop, resolution and private content affect results
Audio inputTranscribe a lecture question or support messageConsent, language, noise and speaker ambiguity matter
Text to speechRead an approved answer aloudGenerated speech is output, not proof that the answer is correct
Mixed inputAnswer a question about a picture or recordingKeep each content item tied to its source and permissions
💡 Note
Multimodal does not mean "understands everything in every file." A model can miss small text, confuse chart axes, or mishear a name. Test on the media your users actually submit and design a useful fallback.
// Part 02 — Plan

Choose the Right Processing Path

Start with the user outcome, not the model feature list. If the task is transcription, a speech-to-text API often returns text that you can review, search, or summarize. If a question depends on both a picture and its caption, send both to a vision-capable model in one request. For long documents, first determine whether the API supports the file and page layout you need; converting a PDF to page images can be a fallback, but it changes cost and privacy considerations.

  1. Define the exact decision or response the feature supports, and identify what must remain human-reviewed.
  2. Check supported file types, size limits, languages, regions and retention terms in current provider documentation.
  3. Validate file type and size on your server; do not trust a filename or browser-provided content type alone.
  4. Minimize the media: crop irrelevant regions, shorten recordings, and remove unrelated pages where appropriate.
  5. Record provenance, consent and processing status without storing raw media longer than the feature needs.
🎯 Pro Tip
Separate media handling from the prompt. A small adapter can validate, normalize and pass media to the provider while the rest of the application keeps the same task interface.
// Part 03 — Vision

Send a Local Image with a Text Question

This example reads an image from disk, encodes it as a data URL, and submits a question with the image as a content part. The OpenAI Python SDK and Responses API are used as one concrete path. Supply the vision model through OPENAI_VISION_MODEL; model availability and accepted image formats depend on your account and the provider's current documentation.

ask_about_image.py
python
import base64
import mimetypes
import os
from pathlib import Path

from openai import OpenAI


def data_url_for(path: Path) -> str:
    media_type, _ = mimetypes.guess_type(path.name)
    if media_type not in {"image/jpeg", "image/png", "image/webp", "image/gif"}:
        raise ValueError("Choose a supported image format")
    encoded = base64.b64encode(path.read_bytes()).decode("ascii")
    return f"data:{media_type};base64,{encoded}"


client = OpenAI()
image_url = data_url_for(Path("receipt.jpg"))
response = client.responses.create(
    model=os.environ["OPENAI_VISION_MODEL"],
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "Describe the visible fields. Do not infer missing values."},
            {"type": "input_image", "image_url": image_url},
        ],
    }],
)
print(response.output_text)

The example is convenient for a small local file, not a production upload endpoint. Check the byte size before reading the whole file, authenticate the user, store uploads outside a public directory, and use a provider-supported file or URL flow where appropriate. Avoid logging a data URL: it contains the complete image payload.

// Part 04 — Documents

Treat Page Images and Extracted Text as Different Evidence

A scanned document has pixels; a digitally generated PDF may also have an accessible text layer and layout. OCR can make searchable text, but tables, stamps and reading order may be lost. A vision model may describe the page, but its extraction can still be wrong. Choose extraction according to the task, preserve page numbers and source references, and show the original excerpt when a learner or support agent needs to verify an answer.

document_job.py
python
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class PageEvidence:
    page_number: int
    text: str
    source_file: str
    extraction_method: str


def prepare_document(path: Path, *, max_bytes: int) -> list[PageEvidence]:
    if not path.is_file() or path.stat().st_size > max_bytes:
        raise ValueError("File is missing or exceeds the upload limit")
    if path.suffix.lower() not in {".pdf", ".png", ".jpg", ".jpeg"}:
        raise ValueError("Unsupported document type")

    # Delegate PDF parsing or OCR to a vetted, sandboxed service/library.
    # Return one evidence record per page and retain page-level provenance.
    return extract_pages(path)


pages = prepare_document(Path("handbook.pdf"), max_bytes=8_000_000)
for page in pages:
    print(page.page_number, page.extraction_method, page.text[:120])

extract_pages is an adapter boundary to your selected PDF or OCR tool; its implementation and limits depend on that tool. Never parse an untrusted upload with a privileged process that can access unrelated files or network services. For a policy assistant, answer with a document title and page reference, then let the user open the original page to confirm the wording.

// Part 05 — Speech-to-text

Transcribe Audio and Keep the Original Around for Review

Transcription converts speech into text; it does not certify who spoke or whether the words are true. Language mixing, background noise, overlapping speakers and names can all affect the result. For a support ticket, let the person edit the transcription before sending it. Keep the raw recording only if there is a documented need and a clear retention policy.

transcribe_audio.py
python
import os
from pathlib import Path

from openai import OpenAI


client = OpenAI()
audio_path = Path("question.m4a")
if not audio_path.is_file():
    raise FileNotFoundError(audio_path)

with audio_path.open("rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model=os.environ["OPENAI_TRANSCRIPTION_MODEL"],
        file=audio_file,
    )

print(transcript.text)

Use a model name from configuration and check the current audio API documentation for supported formats, request sizes, language hints and response options. Do not silently drop low-confidence or unintelligible sections; mark them for review. In a UPI support workflow, a transcribed transaction reference should be checked against the payment system rather than copied directly into an action.

⚠️ Important
A recording can contain personal or financial information even when the user describes it as a simple voice note. Explain what will be processed, restrict who can access it, and avoid sending unrelated audio to an external service.
// Part 06 — Text-to-speech

Generate Speech from an Approved Text Answer

Text-to-speech is useful for accessibility and hands-free use. Generate audio only after your application has applied its normal policy and validation to the text. If your app speaks a payment status, source that status from the bank or payment provider and format it safely; a language model should not invent transaction state.

speak_answer.py
python
import os
from pathlib import Path

from openai import OpenAI


client = OpenAI()
answer = "Your support request has been received. Check the app for updates."
audio = client.audio.speech.create(
    model=os.environ["OPENAI_TTS_MODEL"],
    voice=os.environ["OPENAI_TTS_VOICE"],
    input=answer,
)
audio.write_to_file(Path("answer.mp3"))

Voice identifiers and supported output formats vary by provider and can change, so configure them and check the provider's documentation. If users can download audio, apply the same authorization and retention rules as for the underlying answer. Let users choose text instead, and do not imitate a real person's voice without clear permission.

// Part 07 — Production

Build a Safe, Observable Media Pipeline

Media work adds bytes, processing time and privacy exposure to a feature. Apply authentication and authorization before accepting the upload; enforce request and duration limits; use an allowlist of formats; and scan or sandbox file processing. A background job can make long transcription or document extraction resilient, while a visible status keeps the user informed. Keep provider credentials on the server.

ConcernPractical controlWhat to measure
LatencyQueue long jobs and report progressUpload-to-result time and failure rate
QualityTest representative languages, scans and noiseEdit rate, missed fields and user corrections
PrivacyMinimize retention and restrict accessStored bytes, access events and deletion completion
CostBound media size and avoid duplicate processingRequests, input size and spend by workflow
  1. Store each upload under an opaque ID and associate it with the authenticated owner.
  2. Validate type, dimensions or duration and reject unsupported or oversized media before provider calls.
  3. Send only the required content using server-side credentials and configured model identifiers.
  4. Store the result with a source reference, processing status and retention deadline.
  5. Allow review, correction and deletion; log errors and metrics without logging raw media or secrets.
📌 Real World Example
A Swiggy-style delivery support flow can ask a rider to photograph a damaged package, but the model's description should only help triage. Refund eligibility remains an explicit business rule and a human can review ambiguous cases.
// Part 08 — Pitfalls

Common Mistakes

Trusting extracted text as exact

Why: Blur, handwriting, layout and accents can cause transcription or vision errors.

Fix: Show evidence, allow correction, and validate important values against a trusted source.

Accepting uploads by extension only

Why: The filename does not prove the actual file type or safe contents.

Fix: Validate content, size and format server-side, and isolate parsers for untrusted files.

Sending the entire recording or document by default

Why: Extra pages or audio increase exposure, latency and processing cost.

Fix: Minimize the media to the task and document retention and consent.

Putting provider keys in a browser app

Why: Users can inspect frontend bundles and network requests.

Fix: Proxy media processing through an authenticated server that owns credentials and limits.

// Part 09 — Practice

Prototype a Receipt and Voice-Note Helper

Practice

A review-first multimodal support tool

  1. Write a server-side image adapter that accepts one allowlisted format, rejects oversized files and asks a vision model only for visible fields.
  2. Add a transcription path that reads the model identifier from configuration and displays the text for user correction.
  3. Create a text-to-speech path for a fixed, verified status message; do not let the model create payment status.
  4. Design a small evaluation set with clear, blurry, rotated and partially obscured receipt images plus noisy audio samples.
  5. Document consent, access controls, retention and deletion for original files and derived text.

Hint: Keep provider calls behind adapters. Test file validation and authorization locally with a fake adapter before making paid requests.

🎯 Key Takeaways

  • ✓A multimodal API combines text with image or audio content, but every modality has different input and quality constraints.
  • ✓Treat model output as evidence to review, not as verified identity, financial state or ground truth.
  • ✓Validate and minimize uploads server-side; keep credentials and raw media away from public frontend code and logs.
  • ✓Use transcription for reviewable text and generate speech only from an approved answer.
  • ✓Track provenance, consent, retention, latency, failures and user corrections across the media pipeline.
  • ✓Configure model and voice identifiers and confirm current provider capabilities in official documentation.

Next

What comes next

You can accept images and audio. Next, turn text into vectors and retrieve semantically related passages from your own documents.

Lesson 06 · Embeddings in Practice — Chunking and Similarity Search →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...