Multimodal Models — CLIP, LLaVA, and Vision-Language

Think of a bilingual dictionary — it maps words from English to Spanish and back. CLIP is a bilingual dictionary between visual language and text language. Show it an image of a leather jacket and it gives you a vector. Show it the text "classic leather bomber jacket" and it gives you a similar vector. They are translations of the same concept into a shared numeric language. Similarity in this shared space means semantic similarity across modalities.

The critical insight: CLIP was trained on 400 million (image, text) pairs from the internet. It never needed explicit labels. The training signal came purely from the natural language captions that humans wrote alongside images. This is the largest self-supervised multimodal dataset ever assembled.

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
from PIL import Image

# ── CLIP contrastive loss from scratch ───────────────────────────────
class CLIPContrastiveLoss(nn.Module):
    def __init__(self, temperature: float = 0.07):
        super().__init__()
        self.log_temp = nn.Parameter(torch.log(torch.tensor(temperature)))

    def forward(self, image_emb: torch.Tensor,
                 text_emb: torch.Tensor) -> torch.Tensor:
        """
        image_emb: (B, D) — L2-normalised image embeddings
        text_emb:  (B, D) — L2-normalised text embeddings
        Returns: scalar contrastive loss
        """
        # Cosine similarity matrix (already normalised)
        temp    = self.log_temp.exp()
        sim_mat = (image_emb @ text_emb.T) * temp   # (B, B)

        # Ground truth: diagonal pairs are correct matches
        labels = torch.arange(sim_mat.size(0), device=sim_mat.device)

        # Symmetric cross-entropy
        loss_img = F.cross_entropy(sim_mat,   labels)   # each image → text
        loss_txt = F.cross_entropy(sim_mat.T, labels)   # each text → image

        return (loss_img + loss_txt) / 2

# ── Simulate CLIP training step ───────────────────────────────────────
torch.manual_seed(42)
BATCH = 8
DIM   = 512

# Random L2-normalised embeddings (in real CLIP: output of encoders)
img_emb = F.normalize(torch.randn(BATCH, DIM), dim=1)
txt_emb = F.normalize(torch.randn(BATCH, DIM), dim=1)

# Make pairs 0 and 1 more similar (simulate training signal)
txt_emb[0] = F.normalize(img_emb[0] + torch.randn(DIM) * 0.3, dim=0)
txt_emb[1] = F.normalize(img_emb[1] + torch.randn(DIM) * 0.3, dim=0)

criterion = CLIPContrastiveLoss(temperature=0.07)
loss      = criterion(img_emb, txt_emb)

# Compute similarity matrix for inspection
with torch.no_grad():
    sim = (img_emb @ txt_emb.T) * criterion.log_temp.exp()
    probs = F.softmax(sim, dim=1)

print(f"CLIP contrastive loss: {loss.item():.4f}")
print(f"Temperature: {criterion.log_temp.exp().item():.4f}")
print(f"
Similarity matrix (top-left 4×4):")
print(sim[:4, :4].numpy().round(3))
print(f"
Row 0 retrieval probs: {probs[0].numpy().round(3)}")
print(f"  Correct match (col 0) probability: {probs[0, 0].item():.4f}")

# ── Using pretrained CLIP from HuggingFace ────────────────────────────
print("""
from transformers import CLIPModel, CLIPProcessor

model     = CLIPModel.from_pretrained('openai/clip-vit-base-patch32')
processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')

# Zero-shot image classification — NO training on these categories needed
image  = Image.open('product.jpg')
labels = ['a red leather jacket', 'blue denim jeans', 'leather sneakers',
           'gold wristwatch', 'cotton dress']

inputs = processor(text=labels, images=image, return_tensors='pt', padding=True)
with torch.no_grad():
    outputs = model(**inputs)
    logits  = outputs.logits_per_image   # (1, n_labels)
    probs   = logits.softmax(dim=1)

for label, prob in zip(labels, probs[0]):
    print(f'  {label:<30}: {prob.item():.4f}')
""")

import torch
import torch.nn.functional as F
import numpy as np
from PIL import Image

# ── Application 1: Zero-shot image classification ─────────────────────
# Classify images into any categories without training examples
print("=" * 55)
print("1. ZERO-SHOT CLASSIFICATION")
print("=" * 55)
print("""
from transformers import CLIPModel, CLIPProcessor

model     = CLIPModel.from_pretrained('openai/clip-vit-base-patch32')
processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')
model.eval()

# Shopify: classify product photos into catalogue categories
CATEGORIES = [
    'a photo of a jacket or coat',
    'a photo of a dress',
    'a photo of jeans or trousers',
    'a photo of sneakers or sports shoes',
    'a photo of a wristwatch',
    'a photo of a handbag or purse',
]

def classify_product(image_path: str) -> tuple[str, float]:
    image  = Image.open(image_path).convert('RGB')
    inputs = processor(
        text=CATEGORIES, images=image,
        return_tensors='pt', padding=True,
    )
    with torch.no_grad():
        out   = model(**inputs)
        probs = out.logits_per_image.softmax(dim=1)[0]

    best_idx = probs.argmax().item()
    return CATEGORIES[best_idx], probs[best_idx].item()

category, confidence = classify_product('product.jpg')
print(f'Category: {category}  Confidence: {confidence:.3f}')
""")

# ── Application 2: Text-to-image retrieval ────────────────────────────
print("=" * 55)
print("2. TEXT-TO-IMAGE RETRIEVAL (semantic search)")
print("=" * 55)
print("""
# Build index from product images at indexing time (run once)
def build_image_index(image_paths: list, model, processor) -> torch.Tensor:
    all_embeddings = []
    for path in image_paths:
        img    = Image.open(path).convert('RGB')
        inputs = processor(images=img, return_tensors='pt')
        with torch.no_grad():
            emb = model.get_image_features(**inputs)
            emb = F.normalize(emb, dim=-1)
        all_embeddings.append(emb)
    return torch.cat(all_embeddings, dim=0)   # (N, 512)

# At query time
def search_by_text(query: str, image_index: torch.Tensor,
                    image_paths: list, model, processor,
                    top_k: int = 5) -> list:
    inputs = processor(text=[query], return_tensors='pt', padding=True)
    with torch.no_grad():
        txt_emb = model.get_text_features(**inputs)
        txt_emb = F.normalize(txt_emb, dim=-1)

    # Cosine similarity with all indexed images
    similarities = (image_index @ txt_emb.T).squeeze(-1)
    top_indices  = similarities.argsort(descending=True)[:top_k]

    return [
        {'path': image_paths[i], 'score': similarities[i].item()}
        for i in top_indices
    ]

# Example queries:
# 'red leather jacket with gold zipper'     → retrieves matching products
# 'casual blue jeans for men'              → retrieves casual jeans
# 'party wear dress for wedding ceremony'  → understands context
# 'same as the image' (not possible)      → need image query instead
""")

# ── Application 3: Image-to-image retrieval ────────────────────────────
print("=" * 55)
print("3. IMAGE-TO-IMAGE RETRIEVAL (visual similarity)")
print("=" * 55)
print("""
# Query with an image instead of text
def search_by_image(query_image_path: str, image_index: torch.Tensor,
                     image_paths: list, model, processor, top_k=5):
    query_img = Image.open(query_image_path).convert('RGB')
    inputs    = processor(images=query_img, return_tensors='pt')
    with torch.no_grad():
        query_emb = F.normalize(model.get_image_features(**inputs), dim=-1)

    similarities = (image_index @ query_emb.T).squeeze(-1)
    top_idx      = similarities.argsort(descending=True)[1:top_k+1]  # skip self

    return [{'path': image_paths[i], 'score': similarities[i].item()}
             for i in top_idx]

# Used by Shopify for 'similar products' recommendations
# User takes photo of product they like → retrieve visually similar products
""")

# ── Application 4: CLIP as a feature extractor ────────────────────────
print("4. CLIP FEATURES + DOWNSTREAM CLASSIFIER")
print("""
# Fine-tune a linear head on top of frozen CLIP features
# Much more efficient than training from scratch

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import LabelEncoder

def extract_clip_features(image_paths, model, processor):
    features = []
    for path in image_paths:
        img    = Image.open(path).convert('RGB')
        inputs = processor(images=img, return_tensors='pt')
        with torch.no_grad():
            emb = model.get_image_features(**inputs)
            emb = F.normalize(emb, dim=-1)
        features.append(emb.numpy())
    return np.vstack(features)

# Training: 50 examples per class → strong classification via linear probe
# X_train = extract_clip_features(train_paths, model, processor)
# clf     = LogisticRegression(max_iter=1000, C=1.0)
# clf.fit(X_train, train_labels)
# Accuracy: ~85% with 50 examples per class (vs ~40% training from scratch)
""")

import torch
import torch.nn as nn

# ── Minimal LLaVA projection layer ───────────────────────────────────
class LLaVAProjection(nn.Module):
    """
    Two-layer MLP that maps CLIP vision features into LLM embedding space.
    This is the only new component in LLaVA — everything else is pretrained.
    Training LLaVA = training this MLP (and optionally the LLM with LoRA).
    """
    def __init__(self, vision_dim: int = 1024, llm_dim: int = 4096):
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(vision_dim, llm_dim),
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim),
        )

    def forward(self, vision_features: torch.Tensor) -> torch.Tensor:
        """
        vision_features: (B, n_patches, vision_dim)  e.g. (1, 256, 1024)
        Returns:         (B, n_patches, llm_dim)      e.g. (1, 256, 4096)
        """
        return self.mlp(vision_features)

# ── Shape demonstration ───────────────────────────────────────────────
proj = LLaVAProjection(vision_dim=1024, llm_dim=4096)

# CLIP ViT-L/14 output: 256 patch tokens for a 336×336 image
vision_features = torch.randn(1, 256, 1024)
projected = proj(vision_features)

params = sum(p.numel() for p in proj.parameters())
print(f"LLaVA Projection MLP:")
print(f"  Vision features: {tuple(vision_features.shape)}")
print(f"  Projected:       {tuple(projected.shape)}")
print(f"  Parameters:      {params:,}  ← tiny relative to LLM")

# ── Using LLaVA for visual QA ─────────────────────────────────────────
print("""
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
from PIL import Image

# LLaVA-1.6 (LLaVA-NeXT) — best open-source VQA model
model_id  = 'llava-hf/llava-v1.6-mistral-7b-hf'
processor = LlavaNextProcessor.from_pretrained(model_id)
model     = LlavaNextForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map='auto',
)
model.eval()

# Product quality check — Amazon
image = Image.open('product_listing.jpg').convert('RGB')
conversation = [
    {
        'role': 'user',
        'content': [
            {'type': 'image'},
            {'type': 'text', 'text': (
                'This is a product listing image from an e-commerce platform. '
                'Answer these questions:
'
                '1. Does the image show the product clearly?
'
                '2. Is the background clean and professional?
'
                '3. Are there any visible defects or quality issues?
'
                '4. What is the main product category?
'
                'Be concise.'
            )},
        ],
    },
]

prompt  = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs  = processor(images=image, text=prompt, return_tensors='pt').to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=200, temperature=0.0, do_sample=False)

response = processor.decode(output[0][inputs['input_ids'].shape[1]:],
                              skip_special_tokens=True)
print(f"Quality assessment:\n{response}")
""")

import torch
import torch.nn.functional as F
import numpy as np

# ── Pattern 1: Multimodal product search ──────────────────────────────
print("PATTERN 1: MULTIMODAL PRODUCT SEARCH")
print("""
# Architecture: CLIP embeddings + FAISS index

class MultimodalProductSearch:
    def __init__(self, model, processor):
        self.model     = model
        self.processor = processor
        self.image_index   = None   # FAISS index
        self.product_data  = []     # metadata

    def index_product(self, image_path, product_id, name, price, category):
        img    = Image.open(image_path).convert('RGB')
        inputs = self.processor(images=img, return_tensors='pt')
        with torch.no_grad():
            emb = F.normalize(self.model.get_image_features(**inputs), dim=-1)
        self.product_data.append({
            'id': product_id, 'name': name,
            'price': price, 'category': category,
            'embedding': emb,
        })

    def search(self, query: str, top_k: int = 10):
        inputs = self.processor(text=[query], return_tensors='pt', padding=True)
        with torch.no_grad():
            q_emb = F.normalize(self.model.get_text_features(**inputs), dim=-1)

        # Score all products
        all_emb = torch.cat([p['embedding'] for p in self.product_data])
        scores  = (all_emb @ q_emb.T).squeeze(-1)
        top_idx = scores.argsort(descending=True)[:top_k]

        return [
            {**self.product_data[i], 'score': scores[i].item()}
            for i in top_idx
        ]

# Example queries that CLIP handles without category training:
queries = [
    'wedding guest dress under 200 dollars',      # price + occasion + category
    'office formal shirt for men light colour',   # style + context + colour
    'kids birthday party dress pink frilly',      # demographics + occasion
    'gym wear breathable fabric moisture wicking', # technical attributes
]
# CLIP understands all of these without explicit training on these labels
""")

# ── Pattern 2: Document understanding with LLaVA ─────────────────────
print("PATTERN 2: DOCUMENT UNDERSTANDING")
print("""
# Stripe: extract structured data from payment receipts
# No OCR pipeline needed — LLaVA reads the image directly

RECEIPT_PROMPT = '''
You are a payment receipt parser for Stripe.
Extract these fields from the receipt image as JSON:
{
  "merchant_name": string,
  "amount": number,
  "currency": "USD" or other,
  "transaction_id": string,
  "date": "YYYY-MM-DD",
  "payment_method": "ACH" or "card" or "wire" or other,
  "status": "success" or "failed" or "pending"
}
If a field is not visible, use null.
Respond with JSON only, no explanation.
'''

def parse_receipt(image_path, model, processor) -> dict:
    image = Image.open(image_path).convert('RGB')
    conversation = [
        {'role': 'user', 'content': [
            {'type': 'image'},
            {'type': 'text', 'text': RECEIPT_PROMPT},
        ]},
    ]
    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(images=image, text=prompt, return_tensors='pt').to(model.device)

    with torch.no_grad():
        output = model.generate(**inputs, max_new_tokens=200,
                                 temperature=0.0, do_sample=False)
    raw = processor.decode(output[0][inputs['input_ids'].shape[1]:],
                            skip_special_tokens=True)
    import json, re
    match = re.search(r'\{.*\}', raw, re.DOTALL)
    return json.loads(match.group()) if match else {}
""")

# ── Pattern 3: Quality control classifier ────────────────────────────
print("PATTERN 3: PRODUCT PHOTO QUALITY CONTROL")
print("""
# Shopify: auto-reject product listings with poor quality photos

QUALITY_CRITERIA = [
    'a high quality professional product photo on white background',
    'a blurry or out of focus product image',
    'a product photo with cluttered or messy background',
    'a product worn by a person in a lifestyle photo',
    'a product photo with watermark or logo overlay',
    'a very dark or underexposed product photo',
]

def check_photo_quality(image_path, model, processor) -> dict:
    image  = Image.open(image_path).convert('RGB')
    inputs = processor(text=QUALITY_CRITERIA, images=image,
                        return_tensors='pt', padding=True)
    with torch.no_grad():
        logits = model(**inputs).logits_per_image
        probs  = logits.softmax(dim=1)[0]

    best_idx = probs.argmax().item()
    return {
        'best_match':  QUALITY_CRITERIA[best_idx],
        'confidence':  probs[best_idx].item(),
        'approved':    best_idx == 0,   # only approve if best match is high quality
        'all_scores':  dict(zip(QUALITY_CRITERIA, probs.tolist())),
    }
""")

# ── Two-stage multimodal pipeline — the pattern behind most production systems ──

# Stage 1: CLIP narrows millions of candidates to a handful (cheap, fast)
def stage_one_filter(query_embedding, image_index, top_k=20):
    """~5-15ms even against a 5M-image FAISS index."""
    scores = image_index.search(query_embedding, top_k)
    return scores  # candidate set, not a final answer

# Stage 2: VLM only runs on the narrowed candidate set (slow, expensive)
def stage_two_verify(candidates, user_query, vlm_model):
    """
    ~2-5s PER CALL — only ever called on stage_one's small output,
    never on the full catalogue. This is what keeps VLM cost bounded:
    cost scales with candidates reviewed, not with catalogue size.
    """
    verified = []
    for candidate in candidates:
        result = vlm_model.answer(candidate.image, user_query)
        verified.append({'candidate': candidate, 'reasoning': result})
    return verified

# Rough production cost/latency budget for a search feature at scale:
#   Stage 1 (CLIP + ANN index):  ~10ms,  ~$0.0000001 per query
#   Stage 2 (VLM, top 20 only):  ~3s,    ~$0.02-0.05 per query
#   Running stage 2 on ALL candidates instead of top 20 would be
#   250,000x more VLM calls for the same traffic — this is the
#   entire economic argument for the two-stage design.

Simply feeding image captions into an LLM's text input is not what CLIP or LLaVA do, and it is not what makes a model genuinely multimodal. CLIP trains its image and text encoders jointly, in the same contrastive objective, so both learn to land in a shared geometric space. LLaVA's projection layer is trained specifically so the LLM can attend to visual tokens the same way it attends to text tokens. The defining feature is a shared representation learned end to end for both modalities together, not two independent models stapled at the API boundary with a text description passed between them.

Scale helps, but alignment is an explicit training objective, not a side effect of data volume. CLIP's contrastive loss is specifically engineered to pull matching pairs together and push non-matching pairs apart across the whole batch — without that objective, an image encoder and a text encoder trained separately would produce vector spaces that are not comparable at all, even on billions of examples. Getting alignment right also depends on caption quality (noisy web alt-text weakens it measurably), batch size (more negatives per batch produces a harder, more informative training signal), and temperature tuning. None of that is automatic.

In practice, CLIP-style embeddings show a measurable 'modality gap' — image embeddings and text embeddings cluster in separate regions of the shared space rather than fully overlapping, even for well-matched pairs. Cosine similarity across modalities still works because relative distances are preserved, but treating the space as if an image and its perfect caption should land at literally the same point is not how these models actually behave. This is measurable, has been studied directly, and matters in practice: pooling image and text embeddings together naively (for example averaging them into a single index) tends to underperform keeping the comparison directional.

Fluent, accurate-sounding descriptions and genuine understanding are not the same capability, and VLMs reliably demonstrate the gap. The hallucination failure mode documented earlier in this module — confidently describing details that are not in the image at all — happens precisely because the language model component is doing what language models do: generating plausible next tokens conditioned on the visual features it was given, not verifying claims against ground truth. A model can nail the general gist of a photo while inventing a specific brand name, a count of objects, or an exact piece of text that was never actually visible.

For pure retrieval and classification tasks, a well-tuned CLIP embedding index regularly beats routing every request through a large generative VLM — it is faster by roughly three orders of magnitude, costs a small fraction as much, and a similarity score is often literally all the task needs. Reaching for GPT-4o Vision or Gemini Vision on every request regardless of whether the task calls for free-form reasoning is a common and expensive default. The right model choice depends on what the output actually needs to be — a score, a category, or a written explanation — not on which model scores highest on a general benchmark.

There are roughly three families. Dual-encoder fusion, what CLIP does, keeps the two encoders entirely separate and only compares their outputs at the end via a similarity score — cheap, but limited to retrieval and classification, no generation. Cross-attention fusion, used by models like Flamingo, interleaves dedicated cross-attention layers into the language model so text tokens can attend directly to visual features at multiple depths of the network — more expressive, more expensive, and requires custom architecture changes. LLaVA takes a third, much cheaper path: a small trainable projection MLP maps vision features into the same embedding space the LLM already uses for text tokens, then simply concatenates them as if they were additional text tokens — the frozen LLM's existing self-attention does the fusion work it already knows how to do, with almost no new architecture required.

For a batch of N image-text pairs, CLIP computes the cosine similarity between every image and every text embedding, producing an N by N matrix. The N correct pairs sit on the diagonal; everything off the diagonal is a negative. The loss is symmetric cross-entropy applied twice — once treating each image as a classification problem over the N texts (which text matches this image), once treating each text as a classification problem over the N images — and the two are averaged. This works because it never needs explicit class labels, only naturally occurring (image, caption) pairs scraped from the web, and because larger batches supply more negative examples per step, which makes the discrimination task harder and produces a sharper, more useful embedding space.

Images and text carry information at fundamentally different densities and structures. A single photo contains far more raw information than its one-sentence caption captures — the caption is a lossy, human-chosen summary, so the same image could pair correctly with many different valid captions, and the same caption could plausibly match many different images. The model has to learn which parts of that huge visual signal actually correspond to the sparse textual signal, with no direct supervision pointing at which pixels matter. That many-to-many, unequal-information-density relationship is why naive approaches (like just averaging pixel and word embeddings) fail, and why it took a specifically designed contrastive objective at very large scale to make the alignment work well.

First, the text prompts — CLIP was trained on natural image captions, so a bare category word like 'jacket' underperforms a full sentence like 'a photo of a leather jacket on a white background' by a wide margin; prompt ensembling (averaging embeddings across several prompt templates per class) usually helps further. Second, preprocessing — image resizing and normalisation have to match exactly what the model was trained with, or embeddings come out meaningless. Third, whether the domain is even one CLIP saw much of during pretraining — general web images are well covered, but narrow domains like specific fashion catalogues or medical imagery often need fine-tuning on a small labelled set before zero-shot performance becomes reliable.

I would start by asking what the output actually needs to be. If the feature is 'find products that look like this photo' or 'search products by description,' that is retrieval — a CLIP embedding index against a vector database gives millisecond latency at a tiny fraction of the cost, and a generative model would add nothing but latency. If the feature needs to produce a written explanation — 'why did you flag this listing,' 'what is wrong with this product photo' — that requires actual generation, which means a VLM. In most real systems the answer is both: CLIP handles the high-volume narrowing step, and a VLM is reserved for the much smaller set of cases that need a generated explanation, which keeps the expensive model's cost bounded and justified.