What is Generative AI?

A discriminative model is a critic — shown a painting, they say "Monet" or "Picasso." They have learned boundaries between styles but cannot paint. A generative model is an artist — they have studied thousands of Monet paintings so deeply that they can create a new painting that looks authentically Monet, even though that exact painting never existed.

The critic learns P(style | painting). The artist learns P(painting) in Monet's style — the full distribution of what Monet paintings look like — and samples from it. That is the fundamental difference.

import torch
import torch.nn as nn
import numpy as np

# ── Minimal VAE to illustrate latent space ────────────────────────────
class SimpleVAE(nn.Module):
    def __init__(self, input_dim: int = 784, latent_dim: int = 32):
        super().__init__()
        # Encoder: input → mean and log_variance of latent distribution
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256), nn.ReLU(),
            nn.Linear(256, 128), nn.ReLU(),
        )
        self.fc_mean    = nn.Linear(128, latent_dim)
        self.fc_log_var = nn.Linear(128, latent_dim)

        # Decoder: latent vector → reconstructed input
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128), nn.ReLU(),
            nn.Linear(128, 256), nn.ReLU(),
            nn.Linear(256, input_dim), nn.Sigmoid(),
        )

    def encode(self, x):
        h       = self.encoder(x)
        mean    = self.fc_mean(h)
        log_var = self.fc_log_var(h)
        return mean, log_var

    def reparameterise(self, mean, log_var):
        """
        The reparameterisation trick: sample z = mean + std × epsilon
        epsilon ~ N(0, I) — random noise
        This makes sampling differentiable — gradients can flow through.
        Without this trick, sampling breaks backpropagation.
        """
        std     = torch.exp(0.5 * log_var)
        epsilon = torch.randn_like(std)
        return mean + std * epsilon

    def decode(self, z):
        return self.decoder(z)

    def forward(self, x):
        mean, log_var = self.encode(x)
        z             = self.reparameterise(mean, log_var)
        x_recon       = self.decode(z)
        return x_recon, mean, log_var

def vae_loss(x_recon, x, mean, log_var):
    """
    ELBO loss = Reconstruction loss + KL divergence
    Reconstruction: how well did we reconstruct the input?
    KL divergence:  how close is the latent distribution to N(0, I)?
                    This regularises the latent space to be smooth.
    """
    recon_loss = nn.functional.binary_cross_entropy(x_recon, x, reduction='sum')
    # KL divergence between N(mean, var) and N(0, 1):
    # −0.5 × Σ(1 + log_var − mean² − exp(log_var))
    kl_loss    = -0.5 * torch.sum(1 + log_var - mean.pow(2) - log_var.exp())
    return recon_loss + kl_loss

# ── Shape demonstration ───────────────────────────────────────────────
torch.manual_seed(42)
vae   = SimpleVAE(input_dim=784, latent_dim=32)
x     = torch.randn(8, 784)   # batch of 8 flattened 28×28 images

x_recon, mean, log_var = vae(x)
loss = vae_loss(x_recon, torch.sigmoid(x), mean, log_var)

print(f"VAE shapes:")
print(f"  Input:        {tuple(x.shape)}")
print(f"  Latent mean:  {tuple(mean.shape)}   ← 32-dim latent space")
print(f"  Latent logvar:{tuple(log_var.shape)}")
print(f"  Reconstructed:{tuple(x_recon.shape)}")
print(f"  ELBO loss:    {loss.item():.2f}")

# ── Latent space operations ───────────────────────────────────────────
with torch.no_grad():
    # Encode two images
    mean1, _ = vae.encode(x[0:1])
    mean2, _ = vae.encode(x[1:2])

    # Interpolate between them
    print(f"
Latent space interpolation:")
    for t in [0.0, 0.25, 0.5, 0.75, 1.0]:
        z_interp = (1 - t) * mean1 + t * mean2
        decoded  = vae.decode(z_interp)
        print(f"  t={t:.2f}: z_norm={z_interp.norm():.3f}  decoded_mean={decoded.mean():.4f}")

    # Sample new images from prior
    print(f"
Sampling from prior N(0, I):")
    for i in range(3):
        z_random = torch.randn(1, 32)   # sample from standard normal
        new_img  = vae.decode(z_random)
        print(f"  Sample {i+1}: mean={new_img.mean():.4f}  std={new_img.std():.4f}")

import torch
import numpy as np
from scipy import linalg

# ── FID from scratch — the core calculation ───────────────────────────
def compute_fid(real_features: np.ndarray,
                fake_features: np.ndarray) -> float:
    """
    Compute FID between real and generated image features.
    Features are typically extracted from InceptionV3's penultimate layer.
    real_features: (N, 2048) — features from real images
    fake_features: (N, 2048) — features from generated images
    """
    # Compute mean and covariance of each distribution
    mu_real   = real_features.mean(axis=0)
    mu_fake   = fake_features.mean(axis=0)
    sigma_real = np.cov(real_features, rowvar=False)
    sigma_fake = np.cov(fake_features, rowvar=False)

    # Fréchet distance between two multivariate Gaussians:
    # ||mu_r - mu_f||² + Tr(sigma_r + sigma_f - 2√(sigma_r × sigma_f))
    diff       = mu_real - mu_fake
    mean_term  = diff @ diff   # ||mu_r - mu_f||²

    # Matrix square root of sigma_r × sigma_f
    covmean, _ = linalg.sqrtm(sigma_real @ sigma_fake, disp=False)
    if np.iscomplexobj(covmean):
        covmean = covmean.real   # numerical correction

    trace_term = np.trace(sigma_real + sigma_fake - 2 * covmean)
    fid        = mean_term + trace_term
    return float(fid)

# ── Simulate feature distributions ────────────────────────────────────
np.random.seed(42)
N_SAMPLES    = 1000
FEATURE_DIM  = 2048

# Perfect model: generated features = real features (FID ≈ 0)
real_features    = np.random.randn(N_SAMPLES, FEATURE_DIM)
perfect_features = real_features + np.random.randn(N_SAMPLES, FEATURE_DIM) * 0.01

# Good model: small distribution shift
good_features    = real_features + np.random.randn(N_SAMPLES, FEATURE_DIM) * 0.5

# Poor model: large distribution shift
poor_features    = np.random.randn(N_SAMPLES, FEATURE_DIM) * 2 + 3

print("FID scores (lower = better, 0 = perfect):")
print(f"  Perfect model: {compute_fid(real_features, perfect_features):.2f}")
print(f"  Good model:    {compute_fid(real_features, good_features):.2f}")
print(f"  Poor model:    {compute_fid(real_features, poor_features):.2f}")

print("
FID benchmarks in literature:")
benchmarks = [
    ('StyleGAN2 (FFHQ 256)',     2.8,  'State of the art face generation'),
    ('Stable Diffusion 2.1',     8.6,  'Text-to-image, COCO benchmark'),
    ('DALL-E 2',                 10.4, 'Text-to-image'),
    ('GAN with mode collapse',   80.0, 'Poor diversity'),
    ('Random noise baseline',   300.0, 'No learning'),
]
for name, fid, note in benchmarks:
    bar = '█' * int(min(fid, 100) / 4)
    print(f"  {name:<35}: FID={fid:>6.1f}  {bar}  {note}")

# ── Quick API tour — generate content with each model family ─────────

# ── 1. LLM generation — text ──────────────────────────────────────────
from groq import Groq
import os

client = Groq(api_key=os.environ.get('GROQ_API_KEY', 'demo'))

# Autoregressive generation — sample one token at a time
# response = client.chat.completions.create(
#     model='openai/gpt-oss-120b',
#     messages=[{'role': 'user', 'content': 'Write a product description for a Shopify kurta'}],
#     temperature=0.8,   # higher = more creative/diverse
#     max_tokens=200,
# )
# text = response.choices[0].message.content
print("LLM generation: autoregressive sampling, token by token")
print("  temperature=0.0 → deterministic (always picks highest probability token)")
print("  temperature=1.0 → samples from full distribution")
print("  temperature>1.0 → more random, creative, possibly incoherent")

# ── 2. Diffusion generation — image from text ─────────────────────────
print("
Diffusion generation (Stable Diffusion):")
print("  pip install diffusers torch")
print("""
from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    'runwayml/stable-diffusion-v1-5',
    torch_dtype=torch.float16,
).to('cuda')

image = pipe(
    prompt='A beautiful silk saree in red and gold, product photography',
    negative_prompt='blurry, low quality, distorted',
    num_inference_steps=30,    # denoising steps — more = better quality, slower
    guidance_scale=7.5,        # how strongly to follow the text prompt
    height=512, width=512,
).images[0]
image.save('saree.png')
""")

# ── 3. VAE — encode and decode ────────────────────────────────────────
print("VAE generation:")
print("  Encode image → 128-dim latent vector")
print("  Perturb latent vector → decode → variation of original image")
print("  Interpolate between two latent vectors → smooth transition")

# ── 4. GAN — single forward pass ─────────────────────────────────────
print("
GAN generation (StyleGAN2):")
print("""
import torch
# Load pretrained generator
# G = StyleGAN2Generator(...)
# z = torch.randn(1, 512)    # random latent
# img = G(z)                  # single forward pass → 1024×1024 face
# Total: one matrix multiplication chain, ~10ms on GPU
""")

Underneath, it is still the same machinery covered throughout this track — gradient descent, backpropagation, loss functions, the dot products from Module 05. What actually changes is the training objective (learning P(data) instead of P(label given data)) and the scale (billions of parameters instead of thousands). A transformer-based LLM and a logistic regression model are trained with the same core optimisation loop; generative models did not throw out the fundamentals, they scaled them up and pointed them at a different target.

The mechanical claim is true — the training objective really is next-token prediction, nothing more exotic than that. But 'just' undersells what emerges from doing that at enormous scale: multi-step reasoning, following novel instructions never seen in training, writing working code — none of these were explicitly supervised, they emerged from scaling the same simple objective. At the same time, 'it's just autocomplete' gets used to wave away real capability limits, like hallucination, as if naming the mechanism settles what the system can reliably be trusted to do. Both the dismissal and the hype skip past the same nuance: the training objective is simple, the resulting behaviour is not fully predictable from that simplicity alone.

Hallucination is a structural consequence of how these models generate text, not a stray bug in an otherwise correct implementation. An LLM samples from a learned probability distribution over what token is likely to come next — it is producing what is statistically plausible, not looking up what is verified true, because it has no built-in fact store to check against. Even a model trained flawlessly on its objective will occasionally produce a fluent, confident, wrong statement when nothing grounds the answer in a retrieved or verified source. Retrieval-augmented generation, tool use, and citations reduce the problem by giving the model something to ground against — they do not eliminate it, because the underlying generation mechanism never changes.

Both come from exactly the same sampling process — drawing the next token from a learned probability distribution, with temperature controlling how much the model deviates from its single most likely choice. What we call 'creative' is ungrounded generation in a context where that is desired (a poem, a brainstormed idea); what we call 'hallucination' is the identical mechanism in a context where groundedness was expected (a factual claim, a citation). A model that could never generate anything ungrounded would also be incapable of genuine creative generation — the tension is inherent to sampling, not a separate failure mode that can be independently switched off.

Task fit dominates raw scale far more often than marketing suggests. A GAN still wins for latency-sensitive single-pass image generation that a diffusion model's multi-step denoising cannot match. A VAE's smooth latent space is still the right tool for anomaly detection and molecule generation, tasks where a GAN or diffusion model's strengths are irrelevant. A small model fine-tuned on a narrow task routinely beats a much larger general-purpose one on that task, at a fraction of the latency and cost. 'Generative AI' names a family of architectures, not one model — picking the wrong family for your constraints is a common, avoidable mistake.

A discriminative model learns P(label given data) directly — it models the decision boundary and nothing about how the data itself was produced. A generative model learns P(data), or the joint P(data, label), the full distribution the data came from, which lets you sample new data from it. Worth adding unprompted: some classic generative models, like Naive Bayes, are still used for classification — they learn P(data given label) for each class and use Bayes' rule to get a decision at inference time. That shows 'generative' describes what the model represents, not necessarily how it gets used at prediction time.

It means the model defines, or approximates, a probability distribution over the data (or over data conditioned on some input), and 'generating' something means drawing a sample from that distribution. The output is not retrieved from storage or copied from training data — it is a fresh draw from a learned distribution, which is why the same prompt or noise vector can produce different outputs on different runs, and why evaluating these models requires distributional metrics like FID instead of accuracy against one correct answer.

GANs pit a generator against a discriminator in an adversarial game and produce sharp images in a single forward pass, but training is unstable and prone to mode collapse. VAEs encode data to a regularised probability distribution and decode back, optimising the ELBO — reconstruction plus a KL term — giving stable training and a smooth latent space at the cost of blurrier outputs. Diffusion models learn to reverse a gradual noising process, generating through many small denoising steps, trading slow inference for the best image quality and diversity available today. LLMs autoregressively predict the next token given everything before it, and generation is repeated sampling from that predicted distribution one token at a time.

They hallucinate because generation is sampling from a learned distribution over plausible next tokens, not a lookup against verified facts — there is no ground truth store the model consults before answering. Grounding techniques (retrieval-augmented generation, tool use, requiring citations, RLHF calibration toward saying 'I don't know') substantially reduce the rate in practice. But being honest about the mechanism: none of these fully fix it, because the underlying generation process — sampling the statistically likely continuation — never changes. The realistic framing in an interview is 'substantially mitigated, not eliminated.'

Start from modality and constraints, not from whatever is trending. Text generation almost always means an LLM. Open-ended image generation from a text prompt means diffusion. A latency-critical, single-pass image task (a live filter, a real-time avatar) points toward a GAN because diffusion's iterative denoising is too slow for that budget. Anomaly detection or a task that needs a smooth, interpolatable latent space points toward a VAE. Layered on top of all of that is the build-vs-buy question from earlier in this module: whether prompting an existing API, fine-tuning an open model, or — almost never — pretraining from scratch is the right level of investment for the constraint you are actually solving for.