GANs — Generator vs Discriminator

A counterfeiter (Generator) and a detective (Discriminator) playing an arms race. The counterfeiter starts producing terrible fakes — the detective catches them all. The counterfeiter studies which fake features gave them away and improves. The detective trains on the new fakes and gets better. After 100,000 rounds, the counterfeiter produces fakes so good even experts cannot tell them apart. Neither was told what perfect currency looks like — they learned from each other.

The catch: if the detective gets too good too fast, the counterfeiter receives no useful signal — all attempts score equally bad. If the counterfeiter gets too good too fast, the detective gives up and labels everything as fake. Balance is everything — and maintaining balance is why GAN training is notoriously hard.

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

torch.manual_seed(42)

# ── Generator — noise → image ─────────────────────────────────────────
class Generator(nn.Module):
    def __init__(self, latent_dim: int = 100, img_dim: int = 784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, 256),
            nn.LeakyReLU(0.2),
            nn.BatchNorm1d(256),
            nn.Linear(256, 512),
            nn.LeakyReLU(0.2),
            nn.BatchNorm1d(512),
            nn.Linear(512, img_dim),
            nn.Tanh(),   # output in [-1, 1] — match normalised real images
        )
    def forward(self, z): return self.net(z)

# ── Discriminator — image → real/fake probability ──────────────────────
class Discriminator(nn.Module):
    def __init__(self, img_dim: int = 784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(img_dim, 512),
            nn.LeakyReLU(0.2),   # LeakyReLU prevents dead neurons
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(256, 1),
            nn.Sigmoid(),         # output = P(real)
        )
    def forward(self, x): return self.net(x)

# ── Training loop ─────────────────────────────────────────────────────
LATENT_DIM  = 100
IMG_DIM     = 784   # 28×28 flattened
BATCH_SIZE  = 64
LR          = 2e-4

G   = Generator(LATENT_DIM, IMG_DIM)
D   = Discriminator(IMG_DIM)
opt_G = optim.Adam(G.parameters(), lr=LR, betas=(0.5, 0.999))
opt_D = optim.Adam(D.parameters(), lr=LR, betas=(0.5, 0.999))
# betas=(0.5, 0.999) is standard for GANs — lower momentum than default

criterion = nn.BCELoss()

# Labels — use label smoothing to stabilise training
real_label = 0.9   # soft label: not 1.0 but 0.9
fake_label = 0.0

def train_step(real_images: torch.Tensor):
    B = real_images.size(0)

    # ── Train Discriminator ───────────────────────────────────────────
    opt_D.zero_grad()

    # Real images → D should output ~1
    real_out    = D(real_images)
    labels_real = torch.full((B, 1), real_label)
    loss_D_real = criterion(real_out, labels_real)

    # Fake images → D should output ~0
    z         = torch.randn(B, LATENT_DIM)
    fake_imgs = G(z).detach()   # detach: don't backprop through G here
    fake_out  = D(fake_imgs)
    labels_fake = torch.zeros(B, 1)
    loss_D_fake = criterion(fake_out, labels_fake)

    loss_D = loss_D_real + loss_D_fake
    loss_D.backward()
    opt_D.step()

    # ── Train Generator ───────────────────────────────────────────────
    opt_G.zero_grad()

    z         = torch.randn(B, LATENT_DIM)
    fake_imgs = G(z)
    out       = D(fake_imgs)
    # Generator wants D to think fakes are real → label as 1
    labels_real_for_G = torch.ones(B, 1)
    loss_G    = criterion(out, labels_real_for_G)

    loss_G.backward()
    opt_G.step()

    return loss_D.item(), loss_G.item(), real_out.mean().item(), out.mean().item()

# ── Simulate training on random data ──────────────────────────────────
print(f"{'Epoch':>6} {'D loss':>10} {'G loss':>10} {'D(x)':>8} {'D(G(z))':>10}")
print("─" * 50)

for epoch in range(1, 11):
    # Simulate a batch of real images (normally from DataLoader)
    real_batch = torch.randn(BATCH_SIZE, IMG_DIM)   # fake "real" data
    loss_D, loss_G, dx, dgz = train_step(real_batch)

    if epoch % 2 == 0:
        print(f"  {epoch:>4}  {loss_D:>10.4f}  {loss_G:>10.4f}  "
              f"{dx:>8.4f}  {dgz:>10.4f}")

print(f"
Healthy training targets:")
print(f"  D(x)    ≈ 0.6-0.8  (discriminator not perfect on real images)")
print(f"  D(G(z)) ≈ 0.3-0.5  (generator fooling discriminator some of the time)")
print(f"  If D(x) → 1.0 and D(G(z)) → 0.0: discriminator winning too hard")
print(f"  If D(G(z)) → 1.0: generator winning too hard (or discriminator collapsed)")

import torch
import torch.nn as nn

def weights_init(m):
    """DCGAN weight initialisation — mean=0, std=0.02 for conv and bn."""
    classname = m.__class__.__name__
    if 'Conv' in classname:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif 'BatchNorm' in classname:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

# ── DCGAN Generator — noise → 64×64 image ────────────────────────────
class DCGANGenerator(nn.Module):
    """
    Input: (B, latent_dim, 1, 1)
    Output: (B, 3, 64, 64) — RGB image in [-1, 1]
    Each ConvTranspose2d doubles spatial resolution.
    """
    def __init__(self, latent_dim: int = 100, ngf: int = 64):
        super().__init__()
        self.net = nn.Sequential(
            # latent → 4×4
            nn.ConvTranspose2d(latent_dim, ngf*8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf*8), nn.ReLU(True),
            # 4×4 → 8×8
            nn.ConvTranspose2d(ngf*8, ngf*4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf*4), nn.ReLU(True),
            # 8×8 → 16×16
            nn.ConvTranspose2d(ngf*4, ngf*2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf*2), nn.ReLU(True),
            # 16×16 → 32×32
            nn.ConvTranspose2d(ngf*2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf), nn.ReLU(True),
            # 32×32 → 64×64
            nn.ConvTranspose2d(ngf, 3, 4, 2, 1, bias=False),
            nn.Tanh(),   # output in [-1, 1]
        )
    def forward(self, z): return self.net(z)

# ── DCGAN Discriminator — 64×64 image → P(real) ───────────────────────
class DCGANDiscriminator(nn.Module):
    """
    Input: (B, 3, 64, 64)
    Output: (B, 1) — probability of being real
    Each Conv2d halves spatial resolution (stride=2).
    """
    def __init__(self, ndf: int = 64):
        super().__init__()
        self.net = nn.Sequential(
            # 64×64 → 32×32 (no BN on first layer)
            nn.Conv2d(3, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # 32×32 → 16×16
            nn.Conv2d(ndf, ndf*2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf*2), nn.LeakyReLU(0.2, inplace=True),
            # 16×16 → 8×8
            nn.Conv2d(ndf*2, ndf*4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf*4), nn.LeakyReLU(0.2, inplace=True),
            # 8×8 → 4×4
            nn.Conv2d(ndf*4, ndf*8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf*8), nn.LeakyReLU(0.2, inplace=True),
            # 4×4 → 1×1
            nn.Conv2d(ndf*8, 1, 4, 1, 0, bias=False),
            nn.Sigmoid(),
        )
    def forward(self, x): return self.net(x).view(-1, 1)

# ── Shape verification ────────────────────────────────────────────────
G = DCGANGenerator(latent_dim=100, ngf=64)
D = DCGANDiscriminator(ndf=64)
G.apply(weights_init)
D.apply(weights_init)

z    = torch.randn(4, 100, 1, 1)
fake = G(z)
disc = D(fake)

g_params = sum(p.numel() for p in G.parameters())
d_params = sum(p.numel() for p in D.parameters())

print(f"DCGAN architecture:")
print(f"  Generator:     {g_params:,} parameters")
print(f"  Discriminator: {d_params:,} parameters")
print(f"  Noise input:   {tuple(z.shape)}")
print(f"  Fake image:    {tuple(fake.shape)}  ← (B, 3, 64, 64) in [-1,1]")
print(f"  D output:      {tuple(disc.shape)}  ← P(real) per image")
print(f"
Key: generator and discriminator have similar parameter counts")
print(f"  Too large D relative to G → D wins too easily → vanishing gradients for G")

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

# ── Wasserstein GAN with Gradient Penalty (WGAN-GP) ──────────────────
# More stable than vanilla GAN — the standard for production GANs

class WGANCritic(nn.Module):
    """
    Critic (not Discriminator) — no Sigmoid, outputs real number not probability.
    Higher output = more real. Lower = more fake.
    """
    def __init__(self, img_dim: int = 784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(img_dim, 512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 1),
            # No Sigmoid — critic outputs unbounded real number
        )
    def forward(self, x): return self.net(x)

class WGANGenerator(nn.Module):
    def __init__(self, latent_dim: int = 100, img_dim: int = 784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, 256), nn.ReLU(),
            nn.Linear(256, 512), nn.ReLU(),
            nn.Linear(512, img_dim), nn.Tanh(),
        )
    def forward(self, z): return self.net(z)

def gradient_penalty(critic, real, fake, device='cpu'):
    """
    WGAN-GP gradient penalty — enforces Lipschitz constraint
    by penalising gradients with norm != 1 at interpolated points.
    This replaces weight clipping (more stable).
    """
    B = real.size(0)
    alpha = torch.rand(B, 1, device=device)

    # Interpolate between real and fake
    interpolated = (alpha * real + (1 - alpha) * fake).requires_grad_(True)
    critic_interp = critic(interpolated)

    # Compute gradients of critic output w.r.t. interpolated inputs
    gradients = torch.autograd.grad(
        outputs=critic_interp,
        inputs=interpolated,
        grad_outputs=torch.ones_like(critic_interp),
        create_graph=True,
        retain_graph=True,
    )[0]

    grad_norm = gradients.norm(2, dim=1)
    penalty   = ((grad_norm - 1) ** 2).mean()
    return penalty

# ── WGAN-GP training step ─────────────────────────────────────────────
LATENT_DIM  = 100
IMG_DIM     = 784
BATCH_SIZE  = 64
LR          = 1e-4
LAMBDA_GP   = 10     # gradient penalty weight
N_CRITIC    = 5      # train critic 5x per generator update

G  = WGANGenerator(LATENT_DIM, IMG_DIM)
C  = WGANCritic(IMG_DIM)
opt_G = optim.Adam(G.parameters(), lr=LR, betas=(0.0, 0.9))
opt_C = optim.Adam(C.parameters(), lr=LR, betas=(0.0, 0.9))
# betas=(0.0, 0.9) — no momentum for WGAN — standard recommendation

def wgan_gp_step(real_images):
    B = real_images.size(0)

    # ── Train Critic N_CRITIC times per generator step ────────────────
    for _ in range(N_CRITIC):
        opt_C.zero_grad()
        z    = torch.randn(B, LATENT_DIM)
        fake = G(z).detach()

        # Wasserstein loss: maximise C(real) - C(fake)
        # Critic loss = -[C(real) - C(fake)] = C(fake) - C(real)
        loss_C = C(fake).mean() - C(real_images).mean()
        gp     = gradient_penalty(C, real_images, fake)
        loss_C_total = loss_C + LAMBDA_GP * gp
        loss_C_total.backward()
        opt_C.step()

    # ── Train Generator once ──────────────────────────────────────────
    opt_G.zero_grad()
    z    = torch.randn(B, LATENT_DIM)
    fake = G(z)
    # Generator wants to maximise C(fake) → minimise -C(fake)
    loss_G = -C(fake).mean()
    loss_G.backward()
    opt_G.step()

    return loss_C.item(), loss_G.item(), gp.item()

# Training
print("WGAN-GP training:")
print(f"{'Step':>6} {'W-distance':>12} {'G loss':>10} {'GP':>8}")
print("─" * 40)

for step in range(1, 6):
    real = torch.randn(BATCH_SIZE, IMG_DIM)
    lc, lg, gp = wgan_gp_step(real)
    # Wasserstein distance ≈ -critic_loss (before GP)
    print(f"  {step:>4}  {-lc:>12.4f}  {lg:>10.4f}  {gp:>8.4f}")

print(f"
WGAN vs vanilla GAN:")
print(f"  No Sigmoid on critic → unbounded scores → meaningful gradients always")
print(f"  Wasserstein distance → meaningful even when distributions don't overlap")
print(f"  Gradient penalty → Lipschitz constraint without weight clipping artifacts")
print(f"  Train critic 5× per G step → critic stays ahead for accurate feedback")

import torch
import torch.nn as nn
import torch.optim as optim

# ── Conditional GAN — generate specific fashion categories ────────────
N_CLASSES  = 6   # kurta, saree, jeans, sneakers, watch, handbag
LATENT_DIM = 100
IMG_DIM    = 784
EMBED_DIM  = 50

class ConditionalGenerator(nn.Module):
    """
    Takes noise z + class label c → generates image of class c.
    Class label is embedded and concatenated with noise.
    """
    def __init__(self):
        super().__init__()
        # Class embedding: integer class → dense vector
        self.label_emb = nn.Embedding(N_CLASSES, EMBED_DIM)

        self.net = nn.Sequential(
            nn.Linear(LATENT_DIM + EMBED_DIM, 256),
            nn.LeakyReLU(0.2),
            nn.BatchNorm1d(256),
            nn.Linear(256, 512),
            nn.LeakyReLU(0.2),
            nn.BatchNorm1d(512),
            nn.Linear(512, IMG_DIM),
            nn.Tanh(),
        )

    def forward(self, z, labels):
        # Concatenate noise with class embedding
        c    = self.label_emb(labels)     # (B, EMBED_DIM)
        inp  = torch.cat([z, c], dim=1)   # (B, LATENT_DIM + EMBED_DIM)
        return self.net(inp)

class ConditionalDiscriminator(nn.Module):
    """
    Takes image + class label → judges if image matches label AND looks real.
    """
    def __init__(self):
        super().__init__()
        self.label_emb = nn.Embedding(N_CLASSES, IMG_DIM)

        self.net = nn.Sequential(
            nn.Linear(IMG_DIM * 2, 512),  # image + label image concatenated
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(256, 1),
            nn.Sigmoid(),
        )

    def forward(self, img, labels):
        # Embed label as an image-sized vector and concatenate
        c   = self.label_emb(labels)           # (B, IMG_DIM)
        inp = torch.cat([img, c], dim=1)        # (B, IMG_DIM * 2)
        return self.net(inp)

# ── Training step for cGAN ────────────────────────────────────────────
G = ConditionalGenerator()
D = ConditionalDiscriminator()
opt_G = optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_D = optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
criterion = nn.BCELoss()

def cgan_step(real_images, real_labels):
    B = real_images.size(0)

    # ── Discriminator ────────────────────────────────────────────────
    opt_D.zero_grad()

    # Real images with correct labels
    real_out  = D(real_images, real_labels)
    loss_real = criterion(real_out, torch.full((B, 1), 0.9))

    # Fake images with same labels — G must match the label
    z         = torch.randn(B, LATENT_DIM)
    fake      = G(z, real_labels).detach()
    fake_out  = D(fake, real_labels)
    loss_fake = criterion(fake_out, torch.zeros(B, 1))

    (loss_real + loss_fake).backward()
    opt_D.step()

    # ── Generator ────────────────────────────────────────────────────
    opt_G.zero_grad()
    z      = torch.randn(B, LATENT_DIM)
    labels = torch.randint(0, N_CLASSES, (B,))
    fake   = G(z, labels)
    out    = D(fake, labels)
    loss_G = criterion(out, torch.ones(B, 1))
    loss_G.backward()
    opt_G.step()

    return loss_G.item()

# ── Controlled generation after training ──────────────────────────────
CATEGORIES = ['kurta', 'saree', 'jeans', 'sneakers', 'watch', 'handbag']

print("cGAN: generating specific fashion categories:")
with torch.no_grad():
    for cat_idx, cat_name in enumerate(CATEGORIES):
        z      = torch.randn(4, LATENT_DIM)
        labels = torch.full((4,), cat_idx, dtype=torch.long)
        images = G(z, labels)
        print(f"  {cat_name:<12}: generated {images.shape[0]} images  "
              f"mean={images.mean():.4f}  std={images.std():.4f}")

The GAN-vs-diffusion split is already blurring. Consistency models and adversarial diffusion distillation (the technique behind SDXL Turbo) train a fast, few-step or single-step generator to match a slow diffusion model's output — borrowing GAN-style single-pass speed while keeping diffusion-level training stability and quality. If you are asked in an interview whether GANs are "dead," the accurate answer is that their core insight — a fast single-pass generator — got absorbed into the diffusion toolchain rather than discarded.

Mode collapse is a diversity failure, not a convergence failure — the Generator's weights are still updating, and it can even be fooling the Discriminator quite successfully. What has happened is that it found a narrow region of the output space that reliably scores well and stopped exploring the rest of the data distribution. The loss curves can look completely healthy while every generated sample looks nearly identical, which is exactly why loss values alone cannot diagnose GAN health — you have to directly inspect a batch of generated samples for diversity.

Much of GAN's reputation for instability traces back to the original 2014 formulation with naive hyperparameters, not to anything unavoidable about adversarial training itself. DCGAN's architectural rules, WGAN-GP's Wasserstein objective, spectral normalisation, and label smoothing collectively make training dramatically more reliable — this module exists specifically because the field spent years engineering these stabilisation techniques and they work. Instability was a largely solvable problem, not a permanent property of the minimax game.

As the production landscape above shows, GANs remain the practical choice anywhere single-forward-pass speed matters — real-time filters, upscaling, narrow-domain face generation. It is also worth knowing that modern fast diffusion variants (consistency models, adversarial diffusion distillation) borrow GAN-style objectives to get GAN-like speed out of a diffusion-trained model — the two approaches have converged rather than one cleanly replacing the other.

It is close to the opposite. A Discriminator that classifies everything correctly gives the Generator essentially zero useful gradient — the vanishing-gradient failure mode covered earlier in this module. Healthy GAN training deliberately keeps the Discriminator somewhat "confused," typically D(x) around 0.6–0.8 rather than 1.0. This runs against the instinct that a more accurate classifier is always a better sign, which is exactly why it trips people up the first time they read GAN training logs.

The objective function is adversarial — one network's gain is written as the other's loss — but in practice healthy training behaves more like a cooperative arms race than a zero-sum fight. Each network needs the other to keep improving in order to keep receiving useful learning signal itself: if either one "wins" outright and stops providing a meaningful gradient to the other, both stop improving. Balance, not victory, is the actual target — the forger-and-detective analogy earlier in this module is making exactly this point.

The Discriminator tries to maximise its ability to tell real data from the Generator's fakes; the Generator tries to minimise that same quantity — it wants the Discriminator's job to be as hard as possible. Training alternates between a step that moves the Discriminator toward its maximum and a step that moves the Generator toward its minimum of the same expression, which is exactly the "min over G, max over D" formulation the name describes. At the theoretical equilibrium, the Generator's distribution matches the real data distribution closely enough that the Discriminator cannot do better than random guessing.

Two networks are each other's moving target — unlike standard supervised learning where the loss surface is fixed, here the Generator's loss landscape shifts every time the Discriminator updates and vice versa, so there is no static optimum either one is climbing toward. On top of that, if the Discriminator gets too strong, the Generator's gradient vanishes entirely. Two concrete fixes: switch to a Wasserstein objective with gradient penalty (WGAN-GP), which provides meaningful gradients even when the real and fake distributions barely overlap, and follow the DCGAN architectural guidelines (strided convolutions, batch norm placement, matched learning rates and momentum) that were specifically found to reduce oscillation.

The original GAN objective is based on Jensen-Shannon divergence, which saturates and provides essentially no usable gradient when the real and generated distributions do not meaningfully overlap — a common situation early in training, exactly when the Generator most needs signal. The Wasserstein (Earth Mover's) distance stays smooth and informative even when two distributions are disjoint, because it measures the cost of literally moving probability mass from one distribution to the other rather than a divergence that collapses to a constant. WGAN implements this by removing the Discriminator's Sigmoid (making it a Critic outputting an unbounded score) and enforcing a Lipschitz constraint, originally via weight clipping and later, more stably, via a gradient penalty on interpolated points.

Mode collapse is when the Generator converges to producing a narrow set of outputs — sometimes nearly identical images — regardless of the input noise, because it found a small region that reliably fools the current Discriminator. In practice, loss curves alone will not reveal this, since both networks can look like they are training normally. What actually catches it: sample a large batch from the Generator and check output diversity directly, or compute FID against a diverse validation set — a Generator producing low-variance, repetitive samples will show a poor FID despite individual samples looking sharp, because FID specifically penalises a collapsed distribution shape, not just per-image quality.

Yes, and the reasoning is entirely about latency. A GAN generates an image in a single forward pass — on the order of 10ms on a mobile GPU — while a standard diffusion model needs tens to hundreds of denoising steps to produce comparable quality, which is far too slow for a live video frame rate. If diffusion-level quality is genuinely required, distilled or consistency-model variants of diffusion can get closer to single-step speed, but for a hard real-time constraint like a live filter, a GAN remains the reasonable default rather than a legacy choice.