BERT and the Encoder-Only Family

Reading comprehension in school: you read the full passage, then answer questions about it. You read forwards and backwards, checking context in both directions. A student who only reads left to right and never re-reads misses nuance. BERT is the student who reads the full passage before answering. GPT is the student writing an essay — they cannot read what they have not written yet.

This is why BERT dominates understanding tasks (classification, NER, Q&A) while GPT dominates generation tasks (completion, summarisation, chat). Same Transformer architecture, different direction of attention, completely different use cases.

Note: Later research (RoBERTa, 2019) showed NSP does not help and may hurt. RoBERTa removed it entirely and achieved better results training with MLM only on more data for longer.

# ── Demonstrate MLM manually — what BERT is trained to do ───────────
# In production BERT is pretrained for you — this shows the concept

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

# Load BERT tokeniser and MLM model
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model     = AutoModelForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()

# ── Task: fill in the [MASK] ──────────────────────────────────────────
sentences = [
    "Stripe [MASK] my payment successfully.",
    "The [MASK] was declined due to insufficient funds.",
    "Please [MASK] your transaction after some time.",
]

print("BERT Masked Language Modelling predictions:")
for sentence in sentences:
    inputs = tokenizer(sentence, return_tensors='pt')
    mask_idx = (inputs['input_ids'] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]

    with torch.no_grad():
        logits = model(**inputs).logits

    # Top 5 predictions for the masked token
    mask_logits = logits[0, mask_idx, :]
    top5 = torch.topk(mask_logits, 5, dim=-1)
    top5_tokens = tokenizer.convert_ids_to_tokens(top5.indices[0])
    top5_probs  = torch.softmax(top5.values[0], dim=0)

    print(f"
  '{sentence}'")
    for tok, prob in zip(top5_tokens, top5_probs):
        bar = '█' * int(prob.item() * 30)
        print(f"    {tok:<15} {bar:<30} {prob.item():.3f}")

from transformers import AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# ── BERT tokenisation — what actually goes into the model ─────────────
text_a = "Stripe payment was declined"
text_b = "Please retry after some time"

# Single sentence
single = tokenizer(text_a, return_tensors='pt')
print("Single sentence encoding:")
print(f"  input_ids:      {single['input_ids'][0].tolist()}")
print(f"  token_type_ids: {single['token_type_ids'][0].tolist()}  ← all 0 (sentence A)")
print(f"  attention_mask: {single['attention_mask'][0].tolist()}")
print(f"  tokens:         {tokenizer.convert_ids_to_tokens(single['input_ids'][0])}")

# Two sentences (for NSP, Q&A, sentence pair tasks)
pair = tokenizer(text_a, text_b, return_tensors='pt')
print(f"
Sentence pair encoding:")
print(f"  tokens:         {tokenizer.convert_ids_to_tokens(pair['input_ids'][0])}")
print(f"  token_type_ids: {pair['token_type_ids'][0].tolist()}")
print(f"  ← 0s = sentence A, 1s = sentence B")

# ── Batch encoding with padding ───────────────────────────────────────
batch_texts = [
    "payment declined",
    "transaction successful confirmation received",
    "upi failed",
]
batch = tokenizer(batch_texts, padding=True, truncation=True,
                  max_length=32, return_tensors='pt')
print(f"
Batch encoding (padded to same length):")
print(f"  input_ids shape:      {batch['input_ids'].shape}")
print(f"  attention_mask shape: {batch['attention_mask'].shape}")
print(f"  Padding token ID:     {tokenizer.pad_token_id}")
print(f"  [CLS] token ID:       {tokenizer.cls_token_id}")
print(f"  [SEP] token ID:       {tokenizer.sep_token_id}")
print(f"  [MASK] token ID:      {tokenizer.mask_token_id}")
print(f"
Vocab size: {tokenizer.vocab_size:,}  (WordPiece)")

from transformers import (
    AutoTokenizer, AutoModelForSequenceClassification,
    AutoModelForTokenClassification,
    TrainingArguments, Trainer, DataCollatorWithPadding,
)
from datasets import Dataset
import evaluate
import numpy as np
import torch

# ── Task: DoorDash complaint classification ────────────────────────────
# Categories: delivery_late, wrong_order, missing_items, quality_issue

LABELS   = ['delivery_late', 'wrong_order', 'missing_items', 'quality_issue']
label2id = {l: i for i, l in enumerate(LABELS)}
id2label = {i: l for i, l in enumerate(LABELS)}

complaints = [
    ("Order took 2 hours to arrive, completely cold", 0),
    ("Received biryani instead of ordered pizza", 1),
    ("Only 2 of 4 items delivered, rest missing", 2),
    ("Food was stale and tasted terrible", 3),
    ("Delivery was delayed by 45 minutes", 0),
    ("Wrong restaurant items delivered entirely", 1),
    ("Half my order was not delivered", 2),
    ("Found hair in the food very unhygienic", 3),
] * 50

tokenizer  = AutoTokenizer.from_pretrained('distilbert-base-uncased')
model      = AutoModelForSequenceClassification.from_pretrained(
    'distilbert-base-uncased',
    num_labels=4,
    id2label=id2label,
    label2id=label2id,
)

def tokenise(examples):
    return tokenizer(examples['text'], max_length=128,
                     truncation=True, padding=False)

texts  = [c[0] for c in complaints]
labels = [c[1] for c in complaints]

raw   = Dataset.from_dict({'text': texts, 'label': labels})
raw   = raw.train_test_split(test_size=0.2, seed=42)
tok   = raw.map(tokenise, batched=True, remove_columns=['text'])

collator = DataCollatorWithPadding(tokenizer=tokenizer)
acc_metric = evaluate.load('accuracy')

def compute_metrics(eval_pred):
    preds  = np.argmax(eval_pred.predictions, axis=1)
    return acc_metric.compute(predictions=preds, references=eval_pred.label_ids)

args = TrainingArguments(
    output_dir='./doordash-complaints',
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    weight_decay=0.01,
    evaluation_strategy='epoch',
    save_strategy='epoch',
    load_best_model_at_end=True,
    metric_for_best_model='accuracy',
    seed=42,
    logging_steps=20,
)

trainer = Trainer(
    model=model, args=args,
    train_dataset=tok['train'],
    eval_dataset=tok['test'],
    tokenizer=tokenizer,
    data_collator=collator,
    compute_metrics=compute_metrics,
)

print("Fine-tuning DistilBERT on DoorDash complaints:")
trainer.train()
results = trainer.evaluate()
print(f"
Accuracy: {results['eval_accuracy']:.4f}")

# ── Inference on new complaints ───────────────────────────────────────
model.eval()
new_complaints = [
    "Waited 90 minutes for delivery",
    "Got someone else's order completely",
    "Three items from my cart were not in the bag",
]
inputs = tokenizer(new_complaints, padding=True, truncation=True,
                   max_length=128, return_tensors='pt')
with torch.no_grad():
    logits = model(**inputs).logits
    probs  = torch.softmax(logits, dim=1)

print("
Predictions on new complaints:")
for complaint, prob in zip(new_complaints, probs):
    pred = LABELS[prob.argmax()]
    conf = prob.max().item()
    print(f"  '{complaint[:40]}' → {pred} ({conf:.3f})")

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch, time

# ── Benchmark: accuracy vs speed tradeoff across BERT family ──────────
models_to_compare = [
    ('distilbert-base-uncased', 'DistilBERT'),
    ('bert-base-uncased',       'BERT-base'),
    ('roberta-base',            'RoBERTa-base'),
]

texts = ["Stripe payment declined please help retry"] * 64

print(f"{'Model':<20} {'Params':>8} {'Latency (ms)':>14} {'Throughput'}")
print("─" * 60)

for model_id, name in models_to_compare:
    tok   = AutoTokenizer.from_pretrained(model_id)
    model = AutoModelForSequenceClassification.from_pretrained(
        model_id, num_labels=3
    )
    model.eval()

    params = sum(p.numel() for p in model.parameters()) / 1e6

    inputs = tok(texts, return_tensors='pt', padding=True,
                 truncation=True, max_length=64)

    # Warmup
    with torch.no_grad(): _ = model(**inputs)

    # Time
    start = time.time()
    for _ in range(5):
        with torch.no_grad(): _ = model(**inputs)
    ms_per_batch = (time.time() - start) / 5 * 1000

    throughput = len(texts) / (ms_per_batch / 1000)
    print(f"  {name:<18}  {params:>6.0f}M  {ms_per_batch:>12.1f}ms  "
          f"{throughput:>6.0f} samples/s")

from transformers import (
    AutoTokenizer, AutoModelForTokenClassification, pipeline,
)
import torch

# ── NER with pretrained BERT ──────────────────────────────────────────
# Use a pretrained NER model — fine-tuned BERT on CoNLL-2003
tokenizer = AutoTokenizer.from_pretrained('dbmdz/bert-large-cased-finetuned-conll03-english')
model     = AutoModelForTokenClassification.from_pretrained(
    'dbmdz/bert-large-cased-finetuned-conll03-english'
)

# Pipeline handles aggregation (merging subword tokens into words)
ner_pipeline = pipeline(
    'ner',
    model=model,
    tokenizer=tokenizer,
    aggregation_strategy='simple',  # merge subword tokens
)

# ── Extract entities from Stripe dispute texts ──────────────────────
dispute_texts = [
    "I made a payment of $25 to DoorDash on 15th March 2026.",
    "Amazon charged me twice for an order from New York warehouse.",
    "Amazon debited $49.99 from my Chase account yesterday.",
]

print("Named Entity Recognition on payment disputes:")
for text in dispute_texts:
    entities = ner_pipeline(text)
    print(f"
  Text: '{text}'")
    for ent in entities:
        print(f"    {ent['entity_group']:<6} '{ent['word']}'  "
              f"score={ent['score']:.3f}")

# ── Fine-tuning BERT for custom NER ───────────────────────────────────
# Custom entities: MERCHANT, AMOUNT, DATE, ACCOUNT
print("""
# Custom NER training pattern:
# 1. Annotate data in BIO format:
#    tokens: ["Paid", "Rs", "500", "to", "DoorDash"]
#    labels: ["O",  "B-AMOUNT", "I-AMOUNT", "O", "B-MERCHANT"]
#
# 2. Load model:
#    model = AutoModelForTokenClassification.from_pretrained(
#        'distilbert-base-uncased',
#        num_labels=len(label_list),  # B-MERCHANT, I-MERCHANT, B-AMOUNT, ... O
#    )
#
# 3. Key difference from sequence classification:
#    - Labels are per-token not per-sequence
#    - Subword tokens get label -100 (ignored in loss)
#    - Only the first subword of each word gets the real label
#
# 4. Tokenisation with word_ids alignment:
def tokenise_and_align(examples, tokenizer, label2id):
    tokenized = tokenizer(examples['tokens'], is_split_into_words=True,
                          truncation=True, max_length=128)
    all_labels = []
    for i, labels in enumerate(examples['ner_tags']):
        word_ids   = tokenized.word_ids(batch_index=i)
        prev_word  = None
        label_ids  = []
        for word_id in word_ids:
            if word_id is None:
                label_ids.append(-100)   # special tokens ignored
            elif word_id != prev_word:
                label_ids.append(label2id[labels[word_id]])
            else:
                label_ids.append(-100)   # subword tokens ignored
            prev_word = word_id
        all_labels.append(label_ids)
    tokenized['labels'] = all_labels
    return tokenized
""")

A content moderation team routing every message through even a cheap generative API call at a fraction of a cent and a few hundred milliseconds of latency each would face tens of thousands of dollars a day in inference cost alone, plus a moderation queue with far more latency than the product can tolerate. The same volume through a fine-tuned DistilBERT model, running on a modest, fixed number of GPUs, costs a small fraction of that and responds in single-digit milliseconds. This is why many teams land on a two-stage architecture instead of picking one approach exclusively: an encoder model handles the full volume as a cheap first pass, and only the small share of genuinely ambiguous cases — where the encoder's own confidence sits near the decision boundary — get escalated to a slower, more expensive generative model for a judgment call.

# ── A common production pattern: encoder-first, LLM-on-escalation ─────
# Full volume goes through a cheap, fast, fine-tuned encoder.
# Only the genuinely ambiguous cases pay the latency and cost of a
# generative model call.

CONFIDENCE_ESCALATION_BAND = (0.35, 0.65)   # near the decision boundary

def moderate_message(text, encoder_model, llm_client):
    probs = encoder_model.predict_proba([text])[0]
    confidence = probs.max()

    if CONFIDENCE_ESCALATION_BAND[0] < confidence < CONFIDENCE_ESCALATION_BAND[1]:
        # Ambiguous — worth the extra cost and latency of a generative judgment
        return llm_client.classify(text)

    # Confident — the cheap encoder call is the final answer
    return probs.argmax()

# In practice this cuts LLM calls to a small fraction of total volume —
# often under 5-10% — while keeping the accuracy benefit of a generative
# model exactly where the encoder alone would have been unreliable.

NSP was one of BERT's two original pretraining objectives, alongside masked language modelling, and it was presented as necessary for tasks that require understanding relationships between sentences (like question answering). RoBERTa's 2019 ablation directly tested this by removing NSP entirely and training MLM alone for longer on more data — and matched or exceeded original BERT's results across the board. The actual reason NSP looked useful in the original BERT paper wasn't the objective itself; it was that the NSP-trained runs happened to also get more training data and time. Once that confound was controlled for, NSP contributed nothing and was dropped from essentially every BERT successor since.

[CLS] is just a special token prepended to every input sequence at position 0 — there is nothing architecturally different about how attention treats it compared to any other token; it attends to and is attended to exactly like every other position. Its final hidden state only becomes a useful sentence-level summary because it is explicitly trained to be one — during pretraining, [CLS]'s output feeds the NSP classifier, and during fine-tuning it feeds whatever task-specific head you attach for classification. In a hypothetical BERT with random, untrained weights, [CLS]'s hidden state is exactly as meaningless as any other token's — the "summary" property is a learned behaviour from the training objective, not a structural guarantee from where the token sits in the sequence.

This is precisely the problem the 80/10/10 rule exists to fix. If every masked position were simply replaced with the literal [MASK] token, the model would only ever need to predict from context around [MASK] tokens — but [MASK] never appears in real fine-tuning or inference data, which creates a mismatch between what the model practiced on and what it will actually see in production. The 80/10/10 split — 80% of selected tokens become [MASK], 10% become a random token, 10% stay unchanged — forces the model to keep a robust, context-sensitive representation for every token, not just the ones that happen to be masked, because it can never be sure whether the token in front of it is genuine, corrupted, or masked without checking context regardless.

DistilBERT is roughly 40% smaller than BERT-base by parameter count and retains about 97% of its performance on standard benchmarks — for many production classification and NER tasks, that gap is invisible relative to the 60% latency improvement it buys. On small labelled datasets (a few hundred examples), even simpler non-Transformer baselines like TF-IDF plus logistic regression can outperform a fully fine-tuned BERT, because BERT has enough parameters to overfit before it ever benefits from its pretrained knowledge. Model size is one variable among several — data size, domain match, latency budget, and overfitting risk usually matter more than raw parameter count when picking which encoder-family model to actually deploy.

Bidirectional attention and causal attention are not a strictly-better-vs-strictly-worse pair — they are a direct architectural trade-off. Bidirectionality is exactly what makes BERT unable to generate text autoregressively: if every token could already see every future token during training, predicting the next token would be trivial and would leak the answer, which is why BERT cannot be used for open-ended generation without substantial modification. GPT's causal mask sacrifices access to future context specifically so that next-token prediction remains a genuine prediction problem, which is what makes autoregressive generation possible at all. Each architecture is specialised for what its masking allows it to do — understanding tasks that benefit from full context versus generation tasks that require the model to only ever see the past — neither one dominates the other in general.

Next-token prediction requires that the model, when predicting token t+1, only has access to tokens 1 through t — otherwise the "prediction" is trivial, since the answer is sitting right there in the input. That requirement forces a causal mask, which is exactly what makes GPT unidirectional: each token can only attend to what came before it. BERT's whole design goal is the opposite — every token should attend to every other token in both directions, because that full context is what makes it strong at understanding tasks. Applying a causal mask to get next-token prediction working would directly contradict BERT's bidirectional design, so it needs an objective that can be computed with full bidirectional context without leaking the answer — which is exactly what masked language modelling provides: hide 15% of tokens, let the model see everything else in both directions, and predict only the hidden ones.

If masking always meant literally inserting the [MASK] token, the model would learn a representation that is only ever exercised at [MASK] positions — but at fine-tuning and inference time, [MASK] never appears in real input, so the model's representations for ordinary, non-masked tokens would never get the same training pressure to be context-aware. The 80/10/10 split forces every token's representation to stay robust regardless of what it turns out to be: of the 15% of tokens selected for the MLM task, 80% are replaced with [MASK] (learn to predict from context), 10% are replaced with a random wrong token (learn to notice when a token doesn't fit and correct it using context, not just copy it), and 10% are left unchanged (learn to still use context even when the input token is already correct, since the model can't tell this case apart from the other two). The net effect: BERT can't develop a shortcut of "just copy whatever token is in front of me," because 20% of the time that would be wrong.

The token embedding is the standard WordPiece lookup table, one vector per vocabulary entry. The position embedding is a learned vector per sequence position (0 through 511), added so the model can distinguish token order despite attention being permutation-equivariant on its own — same reason every Transformer needs positional information, GPT included. The segment embedding is BERT-specific: it's a binary signal (sentence A vs sentence B) added to every token, needed because BERT was designed from the start to take two-sentence inputs for tasks like question answering (question + context) or the original NSP objective (sentence A + sentence B), and the model needs an explicit signal for which sentence each token belongs to since bidirectional attention alone can't infer a sentence boundary from [SEP] tokens reliably at every layer. GPT never needs this because it processes a single continuous stream of tokens for autoregressive generation — there's no fixed two-segment input structure to disambiguate.

It's a direct lesson in controlling for confounds before crediting a specific design choice. The original BERT paper presented MLM and NSP together as a package, so it looked like NSP was contributing to BERT's strong results. RoBERTa isolated the variable: same architecture, same MLM objective, NSP removed, but trained for longer on substantially more data — and it matched or beat BERT anyway. That means the original NSP-included runs' apparent success was confounded with the additional data and training time, not evidence that NSP itself was doing useful work. The broader interview-relevant point: when someone claims a specific training component is "necessary" based on a paper that changed several things at once, the right instinct is to ask what was actually ablated — a single change tested in isolation is much stronger evidence than a bundle of changes that all shipped together and happened to work.

Sequence classification uses only the final hidden state at the [CLS] position — a single 768-dimensional vector — fed into a linear layer that outputs class logits for the whole input; the rest of the sequence's hidden states are simply discarded at the head. Token classification (NER) instead uses every token's final hidden state independently, each fed through the same linear layer to produce a label per token — the model is predicting a tag for every position, not one tag for the whole sequence. The architecture-level difference is just where you attach the head (one position vs all positions) — the pretrained encoder underneath is identical. What commonly goes wrong is label alignment: WordPiece splits a single annotated word into multiple subword tokens ("Stripe" becomes "St" + "##ripe"), so you can't naively assign your word-level label to each resulting token — you have to use the tokeniser's word_ids() to give the real label only to a word's first subword and mask out continuation subwords with label -100 so they're ignored in the loss, otherwise the model is trained on silently corrupted labels.