Embeddings in Practice — Chunking and Similarity Search

Generate embeddings, choose a chunking strategy, and run cosine-similarity search over your own documents.

Est. 30 min Updated Suggest a fix
// Part 01 — Concept

An Embedding Maps Meaning to a Vector

An embedding model maps text into a numeric vector. Text with related meaning may land near one another in the embedding space, which makes semantic search possible: a learner can search "my payment is stuck" and find a help article titled "Pending UPI transactions" even when the words differ. The vector is a representation for retrieval, not a human-readable explanation or a truth score.

Similarity depends on the embedding model and its training. Use a compatible model for documents and queries, and check the provider guidance for normalization and dimensions. Never compare vectors produced by unrelated models as though their coordinates had the same meaning.

StepInput / outputDecision to make
PrepareSource documents → clean textWhich content is current, allowed and searchable?
ChunkText → small overlapping passagesHow much context should each result contain?
EmbedPassage text → vectorWhich configured model and batching limits?
RetrieveQuestion vector → nearest chunksWhich similarity, filters and top-k?
💡 Note
An embedding can encode sensitive text just as the original passage does. Apply data minimization, access controls, retention limits and deletion handling to both text and vectors.
// Part 02 — Prepare

Chunk Documents at Meaningful Boundaries

A vector represents the passage you embed. If a chunk is too large, it may combine unrelated topics; if too small, it may lose the context that makes an answer useful. Begin with paragraph or heading boundaries, preserve useful titles, and use modest overlap only when a sentence or concept may cross a boundary. Token limits depend on your embedding provider and model, so consult its current documentation rather than assuming a universal chunk size.

chunk_text.py
python
from collections.abc import Iterator


def paragraph_chunks(text: str, max_chars: int = 1200) -> Iterator[str]:
    """A simple baseline that keeps paragraphs together when possible."""
    if max_chars < 1:
        raise ValueError("max_chars must be positive")
    paragraphs = [part.strip() for part in text.split("\n\n") if part.strip()]
    current = ""
    for paragraph in paragraphs:
        if current and len(current) + 2 + len(paragraph) > max_chars:
            yield current
            current = ""
        if len(paragraph) > max_chars:
            # Production code should split long paragraphs on sentence/token boundaries.
            if current:
                yield current
                current = ""
            for start in range(0, len(paragraph), max_chars):
                yield paragraph[start:start + max_chars]
        else:
            current = f"{current}\n\n{paragraph}" if current else paragraph
    if current:
        yield current

Character count is only a simple demonstration. Production chunkers should consider tokenizer limits, headings, tables, code blocks, document structure, and whether the content can be shown as a coherent excerpt. Store the source document ID, version, section, and offsets so a retrieved passage can be cited and refreshed later.

🎯 Pro Tip
Keep metadata with each chunk. If a learner asks about a specific module, a track or lesson filter can improve relevance and prevent unrelated content from appearing in the result list.
// Part 03 — Generate

Embed in Batches and Keep an Index

Embedding calls can be network-bound and quota-limited. Send small batches supported by your provider, retain a stable chunk identifier, and persist the embedding model/configuration alongside each vector. If the source text or model changes, re-embed the affected records. Do not silently mix vector spaces in one index.

embed_batch.py
python
from dataclasses import dataclass
from collections.abc import Sequence

@dataclass
class Chunk:
    chunk_id: str
    text: str
    metadata: dict


def embed_chunks(chunks: Sequence[Chunk], *, client, model: str, batch_size: int = 64):
    for start in range(0, len(chunks), batch_size):
        batch = chunks[start:start + batch_size]
        # Implement this adapter using the selected provider's documented SDK.
        response = embedding_adapter.create(
            client=client,
            model=model,
            inputs=[chunk.text for chunk in batch],
        )
        for chunk, vector in zip(batch, response.vectors, strict=True):
            vector_store.upsert(
                id=chunk.chunk_id,
                vector=vector,
                metadata={**chunk.metadata, "embedding_model": model},
            )

embedding_adapter is intentionally provider-neutral: SDK method names and batching limits vary. Keep credentials server-side, obey request and token limits, and implement bounded retry handling for transient errors. Use a stable content hash to skip unchanged chunks, but do not expose that hash as an access-control mechanism.

  1. Normalize and version source text while preserving its attribution and permissions.
  2. Chunk by document structure and record chunk IDs, source IDs, headings and offsets.
  3. Request vectors in supported batches using one configured embedding model.
  4. Persist vectors, metadata, model identity and an index version atomically where possible.
  5. Re-embed on source or model changes and remove deleted or unauthorized content from the index.
// Part 04 — Similarity

Compute Cosine Similarity with NumPy

Cosine similarity compares the angle between two vectors. A score closer to one means the vectors point in a more similar direction; the useful score distribution depends on the embedding model and dataset. Normalize vectors or divide by their norms in the formula, and handle zero vectors explicitly.

cosine_search.py
python
import numpy as np


def cosine_similarity(query: np.ndarray, matrix: np.ndarray) -> np.ndarray:
    query_norm = np.linalg.norm(query)
    row_norms = np.linalg.norm(matrix, axis=1)
    if query_norm == 0 or np.any(row_norms == 0):
        raise ValueError("cosine similarity is undefined for zero vectors")
    return (matrix @ query) / (row_norms * query_norm)

# Demonstration vectors only. Replace these with vectors from one embedding model.
query = np.array([0.8, 0.2, 0.1], dtype=np.float32)
documents = np.array([
    [0.7, 0.3, 0.1],
    [0.1, 0.8, 0.4],
    [0.6, 0.2, 0.2],
], dtype=np.float32)
scores = cosine_similarity(query, documents)
for index in np.argsort(scores)[::-1]:
    print(index, float(scores[index]))

This brute-force matrix comparison is a useful baseline for a small collection and for validating a future index. It costs work proportional to the number of vectors, so large corpora use a vector database or an approximate nearest neighbour index. Measure recall and latency against this exact baseline before choosing index settings.

📌 Real World Example
Searching course material can return a Python lesson for "how do I repeat an operation for each row?" even if the lesson uses the word "loop" rather than the phrase in the question. Retrieval still needs filters and evaluation to keep the result relevant.
// Part 05 — Retrieve

Embed the Query and Return Useful Context

At query time, embed the question with the same model family and compatible settings used for the indexed chunks. Retrieve more candidates than you will finally show, apply access and metadata filters, then select a small number of useful passages. Include source titles and citation metadata in the context passed to a generation step so the answer can point back to evidence.

retrieve.py
python
def retrieve(question: str, *, user, index, embed_query, top_k: int = 5):
    query_vector = embed_query(question)
    candidates = index.search(
        vector=query_vector,
        limit=top_k,
        filters={"visibility": "published", "track": "generative-ai"},
    )
    # Apply user- and document-level access checks before constructing context.
    allowed = [item for item in candidates if can_read(user, item.metadata)]
    return [
        {
            "text": item.text,
            "source": item.metadata["source_title"],
            "chunk_id": item.id,
            "score": item.score,
        }
        for item in allowed
    ]

Do access filtering as part of retrieval whenever the store supports a secure metadata predicate, and still enforce it in application logic where appropriate. Filtering after retrieving an unbounded set may be inefficient; returning a forbidden passage to a prompt is a data leak even if the final answer is hidden from the user.

ChoiceTrade-off to test
Chunk lengthContext completeness versus topical focus
OverlapBoundary continuity versus duplicate results and index size
Top-kEvidence coverage versus prompt size and irrelevant passages
Metadata filtersPrecision and access safety versus fewer candidates
// Part 06 — Quality

Inspect Retrieval Before Tuning Generation

If an answer is wrong, first inspect which chunks were retrieved. Keep a small evaluation set of realistic questions with expected source documents and measure whether the right evidence appears near the top. Review failures caused by stale content, bad chunk boundaries, missing metadata, vocabulary mismatch, or access filters. Only after retrieval is sound should you change the answer prompt or model.

inspect_results.py
python
def inspect_query(question, results):
    print("QUESTION:", question)
    for rank, item in enumerate(results, start=1):
        print(f"\n{rank}. score={item['score']:.3f} source={item['source']}")
        print(item["text"][:400])

# Add representative queries and have a reviewer compare these results to expected sources.

Similarity scores are not probabilities of correctness. A score threshold that works for one domain or model may not work for another. Use a validation set and look at ranking and answer quality together. Track changes to the embedding model, chunker, and index configuration so retrieval regressions can be reproduced.

⚠️ Important
Re-embedding with a new model creates a new vector space. Do not query old vectors with a new model and assume nearest neighbours remain meaningful; migrate and validate the index as a coordinated change.
// Part 07 — Pitfalls

Common Embedding Mistakes

Treat an embedding as a summary or factual answer

Why: A vector is a numeric representation used for similarity, not explanatory text.

Fix: Retrieve source passages and let downstream logic or generation use the evidence.

Embed an entire long document as one vector

Why: Unrelated topics blur together and the best passage may not match the query.

Fix: Chunk by structure, preserve metadata, and evaluate chunk boundaries.

Mix vectors from different embedding models

Why: Coordinates from distinct models do not necessarily share a vector space.

Fix: Store model identity and re-index when changing embedding models.

Use a high similarity score as proof of relevance

Why: Score scale and ranking quality vary by model and corpus.

Fix: Evaluate against real queries, inspect results and tune with measured evidence.

// Part 08 — Practice

Create a Mini Course Search Index

Practice

Build and inspect a small semantic retrieval pipeline

  1. Choose a few public or synthetic course paragraphs, add a title and source ID, then create chunks that preserve heading context.
  2. Use one configured embedding provider to create vectors in batches; store the model identifier with each chunk.
  3. Implement cosine similarity in NumPy and rank a small set of vectors, including a test for a zero vector.
  4. Try a question that uses different words from the source and inspect the top results and metadata.
  5. Add a track filter and a user access check; verify an unauthorized chunk never enters the prompt context.
  6. Write three retrieval evaluation questions with expected source IDs and record where chunking or ranking fails.

Hint: Keep a brute-force exact search for the small demo. It gives you a reference when you later add an approximate index.

🎯 Key Takeaways

  • ✓Embeddings are numeric representations that make semantic similarity search possible; they are not answers or truth scores.
  • ✓Use compatible embedding models for query and document vectors and record the model identity.
  • ✓Chunk by structure, preserve source metadata and batch embedding requests within provider limits.
  • ✓Cosine similarity is a useful baseline; brute-force search does not scale to every corpus.
  • ✓Apply access controls and metadata filters before retrieved text reaches a model.
  • ✓Evaluate retrieval with representative queries and inspect evidence before tuning generation.

Next

What comes next

You can embed and rank a small collection. Next, move storage and nearest-neighbour search into vector database systems.

Lesson 07 · Vector Databases — pgvector, Chroma, Pinecone and FAISS →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...