Embeddings: A Vector of Meaning

Anchored in rag_pipeline.py → class Embedder · config.py → MODEL_NAME = "all-MiniLM-L6-v2"

You already have a working RAG. You already call embedder.embed_one("how to fix neovim folding") and the system finds the right blog post. But if someone asked you "what is the number that came out?", what would you say?

By the end of this lesson you will. We'll go from a vague sense that "the model turns text into numbers" to a precise mental model: a 384-dimensional unit vector in a space where distance ≈ meaning. Then we'll look at the four lines in your codebase that turn that idea into a search engine.

1. The mental model

An embedding is a function that maps a piece of text to a list of real numbers. For our model that list has exactly 384 numbers. That's it. No magic, no symbols, no floating-point ghosts. Just a list:

[0.021, -0.118, 0.443, ..., 0.077]   # 384 numbers total

But here's the part that makes it useful: the model is trained so that sentences with similar meaning land near each other in this 384-dimensional space. So when I embed two sentences and their vectors point in roughly the same direction, that's a mathematical claim that they mean roughly the same thing.

2. Why 384 dimensions?

You could embed text in 2 dimensions (and the popular "king − man + woman ≈ queen" demos use exactly that intuition). But 2 dimensions can't distinguish between "neovim folding", "vim folding", and "folding laundry".

More dimensions = more axes to encode meaning along. The trade-off:

• More dimensions = richer representation, but slower similarity computation, more memory, and the risk of overfitting on a small corpus.
• Fewer dimensions = faster, but coarser. You can't tell apart concepts that share many axes.

Our model, all-MiniLM-L6-v2, chose 384 as a sweet spot. It was distilled from a larger model (MiniLM = Mini Language Model) — trained to mimic a bigger model's outputs while being 5× smaller and faster. The L6 means "6 transformer layers", and v2 is the second training run. See the model card on Hugging Face →

3. The math you'll see in the code: cosine similarity

Once you have two vectors, you need a way to say "how close are they". Three options show up in production systems:

1. Euclidean distance — straight-line distance. Penalizes vectors of different magnitudes.
2. Dot product — fast, sensitive to magnitude.
3. Cosine similarity — dot product normalized by length. Measures direction only.

We use cosine similarity. The formula is the one you learned in high school:

cos(θ) = (A · B) / (‖A‖ · ‖B‖)

In NumPy, that's three lines. Read this — it's the same code, just with names:

import numpy as np
from numpy.linalg import norm

def cosine_similarity(a, b):
    return np.dot(a, b) / (norm(a) * norm(b))

The output is in [-1, 1]. For normalized text vectors, you'll typically see [0.2, 0.95] in practice — never quite 1.0 unless the inputs are identical.

4. Now read your own code

Open rag_pipeline.py and find the Embedder class. There are exactly four lines that turn "text" into "the magic vector" worth examining:

rag_pipeline.py · Embedder.__init__

self.model = SentenceTransformer(model_name)
dim = self.model.get_embedding_dimension()

This loads the model. Internally, it's a stack of 6 transformer encoder layers followed by a mean pooling layer that collapses the per-token outputs into a single 384-float vector. SBERT documentation has the architecture diagram →

rag_pipeline.py · Embedder.embed

return self.model.encode(
    texts,
    show_progress_bar=True,
    normalize_embeddings=True
)

Three flags worth understanding:

• show_progress_bar=True — UX, irrelevant to the math.
• normalize_embeddings=True — every output has length 1. We discussed why.
• The return type is a np.ndarray of shape (n_texts, 384).

rag_pipeline.py · Embedder.embed_one

return self.model.encode([text], normalize_embeddings=True)[0].tolist()

One sentence in, one vector out. The [0] is because encode always returns a batch; we asked for a batch of one. The .tolist() converts to a Python list (LanceDB wants lists, not NumPy arrays).

config.py

MODEL_NAME = "all-MiniLM-L6-v2"

One constant. Change this single string and you've migrated the entire embedding pipeline to a different model — a different dimensionality, different speed/quality tradeoff, different cost. The whole architecture is parameterized by it.

5. What the encoder actually does (intuition, no math)

If you read "transformer" and feel your eyes glaze over, here's the one-paragraph version:

The encoder reads your text token by token (roughly: sub-word pieces, like "un" + "##believ" + "##able"). For each token, it produces a vector. Then it does a series of "attention" steps: each token looks at every other token and decides how much to mix in. By the end, each token's vector has been informed by the whole sentence's context.

Then mean pooling averages all the token vectors into one. That's the embedding.

That's it. You don't need to derive the attention equations to use the model — but knowing that "context" is computed across the whole sentence is the difference between treating this as a magic black box and treating it as a function you can reason about.

When you want the full picture, Lilian Weng's Transformer Family post and Karpathy's "Let's build GPT" are the canonical deep dives.

6. Hands-on exercise

"""Quick embeddings playground. Run: uv run python playground.py"""
import numpy as np
from rag_pipeline import Embedder

embedder = Embedder()

# Three sentences. Two are paraphrases. One is unrelated.
sentences = [
    "How do I configure Neovim folding?",
    "What's the best way to set up code folding in Neovim?",
    "How do I cook a hard-boiled egg?",
]

vecs = embedder.embed(sentences)

def cos(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

print("\nPairwise cosine similarity:\n")
for i in range(len(sentences)):
    for j in range(i + 1, len(sentences)):
        score = cos(vecs[i], vecs[j])
        print(f"  {score:.3f}  '{sentences[i][:40]}...'  vs  '{sentences[j][:40]}...'")

print(f"\nEach vector has {vecs.shape[1]} dimensions.")
print(f"Vector length (should be ~1.0): {np.linalg.norm(vecs[0]):.4f}")
print(f"First 5 numbers of sentence 0: {vecs[0][:5]}")

Pairwise cosine similarity:
  0.78  'How do I configure Neovim folding?...'  vs  'What's the best way to set up...'
  0.05  'How do I configure Neovim folding?...'  vs  'How do I cook a hard-boiled egg...'
  0.04  'What's the best way to set up code ...'  vs  'How do I cook a hard-boiled egg...'

Each vector has 384 dimensions.
Vector length (should be ~1.0): 1.0000
First 5 numbers of sentence 0: [-0.0523  0.0821  0.0334 -0.0911  0.0203]

7. What you now know

• ✅ An embedding is a list of 384 floats produced by an encoder neural network.
• ✅ The encoder is trained so that similar meanings produce similar vectors (cosine similarity).
• ✅ Our model is a distilled 6-layer transformer, 80 MB on disk, runs on CPU.
• ✅ The single constant MODEL_NAME = "all-MiniLM-L6-v2" parameterizes the entire embedding pipeline.
• ✅ "Cosine similarity" is just the cosine of the angle between two vectors; with normalized vectors, it's a dot product.
• ✅ The line in your code that actually returns the number is self.model.encode([text], normalize_embeddings=True)[0].tolist().

8. What you don't know yet (next lessons)

• How do you do "search" with embeddings? (k-NN, ANN, vector databases — Lesson 2)
• Why does cosine similarity miss exact keyword matches? (Lexical search — Lesson 3)
• How do you combine vector and keyword rankings? (RRF — Lesson 4)

Citations & further reading

1. Vicki Boykis, What Are Embeddings? — the single best starting point for engineers new to embeddings.
2. Pinecone, Dense Vector Embeddings — clean visual explanations of cosine similarity and normalization.
3. HuggingFace model card: all-MiniLM-L6-v2 — the model we use, with benchmarks and training data.
4. Sentence-Transformers documentation — API reference for the library we depend on.
5. Lilian Weng, The Transformer Family v2 — deep dive on the architecture. Read after the practical side clicks.
6. Lilian Weng, Learning Word Embedding — history from word2vec to modern encoders.