Introduction to Embeddings: Turning Text into Vectors

Introduction to Embeddings: Turning Text into Vectors

Keyword search fails the moment meaning matters. A user asks "how do I get my money back" and your documentation says "refunds are processed within five business days"; no shared vocabulary, no match. Embeddings solve this by representing text as points in a high-dimensional space where semantic closeness becomes geometric closeness. That one idea underpins semantic search, recommendation, clustering, deduplication, and every RAG system being built today.

This article builds the concept from scratch: what a vector actually encodes, how similarity is computed, what happens inside an embedding model, how to choose one, and the practical arithmetic that decides whether your idea survives contact with a million documents.

Meaning as Coordinates#

An embedding model maps any text to a fixed-length array of floats, commonly 384 to 3072 dimensions. The model is a transformer trained so that texts used in similar contexts land near each other in that space. Nobody hand-designs what each dimension means; the geometry emerges from training, and individual axes are famously uninterpretable even though neighborhoods are robust. "Refund policy" and "money back guarantee" end up adjacent despite sharing almost no words, because they appear in similar linguistic company everywhere in the training corpus.

The contrast with older representations makes the value obvious. Keyword search treats text as sets of tokens: overlap or nothing. TF-IDF weights those tokens but still cannot bridge synonyms. An embedding collapses paraphrase, translation across phrasing styles, and even cross-language pairs into the same neighborhood.

Flattened to two dimensions for human eyes, a corpus of documents organizes itself by topic without anyone labeling anything:

Similarity Is Geometry#

Given two vectors, the standard relevance measure is cosine similarity: the cosine of the angle between them, independent of their lengths. The formula is one line of linear algebra:

import numpy as np

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

A tiny worked example with 2D stand-ins for real embeddings:

a = np.array([1.0, 0.0])      # "refund timeline"
b = np.array([0.8, 0.6])      # "when do I get my money back"
c = np.array([-1.0, 0.0])     # "payment enforcement policy"

print(round(cosine(a, b), 3))   # 0.8    strongly related
print(round(cosine(a, c), 3))   # -1.0   opposite direction

Scores near 1 mean same neighborhood, near 0 unrelated, negative means opposing directions. Geometrically, cosine measures the angle between vectors while ignoring how long they are:

Two implementation facts matter in production. First, most embedding models ship unit-normalized vectors (length exactly 1), which makes cosine identical to a plain dot product, letting vector databases use fast dot-product kernels:

resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
vecs = np.array([d.embedding for d in resp.data])
norms = np.linalg.norm(vecs, axis=1, keepdims=True)
vecs = vecs / norms          # now dot product == cosine
sims = vecs @ query_vec      # one matrix multiply answers everything

Second, always compute similarities against vectors produced by the same model. Cross-model comparisons are meaningless; coordinates from different spaces share no geometry.

Inside the Embedding Model#

Embedding models are transformers, but they run as encoders rather than generators. Text enters, passes through transformer layers producing contextual representations for every token, then pooling collapses the sequence into one fixed vector: take the final-layer token embeddings and average them, or route through a designated pooling head. One sentence in, one vector out, no generation loop.

Training is where the magic gets installed. These models learn via contrastive objectives on massive paired datasets: question-answer pairs, duplicated web documents, natural language inference triplets. Related pairs are pulled together in vector space while unrelated pairs are pushed apart, with in-batch negatives providing thousands of contrast examples per step. The result is a model whose geometry directly encodes the relationship judgments the training data expressed.

Choosing an Embedding Model#

The decision has three axes: quality, cost/latency, and deployment constraints.

Hosted APIs (OpenAI's text-embedding-3-small and -large, Cohere's embed family, Voyage) offer strong quality with zero infrastructure and per-token pricing. Open-source options (the sentence-transformers family like all-MiniLM-L6-v2, BGE models, E5) run anywhere, cost nothing per call at inference time, and range from fast-and-lightweight to competitive-with-APIs:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")     # 384 dims, CPU-friendly
vecs = model.encode(
    ["Refunds take five business days.",
     "How do I get my money back?"],
    normalize_embeddings=True,
)
print(vecs.shape)                                    # (2, 384)

Practical guidance after shipping several of these: start with a hosted API for product work because quality differences are real and switching costs later are one re-indexing job; go local when volume makes per-token pricing dominate costs, when data cannot leave the premises, or when latency floors matter more than ceiling quality. Check benchmark leaderboards like MTEB for current rankings rather than trusting year-old blog posts, but treat leaderboard deltas as hypotheses: your domain's documents may rank models differently than generic benchmarks do.

Two newer ideas worth knowing exist. Matryoshka-trained embeddings let you truncate output dimensions adaptively (use 256 of 1536 dims for cheap first-pass retrieval, full dims for reranking). And instruction-tuned embedders accept task prefixes ("Represent this sentence for searching relevant passages:") that measurably improve retrieval when applied consistently to the right side of the pair.

One nuance about dimensions before choosing: more is not automatically better. Higher-dimensional vectors can encode finer distinctions, but they cost proportionally more to store and scan, and past the point where the training data supports them, extra dimensions mostly encode noise. A well-trained 768-dimension model routinely beats a poorly trained 1536-dimension one on real tasks. Judge models by measured performance on your data, never by dimension count.

Test Against Your Own Corpus#

Benchmarks rank models; only your documents decide. Twenty minutes of smoke testing beats weeks of leaderboard archaeology:

import numpy as np

def embed_with(model_name: str, texts: list[str]) -> np.ndarray:
    """Swap any provider or local model behind this one call."""
    if model_name.startswith("local:"):
        from sentence_transformers import SentenceTransformer
        return SentenceTransformer(model_name[6:]).encode(
            texts, normalize_embeddings=True)
    resp = client.embeddings.create(model=model_name, input=texts)
    return np.array([d.embedding for d in resp.data])

def smoke_test(model_name: str, cases) -> float:
    """Each case: (query, expected_match, distractors)."""
    correct = 0
    for query, match, distractors in cases:
        vecs = embed_with(model_name, [query, match] + distractors)
        v = vecs / np.linalg.norm(vecs, axis=1, keepdims=True)
        sims = v[0] @ v[1:].T
        correct += int(np.argmax(sims) == 0)
    return correct / len(cases)

Write fifteen to twenty such triples from your actual domain, including your hardest phrasings and known synonym gaps, then compare two or three candidate models on that set. The winner on your corpus is your model regardless of what generic benchmarks claim, and the case file becomes a permanent regression asset for future model swaps.

Beyond Search#

Search is the headline application but not the only one, and the same geometry powers three more workhorses. Deduplication: near-duplicate support tickets cluster at cosine similarity above roughly 0.95 depending on the model, letting you collapse duplicates before they ever reach an LLM:

def duplicate_pairs(vecs, threshold=0.95):
    sims = vecs @ vecs.T
    np.fill_diagonal(sims, -1)
    i, j = np.where(sims > threshold)
    return [(int(a), int(b)) for a, b in zip(i, j) if a < b]

Clustering runs k-means or HDBSCAN over raw vectors to discover topic structure without labels, which is the fastest way to find out what is actually inside a document pile you inherited. Nearest-centroid classification computes a mean vector per class and routes new inputs by similarity, solving many triage problems with zero training infrastructure. All three reuse the exact index you already built for search, which is part of why embeddings became infrastructure rather than a feature.

The Arithmetic of Scale#

Embedding storage math is refreshingly concrete, and worth doing before any architecture meeting. A 1536-dimension float32 vector occupies 6,144 bytes, about 6 KB. One million documents therefore cost roughly 6 GB of raw vectors before index overhead, and approximate-nearest-neighbor structures add their own multiplier. Halve the dimensions and you halve the bill; quantization to int8 cuts another fourfold with modest recall loss. None of this is exotic; it is multiplication, and doing it early prevents the classic surprise where the prototype was free and month three costs four figures.

Query-side costs deserve equal attention. Every incoming question triggers one embedding call; at ten milliseconds and negligible price that seems trivial until traffic spikes, which is why production systems cache query embeddings for repeated questions and batch document ingestion aggressively (embedding APIs charge per token and reward batches).

Where Embeddings Mislead#

Knowing failure modes separates people who have shipped this from people who have read about it. Similarity is not relevance: the nearest neighbor can be topically adjacent yet answerless, which is why retrieval systems need reranking stages rather than blind trust in top-k. Negation remains weak: "refund approved" sits uncomfortably close to "refund denied" because surface forms dominate subtle logical flips. Short queries against long documents create asymmetric matching problems that naive pipelines handle badly. Domain drift degrades quietly: a general-purpose model underperforms on legal or medical jargon, and you discover it through missed retrievals, not error messages. Model swaps invalidate everything: change embedders and every stored vector becomes noise, forcing a full re-embedding pass, which is why the model choice deserves to be wrapped behind an interface from day one. And stale indexes lie silently: documents updated without re-embedding produce confidently wrong citations, the worst kind of failure because users see authoritative-looking answers built from outdated evidence.

None of these argue against embeddings; they argue against treating embeddings as the whole system rather than the foundation of it.

What Vectors Buy You. Embeddings convert the fuzzy problem of meaning into the concrete problem of distance, and everything downstream inherits that concreteness: search becomes sorting, deduplication becomes thresholds, recommendations become neighbor queries, and RAG becomes fetch-the-relevant-context-before-asking. Start simple: one solid embedding model behind an interface, normalized vectors, cosine similarity, honest evaluation of retrieval hit rates on your own questions. The geometry does the rest, and unlike most abstractions in this field, it will still be true next year.