Chunking Strategies for RAG: Fixed, Semantic, and Recursive

Chunking Strategies for RAG: Fixed, Semantic, and Recursive

Chunking is the least glamorous stage of retrieval-augmented generation and the one that quietly caps its quality ceiling. You can swap embedding models, tune top-k, bolt on rerankers, and still ship mediocre answers if every chunk slices thoughts in half. The reason is mechanical: an embedding pools the meaning of everything inside one chunk, so a chunk containing half of paragraph three and half of paragraph four embeds as neither. Retrieval can only ever be as good as your boundaries.

This article covers the four strategies that matter in practice: fixed-size splitting, recursive character splitting, semantic chunking, and structure-aware parsing. For each: how it works, working code, where it wins, and where it wastes money or loses facts.

Why Chunk Boundaries Decide Everything#

Three forces interact at chunk time. Embeddings summarize: longer chunks blur distinct topics into one average vector that matches nothing sharply. Context windows constrain: each retrieved chunk ships inside your prompt, so oversized chunks burn budget and dilute attention across filler. And answer completeness depends on it: a fact spanning two chunks must appear intact somewhere, which is what overlap and boundary detection are for.

Every strategy below is a different answer to one question: where does a unit of meaning end?

Fixed-Size Chunking: The Blunt Instrument#

Split every N characters (or tokens), optionally with M characters of overlap:

def fixed_chunks(text: str, size: int = 1000, overlap: int = 150) -> list[str]:
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

It is fast, deterministic, trivially parallel, and produces uniform sizes that make downstream token budgeting predictable. It also cuts wherever the counter happens to land, including mid-word, mid-table-row, and mid-argument. Overlap softens the damage by duplicating boundary neighborhoods, but duplication is not repair; the cut still exists, now with a copy.

Two practical notes if you use it anyway. Count in tokens rather than characters when you can, because embedding models bill and limit in tokens, and character counts drift from token counts by twenty to thirty percent on technical text full of identifiers. And log your cut positions during development: skimming a page of raw cuts tells you within a minute whether your corpus tolerates blind slicing or is being quietly mutilated.

Where it genuinely works: uniform unstructured prose, transcription text, log files, any corpus where semantic units are small relative to chunk size, and quick baselines whose job is to give evaluation a floor. Where it fails visibly: anything with structure, because headings and code fences get shredded indiscriminately.

Recursive Splitting: The Sensible Default#

Recursive splitting tries a hierarchy of separators, falling back only when needed: split on double newlines (paragraphs); if pieces still exceed the limit, split those on single newlines (lines); then sentences; then words; finally raw characters. Structure survives whenever structure exists, and the algorithm degrades gracefully to fixed behavior when it does not.

The concept behind LangChain's widely used RecursiveCharacterTextSplitter, implemented directly:

SEPARATORS = ["\n\n", "\n", ". ", " ", ""]

def recursive_chunks(text: str, size: int = 1000,
                     overlap: int = 150, seps=None) -> list[str]:
    seps = seps or SEPARATORS
    if len(text) <= size:
        return [text.strip()] if text.strip() else []
    sep = next((s for s in seps if s in text), "")
    parts = text.split(sep) if sep else [text[i:i+size]
                                         for i in range(0, len(text), size - overlap)]

    chunks, current = [], ""
    for part in parts:
        candidate = f"{current}{sep}{part}" if current else part
        if len(candidate) <= size:
            current = candidate
            continue
        if current:
            chunks.append(current.strip())
        tail = current[-overlap:] if current else ""
        current = f"{tail}{sep}{part}" if tail else part
        while len(current) > size:                       # piece itself too big
            deeper = recursive_chunks(current, size, overlap, seps[seps.index(sep)+1:])
            chunks.extend(deeper[:-1])
            current = deeper[-1]
    if current.strip():
        chunks.append(current.strip())
    return chunks

Read the flow as policy: prefer paragraph boundaries; accept sentence boundaries; only chop words when nothing larger exists. In production most teams reach for the library version rather than maintaining this themselves (langchain_text_splitters.RecursiveCharacterTextSplitter), but understanding the recursion matters because tuning it, separator order, chunk size, overlap, is exactly where quality gets won back from bad documents.

One customization pays for itself immediately: prepend your own separator candidates based on the corpus. Legal contracts split cleanly on numbered clauses ("\n4.2 "), API docs on heading markers, meeting notes on speaker turns. The algorithm does not care what the separators mean; feeding it domain-aware ones converts a generic splitter into a near-custom parser without writing a parser.

This is my default recommendation for prose corpora: better boundaries than fixed-size at nearly identical cost, and no embedding calls spent deciding anything.

Semantic Chunking: Paying for Awareness#

Semantic chunking asks the data where topics shift instead of assuming punctuation knows. Embed sentences individually, measure distance between adjacent sentence embeddings, and cut where similarity drops below a threshold: a topic boundary shows up as a local spike in dissimilarity.

import numpy as np
from openai import OpenAI

client = OpenAI()

def semantic_chunks(text: str, percentile: float = 80) -> list[str]:
    sents = [s.strip() for s in text.replace("\n\n", " ").split(". ") if s.strip()]
    resp = client.embeddings.create(model="text-embedding-3-small", input=sents)
    vecs = np.array([d.embedding for d in resp.data])
    vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)

    sims = np.sum(vecs[:-1] * vecs[1:], axis=1)          # adjacent similarities
    cut = np.percentile(sims, 100 - percentile)
    breaks = np.where(sims < cut)[0]

    chunks, start = [], 0
    for b in list(breaks) + [len(sents) - 1]:
        chunks.append(". ".join(sents[start:b + 1]))
        start = b + 1
    return [c for c in chunks if c.strip()]

The payoff is coherent units even in rambling documents without clean formatting: meeting transcripts, support email threads, concatenated wiki exports. The price is real: you pay one embedding pass per sentence just to decide boundaries, before indexing embeddings ever begin, roughly doubling ingestion cost. On a ten-thousand-document corpus that difference is measured in dollars and minutes; on a ten-million-document corpus it is measured in hours and real money, which is why semantic chunking tends to appear at the expensive end of pipelines, applied selectively to the messy fraction of the corpus rather than uniformly. Thresholds need tuning per corpus; a percentile that suits dense technical docs shreds chatty ones. And short chunks emerge naturally, which may starve context unless you run a merge pass for undersized neighbors.

My honest position after shipping both: semantic chunking earns its cost on messy conversational corpora and is wasted money on well-edited documentation, where headings already mark the boundaries better than cosine similarity would.

Note what the green point teaches: low similarity between two sentences is not always a boundary worth paying for; sometimes it is just two unrelated sentences inside one coherent section. Pure thresholds over-trigger, which is why practical implementations merge tiny fragments afterward.

Structure-Aware Chunking: Respect the Format#

The highest-leverage strategy for real documentation is not statistical at all: parse the format. Markdown headers delimit sections perfectly; split by heading levels first and recurse within sections when they exceed size limits:

import re

def markdown_chunks(text: str, size: int = 1200) -> list[str]:
    sections, current_title, buffer = [], "intro", ""
    for line in text.splitlines():
        if re.match(r"^#{1,3} ", line):
            if buffer.strip():
                sections.append(f"{current_title}\n\n{buffer.strip()}")
            current_title, buffer = line.lstrip("# ").strip(), ""
        else:
            buffer += line + "\n"
    if buffer.strip():
        sections.append(f"{current_title}\n\n{buffer.strip()}")

    final = []
    for sec in sections:                     # oversized sections recurse further
        final.extend(semantic_free_split(sec, size))
    return final

def semantic_free_split(sec: str, size: int) -> list[str]:
    if len(sec) <= size:
        return [sec]
    paras, buf, out = sec.split("\n\n"), "", []
    for p in paras:
        if len(buf) + len(p) > size and buf:
            out.append(buf.strip()); buf = ""
        buf += p + "\n\n"
    if buf.strip():
        out.append(buf.strip())
    return out

Two details carry outsized value. Prepending the heading path ("Refunds > International orders") to every chunk means each embedded unit carries its own addressing context, dramatically improving matching for queries like "international refunds." And code blocks plus tables should never cross chunk boundaries; treat them as atomic units during splitting, since a table row separated from its header embeds as noise.

Atomicity deserves enforcement, not intent. A splitter that respects fences needs to detect triple-backtick spans before any other cutting logic runs and either emit them whole or attach them to the section chunk that introduces them. The same rule protects tables: find the header row, keep every row that follows within the same chunk even if the size limit strains, and accept one oversized table chunk over five useless fragments. In document-heavy corpora these two rules alone recover more retrieval quality than most embedding-model upgrades.

Picking a Strategy Under Real Constraints#

Decision logic distilled from shipping these against actual corpora:

Whatever you choose, three parameters dominate outcomes more than strategy brand: chunk size (start near 800-1200 characters for prose), overlap (ten to fifteen percent of size), and whether metadata like source path and section title ride along (they should). Change one variable at a time and measure retrieval hit rate on your golden question set, because corpus effects routinely invert blog-post conventional wisdom, including this one.

Run the tuning as a proper experiment rather than a vibes session. Fix your question set first, then sweep one parameter across three or four values, recording hit rate per document category each time. An afternoon of that produces a table your team will cite for months, and more importantly it produces the discovery of which category resists every setting, which is usually where a genuinely different strategy, not another knob, needs to enter. Keep the losing configurations in the notes too; knowing that overlap above twenty percent adds cost without recall prevents someone from re-running the same experiment next quarter.

Boundaries Are the Product. Every retrieval failure that is not a vocabulary mismatch is usually a boundary failure wearing a disguise: the right passage existed, but it was fused into a chunk about something else, or halved across two chunks that both scored poorly. Teams that treat chunking as a real engineering surface, format-aware defaults, measured thresholds, atomic tables and code, consistently hit quality targets that neighbors chase with bigger models. Spend the afternoon on your splitter before spending the budget on your embedder.