RAG Pipeline Design: Chunking, Embeddings, and Retrieval Trade-offs

The problem RAG actually solves

Retrieval-augmented generation exists because an LLM's context window is finite and its training data is frozen at some cutoff. RAG lets you ground a model's answers in a specific, current corpus by retrieving relevant chunks of text and feeding them into the prompt at query time, instead of hoping the model already "knows" the answer. Simple in concept; the engineering is entirely in the details.

I'm working through those details in a personal project, ragsume, using this blog's own posts as part of the source corpus. It's a convenient, honest test case since I control the content and know what a "correct" retrieval should look like.

Chunking: the decision that quietly determines everything downstream

Chunking is where I spent the most time, because a bad chunking strategy can't be fixed by a better embedding model later.

• Fixed-size chunks (e.g. by token count) are the simplest to implement, but they cut across sentence and section boundaries indifferently: you can split a heading from the paragraph it introduces.
• Semantic/structural chunking, splitting on markdown headings (##, ###) or paragraph boundaries, keeps related ideas together, which is a big part of why this site's own blog posts are written with clear heading structure per topic. It's a deliberate choice, not an accident: clean headings make the corpus easier to chunk correctly.
• Overlap between chunks helps avoid losing context at a boundary, at the cost of some retrieval redundancy.

There's no universally "correct" chunk size. It depends on the source material's structure and how self-contained each unit of meaning is. For prose-heavy technical writing like this blog, chunking by section (each ## heading with its following content) has worked better for me than a fixed token window.

Embeddings: capturing meaning, not keywords

An embedding model converts text into a vector such that semantically similar text ends up close together in vector space. The trade-off I actually had to reason about wasn't "which embedding model is best" in the abstract. It was matching embedding model choice to the corpus: general-purpose embeddings are a reasonable default for a small personal-blog corpus like mine, where a highly domain-specific model would be overkill.

Retrieval: similarity search is a starting point, not the whole answer

Pure vector similarity search (nearest neighbors in embedding space) is the baseline, but it has a known failure mode: it retrieves text that's semantically similar to the query, which isn't always the same as relevant. Two things worth knowing about before you assume vector search alone is enough:

• Hybrid search: combining vector similarity with traditional keyword/lexical search (e.g. BM25) to catch cases where exact terms matter and pure semantic similarity misses them.
• Re-ranking: running a second, more expensive relevance pass over the top-N retrieved candidates before they go into the prompt, rather than trusting the first-pass ranking blindly.

I haven't implemented full hybrid search and re-ranking in ragsume yet. That's explicitly on the list of what's next, not something I'm claiming is done.

What I'd tell my past front-end self

The instinct from front-end work to treat a pipeline as a sequence of composable, independently testable stages transfers directly here: chunking, embedding, retrieval and generation are each testable in isolation, and most of the debugging value comes from being able to inspect what got retrieved before blaming the LLM's final answer.

Where it lives

ragsume is deployed at ragsume.vercel.app; the code is public on github.com/edefice/ragsume. Hybrid search and re-ranking are still on the roadmap, not shipped, so treat the live version as a snapshot of a project in progress rather than a finished product.