Chunking Strategy Matters More Than Your Embedding Model

I spent two weeks A/B testing embedding models and got a 2% recall improvement. Then I changed how I chunk and got 19%.

What moved the needle:

1. Chunk on structure, not character count. Splitting a markdown doc at headings beats a 512-character window, because a window cuts mid-sentence and the fragment embeds as noise.

1. Prepend the breadcrumb to every chunk. "Billing > Refunds > Partial refunds\n\n". Retrieval on a chunk that says "this must be approved by a manager" is hopeless without knowing what "this" is.

1. Overlap only across the sentence boundary, not a fixed token count. Fixed overlap duplicates whole paragraphs and skews your top-k towards long documents.

What did not matter: the embedding model, once past a reasonable baseline. Also chunk size, within a wide range, once the splits were structural.

Measure recall@10 before and after each change on a fixed question set. If you change two things at once you will learn nothing.