Chunking is product design, not a utility call
Default 512-token splits destroy tables, break FAQ pairs, and separate headings from body text. Users ask "What is the refund window for annual plans?", if the chunking split policy header from table rows, retrieval returns garbage and the LLM fills gaps with fiction.
Start from document types: contracts need clause-aware splits, menus need item-level rows, support macros need one article per chunk. Inspect failures visually in a chunk viewer before tuning embeddings.
Tactics that move retrieval metrics
Use structure-aware parsers for PDFs (headings, lists, tables). Add overlap only where continuity matters, long policies, not isolated Q&A. Attach metadata: source file, page, section, effective date, outlet ID for multi-location restaurants.
Hybrid search (dense + keyword) rescues exact SKU codes, drug names, and legal citations that pure embeddings miss. Re-rank top-k with a cross-encoder or lightweight scorer before passing context to the LLM.
Evaluate chunk changes like code changes
When you re-chunk, rerun the golden question set. Track MRR@k and human graded faithfulness, not just latency. Version your ingestion pipeline so you can roll back a bad splitter without re-embedding everything blindly.
Sabrixa tunes chunking per vertical during consulting engagements and applies sensible defaults in the WhatsApp product so SMB uploads work without a data engineering team.