A misplaced threshold silently killed the BM25 arm of my hybrid search. How I found it, built an eval harness, and swapped embedding models in production.

Earlier this year I wrote about the three rewrites that took a customer support platform's retrieval pipeline down to one contextualized embedding model and a single content_chunks table in Postgres: hybrid search with vector + BM25 + RRF fusion, and a reranker on top. That post ended with a principle: before adding complexity, check if a simpler architecture can do the same job.

Production spent the next five months teaching me the second half of that principle. This is the follow-up.

The confession: my hybrid search wasn't hybrid#

The retrieval query fused two arms with Reciprocal Rank Fusion, a pgvector cosine search and a Postgres full-text (BM25 style) search:

Text
1score = 1.0/(60 + vector_rank) + 1.0/(60 + text_rank)

There was also a sensible-looking guard: a minimum cosine similarity of 0.3, to keep junk out.

The problem: the similarity filter ran after fusion. Chunks that matched only through the full-text arm have no vector similarity score at all, so the filter dropped every single one of them. The BM25 arm had been effectively dead for months.

No errors. No warnings. The query ran, results came back, answers looked plausible. Exact-term queries (product names, codes, phrases in the language the embeddings handle less well) were quietly weaker, and it's very easy to blame that on "embeddings being embeddings."

SQL
1-- before: the floor ran on the fused results.
2-- BM25-only rows have no vector similarity, so all of them were dropped
3WHERE similarity >= 0.3
4
5-- after: the floor lives inside the vector arm only
6vector_arm AS (
7  SELECT ...
8  WHERE 1 - (chunk_embedding <=> $query_embedding) >= 0.3
9)
Before and after of the hybrid search query: the broken version applies the similarity filter after RRF fusion, the fixed version applies it inside the vector arm only

The fix was one line: apply the similarity floor to the vector arm's subquery only, and let BM25-only matches survive fusion. It now has regression tests that assert a keyword-only match actually comes back.

The fix is not the story. The story is that nothing in my stack could have told me. Retrieval fails silently. There is no exception for "your recall just dropped on keyword queries." If you can't measure retrieval quality, you don't know what your pipeline is doing. You only know what you intended it to do.

A glowing safety net catching falling shapes, illustrating evals catching silent retrieval failures

The safety net: evals as infrastructure#

So the next thing I built wasn't a feature. It was the measurement layer:

  • Golden Q&A datasets per tenant. Curated questions grounded in the actual knowledge base, including deliberate decline traps: questions the knowledge base cannot answer, where the correct behavior is to say so.
  • A CLI runner that hits the live endpoint. Real auth, real streaming responses, not a mocked pipeline. If the answer path has a bug, the eval sees it.
  • An LLM judge scoring four dimensions: correctness, groundedness, language match, and decline behavior.
  • Everything tracked in Langfuse as dataset runs with scores, fingerprinted with the app version and a hash of tenant settings, so any before and after comparison is apples to apples.
The eval loop: golden dataset, CLI runner, live endpoint, LLM judge, Langfuse dataset run, compared on every change

Alongside this, the answer path got a proper decline: retrieved candidates are gated on the raw rerank relevance score, and if nothing clears the floor the assistant declines instead of confidently guessing from weak chunks. The decline traps in the golden dataset keep that behavior honest, permanently.

One more change in the same pass, because these are production conversations: PII redaction before embedding. Phone numbers, emails, and the customer's own name are stripped from conversation text before it is chunked and vectorized. Customer PII never enters the embedding space.

Making the model swappable#

The first post flagged one tradeoff of contextualized embeddings: all chunks of a document must be embedded together in one call, so updates are delete-and-replace rather than incremental. That tradeoff grew into full model migration infrastructure:

  1. Per-chunk model tracking. Cosine similarity between vectors from different embedding models is meaningless. Every chunk carries an embedding_model column. The vector arm filters on the active model while the BM25 arm stays unfiltered. A live corpus can sit mid-migration indefinitely without ever mixing incomparable vectors.
  2. Embed-first transactional re-ingestion. New chunks are embedded before the old ones are swapped out, inside a transaction. An embedding API failure now keeps the old chunks searchable instead of leaving a hole in the knowledge base. This also collapsed three duplicated delete, embed, insert code paths into one service.
  3. IVFFlat to HNSW. The corpus churns constantly. Chunks are deleted and re-created whenever a source document changes or a support conversation re-resolves. IVFFlat's fixed cluster lists degrade under churn. HNSW doesn't care.
  4. Change detection in sync jobs. The website sync used to re-embed the entire site every run. Now it skips content that is unchanged and already embedded under the active model.

The payoff: a boring upgrade#

Last week the pipeline moved from voyage-context-3 to voyage-context-4 in production. The upgrade was: change one environment variable, run a background re-embed job, compare eval runs.

  • Eval scores held.
  • Embedding cost dropped another 33%, from $0.18 to $0.12 per million tokens.
  • Live today: roughly 9,500 chunks, including 3,373 resolved support conversations, one table, one HNSW index, one query.

That's the whole point of the infrastructure. The first model migration (from an LLM summarization pipeline to contextualized embeddings) was a three-rewrite saga. This one was a config change, and the next one will be too.

The updated principle#

The original conclusion still holds: check whether a simpler architecture can do the job before reaching for a complex one. One embedding model and one Postgres table continue to outperform the zoo of components they replaced.

But five months of production forces an update: simplicity you can't measure is just hope. My nice simple hybrid search silently wasn't hybrid, and simplicity didn't save me. A regression test and an eval harness did.

Simple architecture wins. Simple architecture with evals is the only kind you can change with confidence.
Artificial IntelligenceRAGLLMs