I added a semantic search to a small blog (about 30 how-to articles, Portuguese, some in English and Spanish). It uses bge-m3 embeddings in Vectorize. The first real test run showed a problem that no threshold fixes, and a fix that is cheaper than I expected.
The problem: near neighbours
A visitor types "How to announce an event?" and gets two results:
- Events, a notice banner and holiday hours (score 0.525)
- Mark an item as sold out (score 0.481)
The first is the right answer. The second is a weak hit: it is about telling customers that something is unavailable, which sounds close to announcing something. The gap is 0.044, inside the "within 0.10 of the best" spread I use, so it is shown.
The other two failing test queries had the same shape, with articles that are close neighbours:
| Query | Best result | Second | Gap |
|---|---|---|---|
| "How to announce an event?" | Events, a notice banner and holiday hours 0.525 | Mark an item as sold out 0.481 | 0.044 |
| "avisar que estamos fechados no feriado" | "Open now" or "Closed" article 0.604 | Events, a notice banner and holiday hours 0.597 | 0.007 |
| "o prato acabou hoje" | Dish of the day 0.568 | Mark an item as sold out 0.530 | 0.038 |
Two things stand out. Nearly everything sits in a narrow band, roughly 0.47 to 0.60, even for unrelated articles. And the gaps between a right and a merely related article are a few hundredths. A minimum score and a "within X of the best" spread, both tuned for short menu entries, cannot cut between them. Tightening the spread removes the weak second result in one case and the right answer in another.
Two of the three "failures" were also my test expectations being too strict: a related article ranked first is a fair answer. Only one result was really weak.
The idea: let the article carry the questions people ask
A long article chunk and a short question are different kinds of text. A short chunk written like the question matches it much more closely. So each article may carry up to two example search phrases in its metadata, indexed as one extra short chunk (title plus the phrases) that points to the same article.
- At most 2 per article, 80 characters each, enforced by the schema. Without a cap, a process that only ever adds would never stop.
- An article without phrases is found through its text, exactly as before. The system degrades to the old behaviour.
- The extra chunk is appended last, so the other chunks keep their keys and vectors, and only the new chunk costs an embedding.
For the example above, "How to announce an event?" becomes a phrase on the Events article, so the question finds it directly instead of competing with the sold-out article.
A caveat on testing: if the phrase is the test query, the test passes trivially. So I added new queries worded differently from the phrases, which show whether the idea generalises.
A phrase can pull its article into the wrong queries
After the phrases were indexed, one case that had passed started failing: "esconder pratos com leite" (hide dishes with milk) now returned the sold-out article first. Its new phrase, "o prato acabou hoje" (the dish ran out today), shares words with the query ("prato", hiding something), and the short phrase chunk scored 0.534 against 0.527 for the diet-filter article that was the right answer.
The fix was the same move the loop makes: add the query as a phrase on the article that should have been found. The lesson is that every added phrase can change other queries, so every change has to be checked against the whole test set, not only against the query it was meant to fix. After that change the whole set was green again.
The feedback loop
Writing phrases by hand does not scale, so I made a weekly loop:
- Take real queries and run them through the actual search.
- Let an LLM judge the query against the returned titles and snippets: is there a result that answers it, and is a wrong one ranked above it?
- When the right article is missing or ranked low, propose the query as a phrase on that article. Never on the wrong articles that came back.
- Check before proposing: add the phrase to a copy, re-embed, re-run every test case. Keep it only if the query now finds the right article and nothing that passed before breaks.
- Open a pull request. I review and approve. When an article already has two phrases, the loop replaces one.
The queries are the awkward part. Storing visitors' searches needs a privacy decision: text only, no IP or cookie, a short retention (7 days), redaction of anything that looks like a number or email, and a visible opt-out next to the search box. Static questions in a dry run of a workflow cover the gap while traffic is low.
Let's see in the coming months if this will improve the blog search.