Add five awkward questions to an eval
A small retrieval test should include questions the system cannot neatly answer.
Chris Carter1 min readEvaluation
Sources enter retrieval, but evidence must be checked for relevance and freshness before it can support a user decision.
Open diagram at full sizeClean questions make retrieval demos look reassuring. Real questions can be incomplete, contradictory, stale, or impossible to answer from the available sources. Include those conditions before expanding the document collection.
A starting set of 20 questions, with five deliberately awkward cases, is a practical review exercise. It is not a statistical claim about reliability or a substitute for evaluating the risks of the actual workflow.
Put it to work
- Choose questions tied to a real user decision, using material you are permitted to evaluate.
- Include missing information, conflicting sources, stale guidance, ambiguous intent, and an unanswerable request.
- Write the expected evidence and acceptable recovery behavior for each case.
- Score answer quality, citation quality, freshness, and the next step offered when evidence is insufficient.
- Review the failures before adding more sources or expanding the workflow.
Read the full essay
This note is adapted from “RAG Is Not a Product Strategy.” The essay develops the argument, includes an interactive framework, and links to further reading.