Community guidelines
Be specific and constructive. No vendor spam — promoting your own product belongs in a listing. Anyone can read; posting needs a free account.
We've a working assistant and every demo produces a different opinion about quality. I want a repeatable evaluation, but we don't have people to label thousands of questions. What is the smallest useful setup?
Start with fifty real questions, not a synthetic benchmark. For each one, mark the source passage and the facts the answer must contain. That's enough to catch regressions in retrieval, citations and refusal behavior.
We tried Ragas for the LLM judge pass, and it can scale the first pass, but calibrate it against human ratings. Ours was too generous to fluent answers. We score factual support and completeness separately from writing quality
edit: do you freeze the model while testing retrieval changes, or evaluate the whole pipeline every time?
Both. Component tests tell you whether retrieval improved. End to end tests tell you whether the user outcome improved. Keep model, prompt and index versions with the result. Otherwise a better score next month may just mean a different judge was in a good mood.