Free · 10 questions · no sign-up · nothing leaves your browser

Eval Readiness Check: is your RAG feature gated, partial, or vibes-only?

Ten questions about how you find out that your RAG or LLM feature answered wrong — and what stops it from happening again. Each answer is worth 0, 1 or 2 points. You get a score out of 20, one of three buckets, and three concrete next steps.

Privacy: there is no backend and no email field. Your answers are kept only in this browser’s localStorage so you can come back to them; “Start over” clears them. No cookies, no analytics, no external requests apart from the fonts.

0–7 · Vibes-only

Changes are checked by hand with a few queries. Every fix is a coin flip.

8–14 · Partial

Some traces, some tests, but nothing blocks a bad release.

15–20 · Gated

A written gate blocks releases. The question is whether it can be trusted.

Question 1Trace capture

For a production request, what do you keep?

0The final answer only, or nothing. 1The query and the answer, but not which chunks were retrieved. 2Query, retrieved chunks with document ids and scores, answer, citations and the abstain flag.

Question 2Trace lookup

A user says “it answered wrong”. How fast can you see the exact trace?

0I can’t. I would have to try to reproduce it. 1Eventually, by searching logs by hand. 2In minutes, by request id or trace id.

Question 3Failure taxonomy

Do you have a written list of failure classes with examples from your own traces?

0No. 1Informal notes, or a list in someone’s head. 2A written taxonomy with definitions and example traces, used when triaging.

Question 4Retrieval vs generation

For your last ten wrong answers: was the right document in the context or not?

0I don’t know. 1I have a feeling about it, not labels. 2Each one is labelled: retrieval miss, generation error, policy or input problem.

Question 5Golden set

Do you have a fixed set of queries with expected facts and expected document ids?

0No. 1A handful of queries I re-run by hand. 2Thirty or more cases with expected facts and acceptable document ids, versioned in the repo.

Question 6Retrieval metrics

Do you measure retrieval on its own — recall@k and MRR against expected document ids?

0No. 1I did it once, manually. 2On every run, computed from the golden set.

Question 7Code checks

Do automated checks run on outputs — citations exist, schema is valid, refusal rules, length?

0None. 1Some scripts, run by hand when I remember. 2In CI, on every change.

Question 8Judge calibration

If an LLM judge scores your outputs, has it been compared with human labels?

0No judge, or it has never been compared. 1I spot-checked a few of its verdicts. 2Measured TPR and TNR on a held-out labelled set; the pass rate is corrected for them.

Question 9Baseline diff

Before changing a prompt, model, chunking or index, do you compare against a stored baseline run?

0No. I try a few queries and see. 1Sometimes, informally. 2Always. I diff which cases newly fail against the baseline.

Question 10Release gate

Is there a written threshold that blocks a release?

0No. 1An informal “looks fine” review before shipping. 2Written thresholds — retrieval, judge pass rate, must-pass cases — enforced automatically.

0 of 10 answered

Your result

0 / 20

—

Three next steps

See the kit — pre-order $29 Copied to clipboard