The RAG System That Confidently Lies
A support bot is giving wrong answers with total confidence. Find out why — the answer is not in the prompt. It takes 75 minutes, the full rubric is published below before you start, and AI tools are allowed.
The brief
The situation. An internal support assistant answers employee questions from a 2,000-document policy corpus. It has been live for six weeks. Complaints are increasing: the answers *sound* right and are frequently wrong.
What you have. Twelve logged failures. For each: the question asked, the chunks that were retrieved, the answer given, and the correct answer from the source document.
Observations already made by the team. The wrong answers cite real policy text — just the wrong policy. Failures cluster around questions where several policies use similar wording (parental leave vs. carer's leave; contractor expenses vs. employee expenses). Three people have already rewritten the system prompt with no measurable improvement.
Configuration. Documents split at 500 characters with no overlap. Single dense embedding index. Top-5 retrieval, no re-ranking. All five chunks concatenated into the prompt with no scores or source labels.
Your task. Diagnose the failure. State clearly whether this is a retrieval failure or a generation failure, and show what evidence in the logs supports your conclusion. Then propose a fix, and — most importantly — describe the measurement that would prove your fix worked before shipping it.
What you hand in
A written diagnosis: where the failure occurs, the evidence for it, your proposed fix in priority order, and the measurement plan that would validate it.
Any tools, including AI assistance. Note that an assistant asked to 'fix RAG' will suggest a generic checklist — the rubric rewards evidence from the supplied logs, which is yours to read.
How is this challenge scored?
Every criterion, with its weight and the skill it produces evidence for. Nothing else is measured — not writing quality, not grammar, not length. You see this before you start, and there are no hidden criteria.
Identifies that retrieval is returning wrong-but-plausible chunks, and cites specific log evidence. Concluding 'the prompt needs work' after three failed prompt rewrites scores 0.
Connects the failure to a concrete cause — 500-char chunks with no overlap severing policy scope from policy text, dense-only retrieval collapsing near-identical wording, or both.
Proposes measuring retrieval quality on its own (recall against known-correct source documents) rather than only judging final answers. This is the Level 3 boundary for this skill.
Orders interventions sensibly and justifies the order. Proposing every possible improvement at once, unranked, caps this criterion at 2.
A concrete before/after measurement on a fixed query set, with a stated threshold for shipping.
Which skills does this prove?
Measures retrieval quality separately from answer quality, and uses hybrid or re-ranked retrieval where the data demands it.
Designs evals that target real failure modes, chooses metrics that resist gaming, and knows the limits of what the eval measures.
Diagnoses failures where the error is misleading, tests the cheapest hypothesis first, and verifies the root cause before fixing.
Questions people ask
- Is the bug findable in 75 minutes?
- Yes — the logs contain enough to reach a confident diagnosis in under an hour. Most of the remaining time goes on the measurement plan, which is where the scoring weight sits.
- What if I diagnose it differently to your expected answer?
- A well-evidenced alternative scores fully. The rubric rewards evidence-backed reasoning, not matching a key.
- Do I need to have built a RAG system before?
- It helps, but the diagnosis is a debugging exercise. Several strong scorers have had no production RAG experience and reasoned it out from the logs.