Self-Generating Chatbot QA: An LLM-as-Judge Evaluation Pipeline
Problem
Every new document-grounded chatbot needed manual QA: someone had to read the uploaded documents, invent test questions, and eyeball the answers. That process did not scale across tenants, and it systematically missed the most dangerous case — questions the documents cannot answer, where the bot should refuse instead of hallucinating.
Trial
I made question generation part of chatbot creation itself: on upload, an LLM generates test questions from each document with a deliberate difficulty mix — roughly 60% fully answerable, 30% partially answerable, and 10% unanswerable 'gotcha' questions — with no domain-specific hardcoding. The in-house model answers each question, a frontier LLM auto-scores the answers as a first pass, and a human reviewer makes the final call. The pipeline is backed by real database schema with validation run and item tables, and results surface in paginated dashboards with passing tests.
Result
Chatbot QA went from an ad-hoc manual chore to an automated run that starts the moment documents are uploaded, with the difficulty mix guaranteeing that hallucination defense is tested on every single bot. The judge-then-human flow kept review effort low while catching cases where the auto-scorer was too generous. The run and item tables made quality regressions comparable across time instead of anecdotal.
Lessons
The unanswerable questions turned out to be the highest-value 10% of the suite, because a RAG bot's failure mode is confident fabrication, not wrong retrieval. LLM-as-judge works well as a filter but not as a verdict: it compresses the human's job from grading everything to auditing the borderline cases.
Next Action
Next is trend reporting across validation runs, so a document update that silently degrades answer quality triggers an alert instead of waiting for the next manual audit.