
Photo by Related names: Blaw-Knox Construction Company Kaiser Engineers Hummel Hummel and Jones Fluor Construction Arrington Construction United States Department of Energy Christianson, Justine, transmitter, Public domain, via Wikimedia Commons
Test an AI tool by measuring it, not demoing it: assemble a small set of real tasks from your own work with known-correct answers, run the tool on every item several times, score the outputs with a mix of automatic checks and human review against written criteria, set a pass threshold for each category of failure, and only then let the tool touch live work. Generative models are non-deterministic — the same input can produce different outputs — so a few manual runs prove almost nothing. OpenAI's guidance is blunt: traditional software testing methods are insufficient for AI architectures, and "evals" (structured tests for measuring a model's performance) are how you test despite that variability (OpenAI: Evaluation best practices).
Disclosure: this article may contain affiliate links.
Why a Demo and a Vendor Benchmark Are Not Evidence
Vibe testing. You tweak a prompt, run it a few times, and the output feels better — but nobody can say whether it is. Google's Stax documentation describes "vibe testing" as a loop that feels more like art than engineering, and notes that general benchmarks don't help you evaluate your stack on your own data and criteria. OpenAI draws a matching line between industry benchmarks, generic scores like ROUGE or BERTScore, and the tests you build yourself.
Benchmark mismatch. A tool that tops a public leaderboard can still be wrong about your invoice formats, your policy language, or your customers' phrasing. Benchmarks test models in isolation; your problem is a workflow.
No baseline. Without a written baseline you can't tell a real regression from noise. Anthropic describes what happens after early prototyping: users report the tool "feels worse" after a change, and the team is flying blind, debugging reactively and hoping nothing else broke (Anthropic: Demystifying evals for AI agents).
Step 1: Write Down What "Accurate" Means Before You Test
Anthropic argues evals matter most early, because they force the team to specify what success means — two people reading the same spec can read an edge case differently, and an eval suite settles it. Claude's developer documentation recommends criteria that are specific, measurable, achievable, and relevant, and demonstrates turning a vague goal ("classify sentiment well") into a testable target such as an F1 score of at least 0.85 on a held-out set, stated as an improvement over a baseline (Claude Platform docs). You need your own numbers, across the dimensions your work depends on:
- Task fidelity — is the answer factually correct (right label, right figure, right clause, code that runs)?
- Consistency — do two people asking near-identical questions get compatible answers?
- Instruction following — does it respect scope, format, and "don't answer this" rules?
- Sourcing behavior — does it attribute claims, or invent them?
- Failure severity, latency, and cost — how many errors are harmful rather than annoying, and is the tool fast and cheap enough to be used?
Then give each failure category a threshold and a consequence. A tool drafting internal summaries can tolerate more misses than one drafting client-facing contract language, and writing thresholds down before testing stops a good demo from moving your goalposts.
Step 2: Build a Test Set From Your Own Work
Your dataset decides the outcome. OpenAI and Google describe the same recipe: mix production data, historical logs, human-curated examples from domain experts, and synthetic examples, covering the happy path, edge cases, and adversarial cases. OpenAI calls "biased design" — datasets that don't reproduce production traffic — an anti-pattern.
A practical minimum:
- 30–50 real tasks from recent work, anonymized as needed.
- 10–15 known-hard cases: sarcasm, typos, contradictory instructions, absurdly long inputs, missing data, requests that should be refused.
- 5–10 adversarial probes: prompts that test whether the tool invents details, ignores a policy, or reveals content it shouldn't.
- A held-out slice you don't touch while tuning prompts, so the final score is honest.
You don't need hundreds of prompts. Stax's guidance is that "an eval set with just ten high-quality prompts is infinitely more valuable than relying on vibe testing alone," and the set should grow over time; Claude's docs push the same way, prioritizing volume over perfection since more items with slightly weaker grading beats a handful of hand-graded gold examples. Two rules keep the set honest: correct answers come from a human who does the work, not from the model; and log everything, because OpenAI recommends mining logs for eval cases and real usage surfaces failures you wouldn't have invented.
Step 3: Run Multiple Trials, Not One Pass
Because outputs vary run to run, one pass measures luck as much as accuracy. Anthropic defines each attempt at a task as a trial and recommends multiple trials per task — especially for agents, which call tools across many turns and can compound mistakes.
For each item, record:
- Pass/fail against your written criteria, and which criterion failed.
- How many of 3–5 runs passed — 2 of 5 on a routine task isn't dependable.
- Outcome, not claims. Anthropic distinguishes the transcript (what the tool said it did) from the outcome (the final state: did the record get written, the file change, the ticket close?).
- Time and cost per task, so comparison isn't accuracy-only.
For agents that browse, edit files, or call internal systems, test tool selection too: Chrome's WebMCP docs say to confirm, before release, that agents understand when to call a tool, how to execute it, and what answers are acceptable — and to keep deterministic tests for interactions that don't go through the model (Chrome for Developers).
Step 4: Grade With Three Graders, and Calibrate the Automated Ones
OpenAI, Anthropic, and Google all converge on three grader types.
- Code-based checks — exact match, regex, valid JSON, required fields, tool-call verification, static analysis, unit tests. Fast, cheap, objective, reproducible; brittle for valid-but-different phrasing and useless for tone.
- Model-based graders (LLM-as-judge) — rubric scoring, natural-language assertions, pairwise comparison. Flexible and scalable, but non-deterministic and only as good as your rubric; Anthropic treats calibration against human graders as a requirement.
- Human graders — subject-matter review, spot-check sampling, blinded comparison. Best quality, slow, expensive, and the standard used to calibrate the other two.
Calibration is the step teams skip. Google says to rate a small sample yourself and iterate on the autorater prompt until you agree with its scores. OpenAI's evaluation-flywheel cookbook flags the trap: test sets skew toward passes, so a judge that always says "looks fine" can score 95% "accurate" while never catching a failure. It recommends measuring the judge's true positive and true negative rates against a human-rated gold standard, split into train, validation, and held-out sets (OpenAI Cookbook). Anthropic's human-grading advice fits a two-week pilot: refine the scorecard over several review rounds, show reviewers examples of bad, middling, and excellent output rather than describing the scale, add a pass/fail threshold next to any numeric score, and use consensus votes when reviewers disagree.
Step 5: Compare Candidates on the Same Set
With a fixed dataset and rubric, comparison becomes a head-to-head. Run ChatGPT, Claude, and Gemini — or a general assistant against a specialist vendor such as a transcription or legal-drafting product — on the same items, the same prompts where the product allows, and the same graders. Declare a winner per criterion, not overall: one tool may lead on factual precision while another wins on format adherence or refusal behavior.
Keep two kinds of tests separate. Anthropic distinguishes capability evals, which target tasks the system struggles with and should start with a low pass rate, from regression evals, which confirm it still handles everything it used to and should sit near a 100% pass rate. A capability test that consistently passes "graduates" into the regression suite that catches drift. OpenAI's documentation also notes its Evals platform is being deprecated — read-only for existing users on October 31, 2026 and shutdown scheduled for November 30, 2026 — so keep your eval set and grading scripts in your own repository or a neutral harness rather than a vendor console.
Step 6: Add Safety, Data, and Governance Checks
Accuracy alone isn't a green light. Google's Responsible Generative AI Toolkit describes four evaluation types across a model's lifecycle — development evaluations, assurance evaluations run by a group outside the build team, red teaming, and external evaluations by independent experts — and recommends testing on your own safety dataset, covering both explicit and implicit adversarial queries, not just public benchmarks (Google Responsible Generative AI Toolkit).
Translate that into four questions before broad access: what happens when someone tries to make the tool say something prohibited; where the input data goes and whether training use can be turned off; how it handles personal or sensitive information; and who signs off, and when access gets revoked. Put the answers in writing — see our guide to writing an AI use policy for a small team.
A Two-Week Test Plan You Can Actually Run
- Days 1–2: Pick one workflow, not "AI in general." Write success criteria and thresholds with the people who do the work.
- Days 3–5: Collect 40–75 tasks with human-verified answers, including edge and adversarial cases, and split off a held-out set.
- Days 6–9: Run 2–3 candidates, 3 trials per item. Log transcripts, verify outcomes, grade objectively, then have two reviewers score a sample independently.
- Day 10: Calibrate any LLM judge against the human-scored sample; fix the rubric, not the model, when they disagree.
- Days 11–12: Score the held-out set once, compare by criterion, and decide: adopt, adopt with mandatory human review, or reject.
- Days 13–14: Run a small pilot with a defined group and a feedback channel, and keep the test set as your regression suite. When the vendor ships a new model version, re-run it before your team does. Our AI pilot guide covers scoping that phase.
Keep evaluating after rollout. OpenAI frames evaluation as continuous — run evals on every change, watch for new sources of nondeterminism, grow the set over time — and Anthropic notes that teams with an eval suite assess a new model in days instead of weeks.
FAQ
How many test cases do I need before I trust a tool? Fewer than you fear. Google says ten high-quality prompts beat vibe testing outright; a realistic pilot is 40–75 items with multiple trials each. Coverage matters more than raw count.
Can I let an AI grade the AI's answers? Yes, with calibration. Grader quality depends on your rubric, and an imbalanced test set lets a judge that never flags a failure look 95% accurate. Keep a human-rated sample and check whether the judge finds failures as well as it accepts passes.
The tool changed after a vendor update. What now? Expect it. Keep a regression suite of tasks the tool used to pass, re-run it on the new version, and treat a drop as a blocking signal rather than a debate.
What if the task has no single correct answer? Use rubrics, Likert-style scales, and pairwise comparison. OpenAI notes that models are better at discriminating between options than at open-ended judgment, so side-by-side comparison beats asking for an absolute score — with a human spot-check on anything subjective.
Does a public benchmark tell me anything useful? Only as background. Benchmarks test models in isolation and don't reflect your data, prompts, tools, or failure tolerance. For day-to-day verification, see our fact-checking workflow for AI answers.
Sources
- OpenAI — Evaluation best practices
- Anthropic — Demystifying evals for AI agents
- Google — Stax evaluation best practices
- Claude Platform — Define success criteria and build evaluations
- OpenAI Cookbook — Building resilient prompts using an evaluation flywheel
- Google — Responsible Generative AI Toolkit: evaluate for safety
- Chrome for Developers — Test WebMCP with evals