Resource / Test template
AI Evaluation Test-Set Template
A reusable table structure for expected outputs, actual outputs, error classes, reviewer decisions and release criteria.
A test set turns “the demo looks good” into a repeatable way to compare AI-assisted behavior against agreed criteria. Use one row per test case, preserve the evaluation version, and record both the result and failure type so teams can find patterns instead of relying on anecdotes.
Suggested columns
| Column | Purpose |
|---|---|
| Case ID | Stable identifier for repeatable comparison |
| Input class | Typical, edge, ambiguous, invalid or escalation case |
| Input reference | Sanitized test data or a secure controlled reference |
| Expected result | Accepted answer or range defined before the run |
| Actual result | Model/application output and schema validation |
| Error class and severity | False positive, false negative, omission, unsafe output or format error |
| Pass/fail and reviewer | Outcome against the rubric, with reviewer and date |
| Model/instruction version | Allows comparisons across changes |
Use it consistently
Define the rubric before running tests. Keep cases representative of intended users and include difficult examples. Avoid collecting more personal data than needed. Label synthetic cases honestly and do not use a test set that contains only examples used to tune the prompt. Record which cases are held out for independent evaluation.
Summarize results by error type, not only overall accuracy. Include number of cases, review burden, cost and latency where relevant. Decide in advance what result blocks release, requires additional human review or triggers rollback. Document who owns the evaluation and when it must be repeated.
Use this template with the AI Workflow Evaluation Notes and the AI triage workflow.
Recommended test-set worksheet
Use a table or spreadsheet with one row per case and the following operational fields: case ID, input class, sanitized input or secure reference, expected result, actual result, schema-valid flag, error class, severity, pass/fail, reviewer, model or prompt version, run date, latency, estimated cost and notes. Keep the rubric in a separate versioned sheet so a change in the definition of “pass” is visible rather than silently changing historical results.
Build coverage before optimizing prompts
Include ordinary cases, boundary values, missing fields, contradictory details, ambiguous phrasing, unsupported requests and examples that should be refused or escalated. For a classification task, inspect each category rather than relying only on overall accuracy: a high aggregate score can hide an unacceptable false-negative rate for a rare but consequential category. Include repeated runs where variability matters and test malformed output, timeout, empty response and provider-unavailable conditions.
Separate examples used to develop instructions from the holdout cases used to judge readiness. If the same small set is reused for every iteration, the result may reflect prompt tuning rather than performance on unseen inputs. Synthetic test cases are useful, but label them as synthetic and review whether they realistically represent how people will phrase requests.
Release gate and review cadence
Define release thresholds before reviewing the final result. Examples include a minimum pass rate for low-risk cases, zero tolerance for a specific safety failure, a maximum manual-review workload, or a requirement that every invalid output is safely rejected. Choose thresholds appropriate to the actual task; do not copy a benchmark from an unrelated system.
Document who can approve release, pause the workflow and request a new evaluation. Repeat the test set after changes to the model, prompt, schema, upstream data, business rules or user interface. Track both quality and operating cost, because a technically correct workflow may still be impractical if latency, review time or usage charges exceed the team’s constraints. Store the smallest amount of raw input necessary and restrict access to any retained evaluation data.
Need a tailored version?
We can shape a resource, workflow or content system around your team and process.
Get in touch