Start with the task, not the model
An AI model is useful only in relation to a task. Before testing one, write down what it will receive, what it should produce and who will use the result. “Summarize customer messages for a support agent” is easier to evaluate than “be good at writing.” Define the boundaries too: must the model use only supplied documents, follow a fixed format or decline requests it cannot answer?
Next, decide what success looks like in the workflow. A draft that saves editing time may be valuable even if it is not publishable as-is. By contrast, an answer used without review may need a much stricter standard. Note which mistakes are tolerable, which require correction and which should stop the workflow. This gives your test results a practical meaning.
Build a test set that resembles real work
Collect examples of the inputs the model would actually encounter. Include ordinary cases, short and long inputs, ambiguous requests, missing information and cases where the right response is to say that the evidence is insufficient. If the workflow involves several types of users or documents, represent each type. A test set made only of polished examples will tell you little about performance on messy ones.
For each example, record the expected outcome before looking at model responses. That might be a correct label, required facts to include, facts that must not be invented or criteria a reviewer can apply. Remove sensitive details unless you have an appropriate reason and permission to use them. Keep a separate set of examples for final comparison; repeatedly adjusting prompts against the same cases can make them look better on the test than they are in use.
Start with a manageable collection you can inspect closely, then expand it as you discover recurring failures. Label important groups, such as document type or input length, so you can see where performance changes. The aim is not to make every imaginable question appear in one test. It is to cover the conditions that will shape your decision.
Choose metrics that match the outcome
For tasks with a clear correct answer, such as categorizing messages, count correct and incorrect outputs. Accuracy is a useful starting point, but it can conceal an important failure when one category is common. Look separately at false positives and false negatives for decisions where those errors have different consequences. For information extraction, measure whether the model finds the required items and whether the items it returns are correct.
Open-ended answers need a review rubric rather than one vague score. A reviewer might check factual support from the supplied material, completeness, instruction-following and readability as separate dimensions. Define what earns a passing result for each, and record concrete failure types. If two reviewers often disagree, clarify the rubric before treating their ratings as a precise measure.
Also measure the resources needed to get a usable result: response time, usage cost where available and the amount of human correction. A model with a stronger first draft may not be the better choice if it takes much more review to make its occasional errors safe to use. Keep quality and operational measures visible side by side rather than hiding them in a single score.
Run comparisons under consistent conditions
Give each model the same task instructions, input examples and access to supporting material, except where a difference is part of what you intend to compare. Record model settings and the exact prompt or workflow used. If your system retrieves documents or calls tools, test the whole workflow as well as the model response; retrieval mistakes can otherwise be mistaken for model mistakes.
Some outputs vary between runs. Repeat selected tests, especially those where consistency matters, and note whether conclusions change. Review results by test group instead of relying only on an overall average. A model may do well on short, straightforward inputs while failing on long documents or requests with missing evidence. Save examples of both successes and failures so the numbers remain interpretable.
Use failures to make a decision
Sort failures into causes you can act on: unclear instructions, missing context, unsupported assertions, wrong format or a task the model does not handle reliably. Try a limited change, such as improving the prompt or adding a verification step, then test it on examples beyond those that prompted the change. If an error has serious consequences, consider requiring human approval or keeping that task outside the AI workflow.
Choose a model against the requirements you set at the start, not against its most impressive response. Document what you tested, where it performed adequately and where you still need safeguards. Recheck it when the prompt, model, source material or workflow changes. Evaluation is not a one-time certificate: it is a way to keep a useful system aligned with the work it actually does.
