Start with the whole process, not just the model
An AI-assisted process includes more than a prompt and its answer. Imagine a team using AI to summarize applications before a person decides whom to interview. The outcome depends on the application form, the information sent to the AI, the summary it produces, the reviewer’s instructions and the final decision. Testing only the summary may miss a pattern that appears when people act on it.
Draw a simple map of the workflow: who enters it, what information is collected, where AI is used, who can change the result and what happens next. Then choose a specific outcome to examine, such as whether an applicant advances to review or how often a request is flagged for manual attention. Define the unit you will count—a person, application or request—before collecting results. This prevents repeated submissions from quietly being treated as separate people.
Decide which comparisons are meaningful and appropriate
Identify demographic groups that might experience different outcomes in this particular process. Relevant comparisons depend on the setting; a language-support workflow may raise different questions from an application-screening workflow. If you have a legitimate reason to collect demographic information, handle it carefully: explain its purpose, limit access, retain only what is needed and avoid asking an AI system to guess a person’s identity from a name, photo or writing style. Such guesses can be wrong and may create additional privacy risks.
Compare groups at the same stage of the workflow, using a consistent definition of the outcome. A simple starting point is the share of cases in each group that receive a particular result. Also inspect errors where you can establish an appropriate reference point: for example, qualified applications that were screened out, or routine requests that were escalated. A difference in overall outcomes and a difference in error rates answer different questions; neither explains its own cause.
Test real cases and controlled examples
Review a set of past cases if you can do so with appropriate permissions and privacy safeguards. Record the AI output, the human action and the final outcome separately. Look for patterns by group, but also by factors that may affect the result, such as application type or request complexity. Small groups or uneven case mixes can make comparisons unstable. Report the number of cases behind each result, and avoid declaring a process fair—or unfair—from a handful of examples.
Controlled tests can help isolate a possible mechanism. Create pairs of otherwise similar, fictional cases and change one detail associated with demographic identity, such as a name or stated pronouns. Run them through the same workflow and compare the AI outputs and downstream decisions. Repeat with varied examples rather than relying on one pair. These tests can reveal sensitivity to a detail, but fictional cases may not represent the full range of real submissions.
Investigate differences without jumping to conclusions
If you find a gap, trace where it first appears. Are some people less likely to complete an input form? Does the AI omit relevant qualifications in its summaries? Do reviewers interpret similar summaries differently? Inspect representative cases, including ones that went against the overall pattern, and ask people familiar with the workflow to check your interpretation. Keep identifying details out of shared review materials unless they are necessary.
Consider the baseline: how were cases handled before AI was introduced, or how does a comparable process work without it? The comparison may be imperfect, but it can help distinguish a problem introduced by the AI from one already present in the surrounding process. Be cautious about assuming that equal rates of every outcome are achievable or appropriate in every context. The aim is to identify avoidable, unjustified differences and the decisions that produce them.
Make a change, then test again
Choose a response tied to the suspected cause. You might remove irrelevant personal details before generating a summary, require the AI to cite specific application evidence, revise a reviewer checklist or send uncertain cases to a person rather than forcing a recommendation. Document what changed and what improvement you expect. Check that the fix does not introduce a different problem, such as hiding information a reviewer genuinely needs.
Run the same comparisons after the change, using new cases where possible. Continue checking periodically because prompts, forms, staff practices and the people using the process can change. Keep a record of definitions, test cases, limitations and decisions so another person can understand what was assessed. A useful fairness test does not end with a reassuring number; it gives a team a repeatable way to notice unequal outcomes and respond to them.
