Research guide · Experiments
Define success before building the proof of concept.
A convincing demonstration is a useful conversation starter. An experiment needs criteria that can also reveal when the approach is unsuitable.
Write a claim that could be disproved.
An AI proof of concept tests a specific technical assumption under limited conditions. Its success criteria should describe the evidence needed for the next decision, rather than the features required for a polished demonstration.
Start with a claim such as: “Given the approved source collection, this approach can produce sufficiently supported draft answers for expert review.” Then define the vague terms. Which questions are included? What counts as sufficient support? What errors would make the approach unsuitable even if the average answer looks useful?
The point is not to make the question being tested impossible to satisfy. It is to prevent the goal from moving after the first results appear. An agreed research question also helps constrain the build: components that do not help test it can wait.
The NIST AI RMF Core calls for documented test sets, metrics and evaluation methods. The steps here are our proposed method for a small, focused test; they are not reported results from an existing benchmark.
Choose a baseline and reserve test examples.
A result has more meaning when compared with an appropriate alternative. The baseline might be a simple search method, a current model configuration or the existing review process. Keep the comparison fair: equivalent input information, a consistent task and clearly stated measurement conditions.
Use development examples to build and refine the approach. Reserve a separate set for assessment where the available material permits it. If every difficult example becomes a prompt adjustment and is then scored as fresh evidence, the final result can overstate how well the approach generalises.
The test set should contain ordinary work as well as meaningful edge cases. Include missing evidence, contradictory material and ambiguous questions where relevant. A system that always produces an answer can look productive while concealing an important failure to recognise uncertainty.
Agree how reviewers will score it
Some tasks have exact answers; others permit several good responses. Define what reviewers should reward and what they should mark as an error. Try the scoring guide on a few examples before using it at scale. Disagreement may reveal an unclear task rather than a weak model.
Use several criteria, with explicit priorities.
Correctness, evidence support, completeness, speed and estimated cost answer different questions. Combining them into one score too early can hide the trade-off that matters. A small improvement in average speed may be irrelevant if the system fails on a critical exception.
| Criterion | Define before testing |
|---|---|
| Task quality | The scoring guide, acceptable outputs and material errors. |
| Evidence support | How each important claim is checked against permitted sources. |
| Knowing its limits | When the system should ask for clarification or decline an answer. |
| Practicality | Relevant speed, usage cost and human review measures. |
| Stop conditions | Failures or constraints that prevent a useful next step. |
Thresholds should follow the task and its consequences. There is no universal percentage that makes an AI prototype successful. Record why the chosen thresholds are appropriate and which ones are mandatory rather than preferences.
Where behaviour varies between runs, consider repeated trials within the scope. Record model identifiers, settings and relevant versions. Without those details, a later team may struggle to distinguish a changed model from a changed experiment.
End with proceed, revise or stop.
A proceed decision means the evidence supports a particular next stage under stated assumptions. A revise decision identifies what should change and why another experiment is justified. A stop decision explains why the current approach does not merit further investment.
Include examples of failures in the handover. They often reveal more about the next engineering decision than a collection of successful outputs. Report the sample's limits and distinguish technical findings from assumptions about future users or operating conditions.
A successful prototype is not the same as a live system. Reliability at real volume, security and any regulatory requirements are the work that turns it into a product. It is worth planning for it before you build the prototype, not after.
Explore AI Proof of Concept for a focused engagement, or read how to evaluate LLM answers when the central uncertainty is output quality. If you are still defining the task, begin with the feasibility study guide.
See a small worked example.
Inspect the source, illustrative answers and review judgments in our answer-evaluation example. The downloadable worksheet shows a possible recording format; it is not a client result or a model benchmark.