A pilot can produce convincing answers while leaving the production decision unresolved. The missing work is usually specific: establishing who may use the system, how outputs will be judged, and what happens when it fails. Those questions belong in the pilot scope.
Define the decision the pilot supports
Choose a single workflow with a business owner who can approve the next step. Describe the current process, its costs and its failure modes before selecting an AI architecture. Define what will count as an acceptable result, which errors are unacceptable, and what evidence the decision-maker needs. A useful pilot can end with a decision to stop; discovering a poor fit is a legitimate outcome.
Build a representative test set
Use cases that reflect the work: incomplete files, conflicting instructions, unusual formats and requests outside scope. Keep development cases separate from an acceptance set. Record the reference outcome and the reason it is correct. Evaluate the final work product and any system action, rather than treating a fluent response as proof that the task was completed.
Test the operating conditions
The first release needs an access model, approved data flows and a named support owner. Exercise failure paths: a source is unavailable, a tool times out, or the reviewer rejects a draft. Confirm the process can continue manually and that retries cannot create duplicate actions. A production decision should state which limitations are accepted and which remain launch blockers.
Build a release packet, not a demonstration deck
The release packet should let a person who did not build the system make an informed decision. Include the workflow boundary, data-flow diagram, test report and list of unresolved exceptions. Pair each exception with an owner and a disposition: fix before launch, accept with a stated constraint, or exclude from the first release. Record the operating version of the model, prompts and tools so that the decision refers to a reproducible configuration.
For a document-intake pilot, a strong review includes the files that were misclassified, the corrections required and the time spent resolving them. It also confirms that an uncertain file reaches the right queue. An attractive extracted summary, viewed in isolation, cannot answer whether the overall workflow is ready.
Expand only after observing real work
Start with a bounded user group and an explicit review period. Compare accepted output, correction effort and exception volume with the baseline. Keep a record of changes to prompts, tools and source collections. Broaden the scope when that evidence supports it, with a new decision for any increase in authority.
Before the next decision
Planning checklist
- Named workflow owner and acceptance criteria
- Representative cases with recorded expected outcomes
- Approved access, support and rollback arrangements
- Release decision with unresolved risks listed
Sources and further reading
Sources checked 5 October 2026. The design recommendations in this guide are AID’s analysis. Sector obligations depend on the use case and jurisdiction.
Apply this to a defined workflow
AID’s Governed Pilot starts with a fixed scope and an agreed success criterion. The release decision is based on the resulting evidence.
