What should an AI agent pilot prove?

An AI agent pilot should show whether one defined workflow can be completed within agreed quality, permission, time and cost limits—not merely whether an impressive demonstration is possible. Agree on the evidence required for a production decision before building the pilot.

This guide uses a hypothetical CRM follow-up workflow. It is a planning example, not a claim about a client deployment or a benchmark for every agent. Its purpose is to help a buyer and implementation team agree on what “ready” means.

Start with a narrow, observable task

Suppose the proposed system reads a permitted enquiry, finds the matching CRM account, drafts a follow-up and queues it for human approval. In this pilot, sending email is explicitly outside the agent's permissions. A useful task record includes the input, account match, evidence used, proposed action, review decision and final outcome.

Define what is out of scope: merging contacts, changing account ownership, accessing another customer's records or sending without approval. A fluent draft should not compensate for crossing one of those boundaries.

Build the test set before optimizing the demonstration

Choose representative examples and deliberately difficult cases. Keep a held-out set separate from examples used during development. Re-run cases where the model can produce variable outputs, record the model and configuration, and retain failures. Do not select only successful runs for the report.

Download the pilot test sheet. It includes these cases plus access revocation and empty-input checks. The result fields are intentionally blank: a checklist is not evidence that tests have passed. Use synthetic or approved, appropriately redacted data.

Measure completed work and the cost of exceptions

Record the number of attempted tasks, successful tasks under the agreed rubric, handoffs, prohibited actions, latency and total operating cost. A safe handoff may be correct behavior, but it is not automatically an autonomously completed task. Report it separately.

For example, if 40 hypothetical attempts produce 28 acceptable completions, 10 safe handoffs and 2 failures, autonomous completion is 28/40, or 70%. That arithmetic describes the example only. It does not establish an acceptable target. The business must decide which tasks and failure consequences it can tolerate.

Separate total pilot spend from recurring run cost. Divide recurring spend by acceptable completions only when that denominator matches the business question; disclose excluded human review and exception costs. Our operating-cost guide explains the budget components, and the cost calculator helps structure planning assumptions. Neither predicts a pilot's measured performance.

Choose a production decision, not a vanity score

• Proceed: the agreed criteria are met, the evidence covers the intended scope, and someone owns operations and recovery.
• Narrow: a subset works, but ambiguous tasks or integrations need a human queue.
• Rework: a fixable issue, such as account matching or inconsistent source data, prevents acceptance.
• Stop: the workflow does not justify its cost or cannot meet a critical control requirement.

Document who can approve release, how the system is stopped and which changes trigger re-evaluation. Do not describe a pilot as production-ready until these operational responsibilities are settled.

Keep the pilot scope connected to the quote

A quote should distinguish integration work, evaluation work and ongoing operation. Ask who supplies the test cases, who labels acceptable results, how many iterations are included and what happens if the pilot fails. Our development cost guide helps identify the components to discuss; it is not a promise of results at a fixed price.

Some workflows need rules, not model-selected actions. The published product-monitoring project is a useful example of a narrow monitoring-and-notification task. Start with the simplest design that can pass your acceptance tests.