How to plan an AI pilot project that gives you a clear answer

What a pilot is, and what it isn't

A pilot is a time-boxed test of a specific change, with a decision at the end. It is not a demo, a proof that AI is clever, or the first phase of a project that has already been approved. If nobody is prepared to stop it when the numbers disappoint, it is not really a pilot.

For a small service business, the right size is usually one workflow for one team for about 90 days. Examples: AI-drafted proposals for the sales team at an agency, automated document requests for tax season at a CPA firm, or AI-assisted triage of maintenance requests at a property management company.

Choose a workflow that is frequent enough to generate data quickly. A pilot on something that happens twice a month will not produce enough examples in 90 days to judge.

Days 1 to 30: diagnose and define

The first month is about agreeing what you are testing and how you will judge it. Do not build anything yet. Work through these steps with the people who do the work:

Keep the paperwork light. A shared document with the map, the baseline spreadsheet and the criteria is enough. The point is agreement, not documentation for its own sake.

• Map the current workflow step by step, including which systems and people are involved.
• Time 10 to 20 recent real examples to get a baseline, such as minutes per proposal, hours to first reply or correction rate.
• Agree the scope: which part of the workflow changes, which team, and what stays exactly as it is.
• Identify the human review step: who approves output before it reaches a client, a price or a record.
• Check data handling: which tools are approved for the data involved, and whether any regulation applies.
• Write acceptance criteria and a break-even number, and get the owner to sign off.

Writing acceptance criteria that mean something

Acceptance criteria are the conditions under which you would keep the change. They should be specific, measurable and agreed before anything is built. Vague criteria such as "saves time" or "staff like it" make every pilot look successful.

Good criteria for an AI proposal pilot might read: staff time per proposal falls from a baseline of about 90 minutes to under 45 minutes; fewer than one in ten drafts needs a major correction; at least four of the five sales staff use it for most proposals by week ten; no pricing errors reach a client. The numbers are illustrative. Yours should come from your baseline.

Include at least one quality or safety criterion alongside the speed one. Faster but wrong is not a win.

Days 31 to 60: build and test with real work

Build the smallest version that could meet the criteria. Often that means templates, an approved AI assistant, and one or two automations rather than custom software. Then test in two stages.

Stage one: run 20 to 50 past examples through the new process and compare the outputs with what was actually sent. Include awkward cases, such as an unusual scope, a client with special terms or a messy set of notes. Fix the obvious problems.

Stage two: use it on live work with a person reviewing every output. Ask reviewers to log corrections in a simple sheet: what was wrong and how long it took to fix. That log becomes your correction rate and tells you where to improve the prompts, templates or data.

Keep the old process available during this phase. If the new version stalls on a busy day, staff should be able to fall back without missing a deadline. Make it clear that falling back is fine as long as they note why, because those notes tell you what to fix.

Days 61 to 90: measure and decide

In the final month, the change runs as part of normal work and you measure it against the baseline. Track usage weekly, because adoption problems surface here. If people avoid the new process, ask why: it may be slower in practice, the output may need too much fixing, or the old habit is simply easier.

At day 90, hold a decision meeting with the owner and the team. Compare the measures with the acceptance criteria and make one of three calls:

Collect qualitative feedback as well, in a short survey or a 15-minute conversation with each user. Ask what they would miss if it were switched off tomorrow. If the answer is nothing, take that seriously, whatever the numbers say.

• Keep and extend: criteria met. Document the process, train anyone not in the pilot, and consider the next workflow.
• Adjust and re-test: close but not there. Change one or two things and run another 30 days.
• Stop: criteria clearly missed. Record what you learned and switch it off. That is a legitimate result.

Common reasons pilots fail

Most failed pilots fail for organizational reasons rather than technical ones. Watch for these:

Most of these are prevented by the plan itself: a written scope, a baseline, named people and a date in the calendar for the decision. That is why the planning month is worth the time.

• The scope grows mid-pilot, so nothing is measured cleanly.
• No baseline was taken, so nobody can say whether anything improved.
• The people who do the work were not involved in the design and do not trust the output.
• There is no named owner with time set aside, so issues sit unresolved for weeks.
• The review step is skipped under time pressure, and an error reaches a client and kills confidence.
• The decision meeting never happens, and the pilot drifts into a half-used tool.

A one-page pilot plan

Put the whole plan on one page so everyone can see it: the workflow and team, the problem in one sentence, the baseline numbers, the acceptance criteria and break-even, the review step, the tools and data rules, the owner and the dates for the three phases and the decision meeting.

This is the same Diagnose, Deliver, Measure structure InTheory uses for its 90-day engagements. Whether or not you work with anyone outside, the discipline of a written plan and a scheduled decision is what makes a pilot worth running.

Share the result with the whole team, whichever way the decision goes. People learn as much from a pilot that was stopped for good reasons as from one that succeeded.