Acceptance criteria for automation projects

What acceptance criteria are for

Acceptance criteria answer a simple question: how will we know this is finished and working? They turn a description like the automation creates new clients in the system into checks you can run, such as: for each of 25 signed engagement letters, a client record exists in the practice management system within five minutes with the correct name, entity type and partner assigned.

Without criteria, acceptance becomes a matter of mood. The builder says it is done, someone finds a problem two weeks later, and an argument starts about whether that problem was part of the job. Written criteria protect both sides and make the final payment straightforward.

Good criteria have three qualities. They are specific enough that two people would agree on pass or fail. They use your real data rather than a vendor's sample. And they cover what should happen when things go wrong, not just the happy path.

Write them before the build

Criteria belong in the scope document, agreed before work starts. Writing them first changes how the work is built, because the builder knows what will be tested. It also forces useful conversations early. Should a duplicate contact be merged or flagged? What happens when a client has no email address? Those questions are cheap before the build and expensive after it.

Involve the person who will use the automation every day. An office manager or dispatcher knows the odd cases that a manager or consultant will miss.

Keep the list short enough to actually run. Eight to twelve clear criteria usually cover a single workflow. Fifty detailed criteria tend to be skipped, which defeats their purpose.

Build a test set from real examples

Gather 20 to 50 real examples of the inputs the automation will handle: past enquiries, intake forms, invoices, call notes or estimates. Remove or mask personal details where needed. Include the awkward ones on purpose: the handwritten form, the client with two locations, the enquiry in Spanish, the invoice with a credit line.

For each example, write the expected result. That set becomes your test. Run it before launch and again after any significant change, such as a software update or a new field in the CRM.

Store the test set somewhere safe and permanent, such as a restricted folder in SharePoint or Google Drive, along with the expected results. Treat it as part of the system. When the builder leaves or a new person takes over, the test set is what lets them check their changes without guessing.

Set criteria by type of step

Different steps need different kinds of tests.

• Rules-based steps, such as copying data or sending reminders on a schedule: expect every test example to pass. A reminder that sends on the wrong day is a defect.
• AI steps, such as extracting fields from a PDF or drafting a reply: set a pass rate on the sample, for example at least 45 of 50 fields correct, and require that uncertain cases are flagged for a person.
• Timing: how quickly something must happen, such as a lead receiving a first reply within five minutes during business hours.
• Failure handling: what happens when a step fails, who is notified, and how nothing is silently lost.
• Security: who can see the data, and confirmation that the automation runs in company-owned accounts.

Include the human review step

Wherever output reaches a client, or touches price, legal, medical or financial advice, the criteria should name a review step and test it. For example: every AI-drafted proposal is saved as a draft, never sent automatically, and the account manager must approve scope and price before it goes out. Test that the draft cannot be sent without that approval.

Track the correction rate during acceptance. If reviewers are rewriting most drafts, the automation is not saving time yet, even if it technically works. A reasonable criterion might be that reviewers make only minor edits on most drafts in the test set, with the threshold agreed in advance.

Test the review step under realistic pressure as well. On a busy Monday, will the reviewer actually have time to check 30 drafts? If not, the criteria should limit volume, route drafts to more than one reviewer, or flag only the cases that genuinely need attention. A review step that people skip when busy offers little protection.

Example criteria for common workflows

The examples below are illustrative. Adjust the numbers to your volumes and the risk involved.

Notice that each criterion is something a non-technical person can check by looking at results. That is the point. If a criterion can only be verified by the builder, rewrite it.

• Lead response: every web form enquiry receives an acknowledgment within five minutes during business hours, and a task is created for the right salesperson with the enquiry details attached.
• Document intake: on 40 sample client documents, the extracted name, date and amount match the source on at least 38, and every low-confidence result is routed to a person.
• Appointment reminders: reminders send at the agreed intervals, stop when the appointment is cancelled, and respect patients who opted out.
• Invoice follow-up: reminders go only to invoices that are actually overdue in QuickBooks Online, never to clients on a payment plan flagged in the system.
• Call notes to CRM: summaries attach to the correct contact in HubSpot, and no summary is created when the call recording is missing.
• Proposal drafting: drafts pull the current rate card, are saved as drafts only, and require approval by a named person before sending.

Sign-off and what happens next

Name the person who signs off, usually the internal owner of the workflow, and the date of the acceptance test. Record the results in a simple table: example, expected, actual, pass or fail. If a criterion fails, the builder fixes it and the affected examples are rerun.

Acceptance is not the end. Agree on a short monitoring period, perhaps 30 days, where the internal owner checks a sample of live results each week. Real traffic always reveals cases the test set missed. The criteria then become the standard you use whenever something changes, which keeps the automation trustworthy long after the builder has moved on. InTheory includes acceptance criteria in every scope for this reason.