AI hallucinations: what they mean for your business and how to control them

What a hallucination is, in plain terms

AI language tools generate text by predicting what words are likely to come next, based on patterns in a vast amount of training data and whatever you give them. They are very good at producing text that sounds right. They do not check facts the way a person looking something up does, and they rarely say "I don't know" unless prompted.

So when the honest answer is missing from what the tool knows or was given, it may produce a plausible substitute: a case citation in the right format, a policy that sounds like your policy, or a number in the right range. That is a hallucination. Vendors acknowledge this. Microsoft's documentation for its Copilot, for example, cautions that generative AI responses may not be fully factual and tells users to review output before sending it to others.

Hallucinations are more likely when the question asks for specific facts the tool was not given, when it asks for citations or quotes, and when the topic is niche or recent. They are less likely when the tool is asked to work only from text you supply, although they can still happen there too.

Two cases every owner should know

In Mata v. Avianca, a federal case in New York, lawyers filed a brief citing court decisions that did not exist. One lawyer had used ChatGPT for research and did not verify the cases. On June 22, 2023, the judge fined the two lawyers and their firm $5,000 and required them to send letters to their client and to the judges named in the fake opinions. The court's focus was not only the use of AI but the failure to check, and the lawyers' conduct after the problem was raised.

In Moffatt v. Air Canada, decided by a British Columbia tribunal in February 2024, an airline's website chatbot gave a customer incorrect information about bereavement fares. The airline argued it was not responsible for what the chatbot said. The tribunal disagreed, finding the airline responsible for all the information on its website, whether from a static page or a chatbot, and awarded the customer damages.

The lesson in both is the same. If you rely on or publish AI output, you own it.

Neither case involved a small service business, but both describe situations a small firm can easily recreate: a staff member trusting a confident answer, and a customer-facing tool stating a policy nobody checked.

Where the risk sits in a small service business

Hallucinations matter most where an error is costly and might not be noticed. Common risk points:

• Client-facing documents: proposals, engagement letters, reports and emails that state facts, figures or terms.
• Pricing and estimates: invented line items, wrong quantities or prices that do not match your price list.
• Professional advice: legal citations, tax positions, coverage descriptions or clinical guidance.
• Website chat and FAQ assistants: answers about policies, warranties or availability that you may be held to.
• Internal knowledge assistants: confident but wrong answers about procedures that staff then follow.
• Summaries: a meeting or document summary that omits or invents a key point.

Controls that actually work

No setting eliminates hallucinations. Layered controls reduce the chance that one causes harm:

Reviewers need time to do this properly. If AI drafting saves 30 minutes but the review is squeezed into two, the controls exist on paper only. Build realistic review time into the workflow and the ROI calculation.

• A named reviewer: the person who sends, files or signs the work checks it, and your AI policy says so.
• Verify against the source: facts, figures, citations and quotes are checked against the original document or system, not against the AI's own explanation.
• Ground the tool: give it the relevant documents, price list or policy and ask it to answer only from them and quote the passage it used.
• Narrow the job: ask for a summary of a specific document rather than open-ended knowledge.
• Structured templates: have AI fill clearly marked sections of a template, with prices and terms pulled from your system rather than written by the model.
• Track corrections: log the errors reviewers catch, so you can see where the tool is weak.

Special care for public-facing AI

Website chatbots and AI receptionists talk to customers without a person reviewing each answer, which is exactly the situation in the Air Canada case. If you deploy one, limit it to a defined set of topics, answer from a reviewed knowledge base of your own content, and have it hand off to a person for pricing, commitments, complaints and anything it is unsure about.

Review transcripts weekly for the first months. Check that answers match your current policies, and update the knowledge base when policies change. Make it clear to visitors that they are talking to an AI assistant, and give them an easy route to a person.

Write the boundaries into the assistant's instructions and test them. Ask it questions you know it should decline, such as a discount request or a legal question, and check that it hands off.

Regulated professions

For lawyers, ABA Formal Opinion 512 makes the point directly: output from a generative AI tool must be carefully reviewed before it is submitted to a court, including analysis and citations. Similar logic applies in other fields. A CPA signing a return, an adviser making a recommendation or a clinician giving guidance remains responsible for the content, whatever tool helped prepare it.

Build verification into the workflow rather than relying on memory. A checklist step, such as "citations checked against the primary source by the signing attorney," is easier to follow and to prove than a general instruction to be careful.

Measuring and managing the risk over time

Treat hallucination risk like any other quality issue. Track the correction rate on AI-assisted work, note which types of errors appear, and review them in your six-month governance check. If one use keeps producing errors, narrow it, add grounding or move it back to a person.

InTheory builds these review steps into every workflow it designs: people approve prices, advice and anything sent to clients. The aim is to get the time savings of AI drafting without handing over the judgment.

Make it safe to report errors. If staff fear blame for missing a hallucination, they will quietly fix problems rather than log them, and you lose the information you need to improve the process.