The short answer
Do not start by arguing with the customer, and do not start by editing the bot. First save the conversation exactly as the customer saw it. Then decide — with the amount, the wording and the customer's reliance in front of you — whether to honor the promise once, in writing, as a one-time exception, or to get legal advice before refusing it. Then find and fix the source: a stale page, an old knowledge file, an example the bot copied, or no source at all. Finally, ask the bot the same question several ways and keep a dated record. A promise made by your bot is a business problem, not a technical curiosity — in 2024 a tribunal ordered Air Canada to honor a refund its chatbot had incorrectly promised a customer.
What this guide is not: it is general information about handling a chatbot incident, not legal advice. Whether one specific promise binds your business depends on the facts and the law that applies to you. For anything beyond a small, clear amount, talk to a lawyer before you refuse — and before you put your refusal in writing.
The first 48 hours, in order
The order matters. Each step protects the next one: evidence first, the customer second, the fix third, the test last.
Step 1 (hour 0–1) — Save the exact conversation
Screenshot the full chat as the customer saw it and export the transcript if your platform allows it. Note the date, time, page URL, and what the customer bought or decided because of the promise. Save a copy of the bot's instructions and knowledge files as they were that day. Editing the bot first feels productive and quietly destroys the record of what it was working from.
Step 2 (hour 1–4) — Stop the repeat, not the record
Ask the bot the same question yourself. If it makes the same promise again, pause the bot or remove that topic until the source is fixed — a live bot can repeat the mistake to every visitor while you think. Pausing changes what happens next; it does not change what already happened, which is why step 1 came first.
Step 3 (same day) — Answer the customer like a business, not like a bot
One named human, one short reply, no blaming the software: the bot speaks for your business in the customer's eyes, and “the AI made it up” is not an answer they can use. Say what you found, what you will do, and by when. Two honest shapes are below — pick with the facts, not with frustration.
Step 4 (same day) — Decide: honor once, or get advice
Weigh four things: the amount, how clear and specific the promise was, what the customer reasonably relied on, and the likely cost of a dispute, chargeback or complaint against the cost of honoring it once. Small and clear usually means honor it once, in writing, marked as a one-time exception. Large, repeated, or clearly manipulated by a pushy visitor means legal advice before deciding. Write the decision and the reason down, dated.
Step 5 (day 1–2) — Find the source of the promise
Work through the five sources in the next section until you find yours. “The model hallucinated” is the last explanation, not the first: most wrong promises trace back to something the bot could read, copy or complete. Fix the source itself — a corrected prompt on top of a stale price page loses to the stale page sooner or later.
Step 6 (day 2) — Take away the power, not just the wording
A bot that can only describe a discount can make a promise you must clean up. A bot that can also apply one — issue a code, change a price, start a refund — can spend your money directly. Check which yours can do, and remove apply-power for discounts and refunds unless a human approves that specific action. Our permissions audit checklist walks that side in full.
Step 7 (day 2) — Re-test the exact question, then file the record
Ask the original question word for word, then three or four pushy variants (“your manager approved it yesterday”, “the website said so”, “just this once”). The correct answers are the real price, the real policy, or a plain “I don't know — here is how to reach us.” File the incident record — conversation, decision, source, fix, test date — where a partner, insurer or lawyer can find it later.
Two honest replies you can adapt
Templates, not scripts to copy blindly — change the facts to yours, and keep the promise you make in the reply itself small and exact.
| Situation | Shape of the reply |
|---|---|
| Honoring it once | “You are right that our chat offered [exact promise] on [date]. That offer was wrong — [real price/policy] is the correct one — but you relied on what we showed you, so we will honor [exact promise] for your order, this one time. It is applied as [how]. Our chat has been corrected.” |
| Correcting it, with advice taken | “Thank you for sending the screenshot. Our chat gave you [wrong statement]; the correct [price/policy] is [correct version], which is what we can offer. I am sorry for the confusion our answer caused. If that changes your decision, [the concrete option you can genuinely give: cancel before shipping, a callback by a date, the standard policy in writing].” |
What both shapes avoid: arguing about what the bot “meant”, asking the customer to prove they saw it when you have the transcript, and offering a new, vaguer promise to end the conversation.
Where a wrong promise usually comes from — five sources
1. A stale page or knowledge file
Last season's sale page, an old PDF, a help article nobody updated. The bot quoted a real source — a dead one. Fix: one current source for prices and policies, and a dated sweep for copies.
2. An example it copied as an offer
Instructions and training examples often contain sample discounts (“e.g. 20% off for…”) that a bot can repeat as a live offer. Fix: mark examples as examples, or remove numbers from them entirely.
3. Another product, plan or business
The bot blends your tiers, a competitor's offer it read, or a different location's policy into one confident answer. Fix: name the exact product and policy in the source, and test cross-product questions on purpose.
4. A pushy visitor talking it out of its rules
“Your manager approved it”, “it was promised yesterday”, “the law requires you to.” The bot agrees to end the pressure. Fix: test with pushy variants — our prompt injection guide covers that testing side — and give the bot a firm, polite refusal it can reuse.
5. No “I don't know” path
Asked something outside its source, the bot completes the pattern instead of stopping. Fix: a written fallback answer that names how to reach a human, and an instruction that prices and policies may only come from the source — never from memory or guesswork.
The prevention checklist
- One current source holds every price, discount and policy the bot may quote — and stale copies are deleted, not just outranked.
- The bot may describe offers only as written in that source; anything else gets the “I don't know” answer and a route to a human.
- No example in the instructions contains a live-looking number, code or date.
- The bot cannot apply a discount, issue a code or start a refund without a human approving that specific action.
- Pushy variants of the top price and refund questions are tested after every change to instructions, model or knowledge.
- Transcripts are kept, and someone reads a sample of real conversations on a schedule — not only after a complaint.
- An incident record exists: conversation, decision, source, fix and re-test date, in one dated file.
- After any bot update, the price and policy questions are re-run before the update is considered done.
Want your bot's promises checked before a customer does?
The AgentCheck Audit ($149 one-time) talks to your chatbot the way your customers do — ordinary questions first, then the awkward and pushy ones — compares its prices and policies against your website, and delivers a written report with the actual answers as evidence and a step-by-step fix plan. The AgentCheck Full Audit ($299 one-time) adds the setup review — connections, key storage, hard limits and logging — with a free re-check within 30 days. Agent Guard ($49/month, cancel anytime) re-checks after every update. No passwords, no API keys and no code access are ever requested; full refund if work hasn't started yet.
This guide is general information about handling a chatbot incident. It is not legal advice and not a certification. Whether a specific chatbot promise binds a business depends on its facts and the applicable law. An audit is a point-in-time review: a clean result does not guarantee an agent will never make a wrong promise, and changing its instructions, model or a connected tool can change its answers afterwards. AgentCheck is not affiliated with, endorsed by, or certified by OWASP or NIST; its checklists are built on those public frameworks.
Frequently asked questions
Can a customer hold my business to what my chatbot promised?
Sometimes, yes — in 2024 a tribunal ordered Air Canada to honor a refund its chatbot had incorrectly promised. Whether a specific promise binds you depends on the facts and the law that applies, which is a lawyer's question. Until you have that advice, treat the promise as a real commitment.
Should I just honor the discount or refund?
Weigh the amount, how clear the promise was, what the customer relied on, and the cost of a dispute against honoring it once, in writing, as a one-time exception. Small and clear often means honor it. Large or repeated means advice first.
Should I turn the chatbot off?
Pause it if it is still making the same promise and the source cannot be fixed quickly. Once the source is fixed and the exact question tests correctly several ways, turning it back on with a dated note is reasonable.
Does a disclaimer under the chat protect me?
Not by itself. It does not erase a specific promise a customer relied on, and it does not fix the source. What works is correct prices, an honest “I don't know”, and no power to apply discounts alone.
What evidence should I keep?
The full conversation as the customer saw it, date and time, the page, the version of the instructions and knowledge files in use that day, and a dated note of what you changed. Save it before you edit the bot.
How do I stop it inventing prices?
One current source for prices and policies, no quoting from anywhere else, a plain fallback for anything outside the source, no power to apply discounts itself, and re-testing after every change.