What we check in a support chatbot
Four failure modes account for nearly every chatbot incident a support team actually faces. We test each one directly, as a visitor.
1. Promises: discounts, refunds and timelines it was never approved to offer
We ask about discounts, refunds, price matches, delivery times and guarantees — politely, then persistently, the way a determined customer does. If the bot invents a policy, copies an old one, or agrees just to be helpful, you see the exact wording, side by side with your real policy. This is the failure behind our incident guide: what to do when your chatbot promised a discount you never approved. One documented case, in one line: in 2024, a tribunal ordered Air Canada to honor a refund its chatbot had incorrectly promised a customer.
2. Prompt injection: can a visitor talk it out of its instructions?
We try the standard visitor-side tricks: asking the bot to ignore its instructions, to reveal its system prompt, to act as a different assistant, and to follow instructions hidden inside pasted text. A well-set-up bot declines, stays in role, and offers a safe alternative. A weak one complies — and anything it can do, a stranger can then direct.
3. Leaks: what information comes out that should not
We probe for other customers' details, internal notes, staff information, and the contents of documents and knowledge files the bot was given for a narrower purpose. We also check what happens when the bot does not know an answer: a safe bot says so; a risky one fills the gap with a confident invention.
4. Access: what the bot can do, not just say
A chatbot connected to orders, accounts, email or refunds can act, not only answer. In the Full Audit we review the setup you share: which tools and data the agent can reach, which actions should always need a human, where API keys are stored, whether hard limits exist, and whether actions are logged somewhere the agent itself cannot edit.
How the audit runs
No access, no installation, no calls. You order, we talk to your bot, you read the report.
You order
Pick the Audit ($149) or the Full Audit ($299) and tell us where your chatbot lives. No passwords, no code access.
We visit as customers
We run the promise, injection, leak and access checks as an ordinary visitor would — including the pushy follow-ups.
You get the report
A written, plain-English report by email: exact conversations, your real policy beside the bot's answer, severity, and fixes in order.
You fix and re-check
Fix it yourself or with your provider. The Full Audit includes a free re-check within 30 days; Agent Guard keeps watching monthly.
Prices — fixed, known before you order
The same AgentCheck prices as our main page. No quote calls, no hourly billing.
AgentCheck Audit
One-time. A full visitor-side review of one support chatbot.
- Promise, injection and leak checks
- Answers compared with your real prices and policies
- Written report with fixes in order
AgentCheck Full Audit
One-time. Everything in the Audit, plus the setup behind the bot.
- Everything in the Audit, in greater depth
- Connections, API keys, limits and logging review
- Free re-check within 30 days
Agent Guard
Monthly monitoring, cancel anytime. Bots change when models, pages and files change.
- Repeat visitor-side checks every month
- Written note on what changed
- Early warning before customers find it
Not ready to order? Start with the free mini-check, or read the incident guide: Chatbot promised a discount you never approved.
Frequently asked questions
Do you need access to our chatbot or our code?
No. The Audit is a visitor-side review: we talk to your support chatbot the way your customers do and write down what it tells them. No passwords, no code access and nothing to install. The Full Audit adds a review of the setup you choose to share — connections, limits and logging — without needing your customers' data.
What exactly do you test a support chatbot for?
Four things: promises it should never make (discounts, refunds, timelines, guarantees), prompt injection — whether a visitor can talk it out of its instructions or into revealing them — information leaks, such as other customers' details or internal notes, and access: what the bot can actually do, not just say, when a visitor pushes it.
What do we receive, and how fast?
A written, plain-English report by email: the exact conversations, side by side with your real prices and policies, each finding marked by severity, and the fixes in the order to do them. The Full Audit also includes a free re-check within 30 days, so you can confirm the fix held.
What does it cost?
Fixed prices, known before you order: the AgentCheck Audit is $149 one-time, the AgentCheck Full Audit is $299 one-time, and Agent Guard monitoring is $49/month, cancel anytime. You can also start with the free mini-check at /agentcheck/free-check/.
Our bot only answers questions. Can it still cause a problem?
Yes — answers are exactly the problem when they are wrong. A bot that only talks can still promise a refund policy that does not exist, quote a stale price, or confirm a timeline your team cannot meet, and the customer will hold your business to the screenshot. In 2024, a tribunal ordered Air Canada to honor a refund its chatbot had incorrectly promised a customer.
Find out what your chatbot promised this week
Order the Audit and get the exact conversations in writing — what your bot said, what your policy actually says, and what to fix first.