How to audit AI chatbot answers: a 20-minute weekly routine for small support teams

A weekly routine for checking what your AI support agent tells customers: a three-slice sample, six grades, six causes with their fixes, and the arithmetic on what 20 conversations can and can't prove.

Talvo's AI performance panel, in front of the inbox where AI replies show the articles they cited. Shown with our demo workspace.
On this page
  1. Pick the 20 conversations on purpose
  2. Six grades for every AI answer
  3. Trace each bad answer to its cause
  4. How to audit AI chatbot answers in 20 minutes
  5. A table you can copy
  6. When to read more than 20
  7. What not to do
  8. Running the audit in Talvo, as it works today
  9. Frequently asked questions

You switched on an AI agent and you still have a quiet worry: what is it actually telling people? The way to audit AI chatbot answers without building a QA department is boring and reliable. Read 20 conversations a week, grade each answer on one six-grade scale, trace every bad answer to the article (or missing article) behind it, and fix the source rather than the symptom. Reading and grading takes about 20 minutes; fixes are extra.

Pick the 20 conversations on purpose

Twenty isn't magic. It's roughly what a busy founder will keep reading, and it finds problems if you split it into three slices.

SliceHow manyHow to chooseWhat it's for
Random AI-resolved10A rule fixed in advance, e.g. every third conversation the AI closed on Tuesday and WednesdayYour honest read on a typical answer
Handed off5The most recent hand-offs, skipping plain "can I talk to a person" requestsGaps in your content, and hand-offs that weren't needed
Low-rated or reopened5Any conversation with a 1 or 2 rating, or where the customer came back about the same issueWhere the damage is

The random slice is the only one that tells you how often the AI is wrong. The other two are biased on purpose toward where problems cluster. Count them separately and never blend the three into one "accuracy" figure.

If the AI handled fewer than about 20 conversations this week, skip the sampling and read all of them. If you can't find five low-rated conversations, fill the slot with more random ones.

What 10 random answers a week can and can't tell you

Small samples are good at noticing a problem that happens often and bad at proving a problem is rare. If a fraction p of answers are wrong, the chance that a sample of n answers contains at least one is 1 − (1 − p)^n.

Answers readWrong 2% of the timeWrong 5% of the timeWrong 10% of the time
10 (about one week's random slice)18%40%65%
2033%64%88%
40 (four weeks)55%87%over 98%
80 (eight weeks)80%98%over 99%

Each cell is the chance you'll see at least one wrong answer. The rows count answers: if your conversations hold more than one AI answer each, count answers for n, and remember that the random slice only covers conversations the AI closed. So one week of ten random answers will usually miss a problem in one answer in twenty. Four weeks of the same routine will usually catch it.

The reverse matters just as much. With zero errors in 20 random answers, the true error rate could still be as high as about 14% at 95% confidence (the exact one-sided binomial bound is 1 − 0.05^(1/20)). With zero in 40, it's about 7%. That's why the tracking table below keeps a rolling four-week total for the random slice, and why you should say "no wrong answers in the last 40 we checked", never "the AI is accurate".

The NIST handbook's section on proportions explains the Wilson and exact methods behind intervals like these. Our NPS and CSAT calculator uses the two-sided Wilson interval, which runs a little wider: for 0 wrong in 20 it puts the upper limit near 16%, and for 0 in 40 near 9%. The point is the same either way.

Six grades for every AI answer

Grade each AI message that tried to answer something, not the conversation as a whole.

The examples below come from an invented business, so treat them as illustrative: Kiln & Co., a seven-person online store selling pottery clay, glazes and small kilns.

1. Correct and sourced. The answer is right, complete, and points to the article it came from. A customer asks whether Kiln & Co. ships glaze to Canada. The AI says yes, mentions duties, and cites the "International shipping" article, which says the same thing. Nothing to do.

A visitor asks how to reset their password and Talvo AI answers in the widget, citing the Resetting your password article.
Talvo's chat widget, shown with our demo workspace: the AI answers and names the help article it used, which is what a "correct and sourced" answer looks like to the customer.

2. Correct but unsourced. The answer is right, but nothing shows where it came from. Asked how to store opened clay, the AI gives sound advice (wrap it, keep it out of frost) with no article attached. Next quarter you can't check it, and neither can the customer. Find out what it was drawn from; if customers keep asking, give it a public article.

3. Outdated. The answer matches an article, but the article is stale. Kiln & Co. raised its free-shipping threshold from $75 to $99 in September. Nobody edited "Shipping rates", so the AI tells a customer that a $80 order ships free, and cites the article to prove it.

4. Wrong. The answer says something false. Asked "Can I return an opened bag of clay?", the AI says yes, within 30 days. The real policy is unopened bags only, but that condition sits in a separate "Return exceptions" article, while "Returns" leads with "30-day returns" and never mentions it. This is the grade that costs money.

5. Should have handed off. The AI answered something a person needed to handle. A customer writes that their new kiln arrived with a cracked element. The AI offers troubleshooting steps. The customer needed a damage claim and a replacement, which no article can do.

6. Handed off when it could have answered. A customer asks, "Do you do will-call?" The AI hands off. The answer is in the "Local pickup" article, but the article never uses the customer's words, so the AI didn't find it. Each one is a conversation your team handled for nothing.

Grades 3, 4 and 5 are the ones to fix this week. Grade 4 about money, safety or legal terms is the one to fix today.

Trace each bad answer to its cause

Correct the customer, then find the cause, because the same answer is going out to everyone who asks. Nearly every bad answer comes from one of six causes.

CauseWhat you'll seeThe fix
Missing articleA hand-off, or an answer stitched together from loosely related articlesWrite one article that answers that question, titled the way customers ask it
Contradictory articlesThe AI cites one article while another says the opposite, or cites bothMerge them into one, then unpublish the loser
Outdated articleThe right article is cited, but it's staleEdit it, then search for every other article that repeats the old fact
An internal note or file the AI shouldn't useThe answer contains something no public article says: a wholesale price, a staff-only workaroundRemove the file or snippet, or rewrite it so it holds only what customers may hear
A question that needs a personAny answer here is wrong, however well-sourcedAdd the topic to your vendor's always-hand-off list, and write an article that says how to reach you
The article exists but uses different wordsA hand-off on a question your help center already answersPut the customer's phrase in the title or first sentence

Two of these deserve a closer look.

Contradiction is the classic failure. In the Air Canada case, the airline's chatbot said a reduced bereavement fare could be claimed within 90 days of the ticket being issued, while the page it linked to said the policy doesn't apply after travel is completed. The tribunal found that Air Canada "did not take reasonable care to ensure its chatbot was accurate" (Moffatt v. Air Canada, 2024 BCCRT 149, para 28). Our breakdown of the Air Canada chatbot ruling goes through it. For an audit, the lesson is practical: when you find one wrong answer, search your help center for the topic and read every article that mentions it.

Internal material is the one people forget: if an answer contains something you'd never publish, the source is almost certainly a private file or saved answer, not your help center. For the wording fix in the last row, our guide to writing help articles an AI can answer from goes into detail.

How to audit AI chatbot answers in 20 minutes

  1. Pull the sample (3 minutes)Ten random AI-resolved conversations using your fixed rule, five hand-offs, five low-rated or reopened. Note the slice next to each one.
  2. Read and grade (12 minutes)One grade per AI answer from the six above. Don't fix anything yet. Write the question and the grade in one line.
  3. Mark the causes (3 minutes)For every grade 3, 4, 5 or 6, write which of the six causes it is and which article, file or topic is involved.
  4. Assign the fixes (2 minutes)Money, safety or legal answers get fixed today. Everything else gets an owner and a day this week.
  5. Re-test after fixingA fix you haven't tested is a guess. Ask the exact question again in your own widget, in the customer's words, check the answer and its source, and record the result. This sits outside the 20 minutes.

A table you can copy

One row per week. Only the random columns measure accuracy; the rest count work.

WeekRandom wrong (3–5) / readRolling 4 wks randomHand-offs that weren't neededShould have handed offOutdatedFixes shippedFixes re-tested OK
Oct 261 / 101 / 1020133
Nov 20 / 101 / 2011021
Nov 9

The figures are illustrative. Add a notes column for which questions failed: the same question returning in week three is the most useful thing the table shows.

When to read more than 20

Widen the sample when
  • Any answer about money, refunds, safety or legal terms was wrong
  • A fix failed its re-test
  • You changed prices, policies, shipping or plans this week
  • You imported or rewrote a batch of help articles
  • You changed the AI model or its settings
  • AI conversation volume roughly doubled
  • The rolling four-week random slice shows two or more wrong answers

When one of these fires, read every AI conversation on the affected topic that week rather than a bigger random sample. Searching the topic finds more than sampling does.

What not to do

Don't steer by resolution rate. A rising rate can mean the AI is answering more questions correctly, or that it's answering more questions it shouldn't, and the number looks the same either way. The rate needs an accuracy check beside it, as our piece on what a good AI resolution rate is explains, and this routine is that check.

Don't change the model first. A different model reading the same contradictory articles will contradict itself just as confidently. The cause is almost always content, and content is cheaper to fix.

Don't fix tone to fix facts. Brand-voice settings change how an answer sounds, not whether it's true.

Don't treat a good rating as a correct answer. Customers rate how the whole conversation felt, often before they find out the refund didn't go through. Ratings tell you what to read, not whether it was right.

And don't shrink the hand-off list to make the numbers look better. A hand-off on a refund dispute is the system working. Our guide to AI-to-human hand-off rules covers where the line should sit.

Running the audit in Talvo, as it works today

Here's what Talvo gives you for this routine and what it doesn't, as of October 2026. Talvo is new and in beta, so check the current version.

Seeing the source of each answer. In the inbox thread, every AI reply shows the titles of the articles it cited as small chips. The customer sees the same chips under the AI's reply in the widget. That makes grades 1, 3 and 4 quick to check. The chips show the title only and aren't links, so open the article from the Knowledge page and compare.

Talvo shared inbox: a customer asks how to rotate an API key, Talvo AI answers, and the AI brief summarizes the conversation.
Talvo's shared inbox, shown with our demo workspace: the AI answered and summarized the thread.

The blind spot: private files and snippets. Talvo's AI can also answer from internal knowledge files and private snippets (Settings → General → AI knowledge files). Those are never shown as sources, to customers or your team: an answer drawn only from a file or snippet has no chip, and nothing tells a teammate which file or snippet was used. Talvo turns an uncited answer to a real question into a hand-off, so an AI reply with no chip is either a greeting or thanks, or it came from a file or snippet. That makes a substantive answer with no chip your cue to check your files and snippets by hand. Keep that list short so the check stays quick.

Finding the conversations. There is no list of all AI answers, and no inbox filter that shows only AI-handled conversations. You open conversations one at a time, which is why the fixed random rule matters. Conversation tags work on every plan, so tag each one you've graded (say ai-audit) and filter by the tag to find them again next week.

You can't mark an AI answer wrong or correct it in place. There's no thumbs-down, flag or rating on AI messages, and customers rate the whole conversation, 1 to 5 stars, when it's resolved ("How was your support experience?"), not individual answers. Your grades live in your table or an internal note. "Teach the AI" sits under a teammate's own reply, not under the AI's messages: you reply to the customer yourself, then use Teach the AI on your reply, which saves the question and answer as a private snippet the AI and copilot use from then on. It's on every plan. Snippets carry no chip, so answers taught this way look unsourced in later audits. When a fact belongs in your help center, edit the article instead.

Reading a hand-off. Every AI hand-off leaves an internal note that starts "AI handed off:" and names the reason: no published articles, related articles that don't contain the answer (up to three are linked on the note), only internal files came close, no article covers the question, the AI budget is used up, or the AI couldn't be reached. The first four map straight onto the cause table above, so for the hand-off slice the note often tells you which fix to make.

Forcing a person. Settings → General → AI controls has an "Always hand these topics to a person" list. A message that matches a listed word or phrase goes to your team before the AI tries an answer. That's the fix for grade 5.

Run this audit on your own help centerTalvo's Free plan includes 25 AI resolutions a month during the beta, with citations on every article-based answer. No credit card.
Start free

The AI performance panel (Scale only). On Talvo's Scale plan, $299 a month or $249 a month billed annually, the Analytics page adds an AI performance panel with 7, 30 and 90-day views. It shows the engaged, resolved and handed-off funnel, CSAT for conversations the AI handled alone next to conversations your team replied in, and "Answers grounded in KB": the share of AI answers that cited at least one article. An answer drawn only from a file or snippet cites no article, so it counts against that percentage. "Where the AI hands off" lists the most common topics of handed-off conversations, each with a "Draft article" button that writes a draft from customers' questions into Knowledge → AI drafts for review. That button relies on the AI drafts feature (Talvo's Growth plan, $149 a month, and up, so it's included on Scale).

Talvo's AI performance dashboard: the engaged, resolved and handed-off funnel, grounding rate, CSAT, and the topics where the AI hands off.
Talvo's AI performance panel (Scale plan), shown with our demo workspace: the engaged, resolved and handed-off funnel, grounding rate, CSAT and the topics the AI hands off most.

Read that panel as a pointer, not the audit itself. A high grounding percentage says answers had a source, not that the source was current. On Talvo's Free, Starter ($49 a month) and Growth ($149 a month) plans there is no panel, and the routine above is the whole method. See the AI performance dashboard help article and how AI auto-resolve works for the details.

Choosing a tool? Ask every vendor this audit's questions: can I see the source, flag a bad answer, find every AI conversation? Our knowledge base chatbot guide covers the rest, and the customer service QA scorecard does this job for your team's own replies. For how AI customer service tools bill and hand off, see AI customer service software.

Frequently asked questions

How do you audit AI chatbot answers?

Read a fixed sample of AI conversations every week, grade each AI answer on a short rubric, and trace every bad answer to the content behind it. A workable sample for a small team is 10 random AI-resolved conversations, 5 hand-offs and 5 low-rated or reopened ones. Fix the article, file or hand-off rule that caused the answer, then ask the same question again to confirm the fix.

How many AI chatbot conversations should I review each week?

About 20 a week is a sample a small team will actually keep up, split into random, handed-off and low-rated slices. If the AI handled fewer than about 20 conversations that week, read all of them. Read more when a money or legal answer was wrong, after a price or policy change, or after you change the AI's settings.

Can a small sample tell me how accurate my AI agent is?

Only roughly, and only from the random part of the sample. Finding no errors in 20 random answers still leaves the true error rate as high as about 14% at 95% confidence, and about 7% after 40. Small samples are good at catching frequent problems, so keep a rolling four-week total rather than judging one week.

Why does my AI chatbot give wrong answers?

Usually because of the content it reads, not the model. The common causes are a missing article, two articles that contradict each other, an article that is out of date, a private document the AI shouldn't be using, or a question that needs a person. Each has a specific fix, from editing or unpublishing an article to adding the topic to an always-hand-off list.

Should I switch AI models if my chatbot answers wrong?

Not first. A different model reading the same contradictory or outdated articles will repeat the same mistakes. Trace the wrong answer to its source and fix the content; change the model only if answers stay wrong when the content is clearly right.

Is customer satisfaction a good measure of AI answer quality?

Not on its own. Customers usually rate the whole conversation, often before they discover an answer was wrong, so a good rating can sit on a bad answer. Use low ratings to choose which conversations to read, and grade the answers yourself.

Can I see which source Talvo's AI used for an answer?

For help articles, yes: each AI reply shows the titles of the articles it cited, in the inbox and in the widget. Answers drawn only from private knowledge files or snippets show no source, and a teammate can't see which file or snippet was used. Check those by hand when an AI answer has no source chip.

Sources

Other products' prices and features change. We checked these pages; if something is out of date, tell us and we'll fix it.

Let the agent take the first shift.

Three steps, and the agent is answering.

  1. Add the messengerOne script tag, just before </body>.Install the chat widget
  2. Import your help centerCrawl your own site, or bring your articles over from your old help desk.Import by crawling your site
  3. Invite your teamOne flat price for the workspace, up to your plan's teammate cap.Invite your team
Free forever plan · no credit card
<!-- Settings → Chat widget → copy your snippet -->
<script async src="https://talvo.chat/_api/widget/loader?app=your-app-id"></script>
</body>