You have been on the receiving end of a bad support bot. It greeted you with a cheerful name, offered four buttons that were not your problem, misunderstood your question twice, and then told you to email support. You closed the tab and thought less of the company.
So when someone proposes putting one on your own site, your reaction is reasonable: not that. Your customers are worth more than a widget that makes them angrier than they were before they clicked.
The good news is that the bad bots are bad for specific, avoidable reasons. An AI customer support chatbot built on a few strict rules behaves differently: it answers only from material you wrote, it shows the customer where the answer came from, it says “I do not know” when it does not, and it puts a person one click away. This article lays out those rules, the review loop that keeps the bot honest over time, and what it costs to run.
What this actually is
A support chatbot is a text conversation on your website, in your app, or in a messaging channel, where the customer types a question and software answers it. The older kind followed a decision tree: click a button, get a canned reply. The current kind uses a language model (software that reads and writes natural text, the family behind the current Claude and GPT-class models) so the customer can ask in their own words.
The important design choice is where the answers come from. Left alone, a language model will answer any question from its general training, confidently and sometimes wrongly. A well-built support bot is not allowed to do that. Instead, every time a customer asks something, the system first searches your own material (help articles, policies, product details) for the relevant passages, then hands those passages to the model and tells it to answer only from them. This pattern is called retrieval-augmented generation, or RAG, and it is explained from scratch in What Is RAG? Retrieval-Augmented Generation for Business Owners.
The everyday analogy is a new support hire on their first week with a binder. You do not want them improvising answers about your refund policy from memory of some other company. You want them to open the binder, find the page, read it to the customer, and say “let me get my manager” when the page is not there. The bot is that hire, the binder is your knowledge base, and the rules below are the training.
The rules for an AI customer support chatbot customers accept
1. Answer only from your own material, and say so
The bot’s instructions must say, in effect: answer using only the provided passages; if the passages do not contain the answer, say you do not have that information and offer a person. This is the rule that prevents the bot from inventing a return window you do not offer or a feature you never built. In the systems we build, this is the first line of the instructions, and it is tested with questions the material deliberately does not cover.
The corollary is that the bot is only as good as the material. If your help content is thin, out of date, or lives in someone’s head, the bot will be thin too. The knowledge base is the product, and Knowledge Base for an AI Chatbot: What to Write and How covers how to build one the bot can actually use.
2. Show the source under every answer
Each answer should link to the article or policy it was drawn from. “Refunds are accepted within 30 days of delivery for unused items. Source: Returns and refunds policy.” This does three things. The customer can check the answer. Your team can spot which article produced a wrong answer and fix it. And the bot cannot quietly drift into unsupported claims, because an answer with no source stands out immediately in review.
Technically this means the retrieval step keeps track of which passages it found, and the model is told to attach them. It is a small amount of work and it is the feature that makes the bot trustworthy.
3. Admit ignorance in plain words
“I do not have information about that. Would you like me to connect you with our support team?” That sentence, said promptly, is worth more than any clever answer. Customers forgive a bot that does not know. They do not forgive a bot that pretends. Make the admission fast (no three rounds of “could you rephrase that”) and make it lead directly to a person.
Log every one of these admissions. They are your list of gaps in the material, and a week of them tells you exactly what to write next.
4. Put a person one click away, always
Every screen of the conversation should have a visible way to reach a human. Not buried in a menu, not after the bot has tried three times. A button or a typed “talk to a person” ends the bot’s involvement immediately. The person receives the full conversation plus a two-line summary, so the customer does not repeat themselves. The rules for when the bot must hand off on its own (anger, money, legal questions, uncertainty) and how the handoff works technically are in AI Chatbot Escalation to Human: When and How the Handoff Works.
A handoff that lands in a queue nobody watches is the second-worst experience after a wrong answer. Decide where handoffs go (a shared inbox, a help desk tool like Zendesk or Intercom, a Slack channel) and who owns them, before the bot goes live.
5. Keep the tone plain and short
Support customers want the answer, not a personality. Instruct the bot to answer in two to four sentences, in plain language, with no cheerfulness padding. No “Great question.” No exclamation marks. No apologizing three times. If your brand is casual, the bot can be casual, but brief, not chatty.
Also decide what the bot will not discuss: competitors, pricing that is negotiated, anything medical or legal, and anything about individual accounts it cannot see. A written list of off-limits topics with a standard redirect for each keeps the bot out of trouble.
6. Do not let it act on accounts without verification
A support bot that can look up an order, change an address, or issue a refund is powerful and dangerous. Any action beyond answering questions needs the customer verified (logged in, or a code sent to their email or phone) and a list of allowed actions with limits. “Look up order status for a verified customer” is fine. “Issue refunds up to $50, log every one, hand anything larger to a person” is fine with review. “Do whatever the customer asks” is how a bot gets talked into giving away money. Allowed-action lists and human approval for anything irreversible are covered in AI Hallucination Guardrails for Business Applications.
7. Disclose that it is a bot, and name what it can do
The first message should say it is an automated assistant, list the two or three things it is good at, and mention that a person is available. “Hi, I am the automated assistant. I can answer questions about orders, shipping and returns. For anything else, or if you would rather talk to a person, just say so.” Setting expectations up front prevents most of the frustration that comes from customers asking things the bot was never meant to handle.
8. Review the conversations every week
Read a sample of twenty to fifty conversations each week. Look for wrong answers, answers with no source, questions the bot could not answer, customers who asked for a person, and customers who gave up. Each finding becomes either a fix to the material or a fix to the instructions. Track a few plain numbers over time: what share of conversations ended with a handoff, what share ended with the customer apparently satisfied, and the top ten unanswered questions.
This is not optional. An AI customer support chatbot without a review loop degrades as your products and policies change and the material falls behind. With one, it gets better every week, because every gap it exposes gets filled.
9. Test it before launch with real questions, including hostile ones
Before the bot meets a customer, collect fifty to a hundred real questions from your support inbox, including the awkward ones, and run them through. Check every answer against the source. Then try to break it: ask for a refund it should not give, ask it to ignore its instructions, type something abusive. It should decline, redirect, or hand off cleanly every time. Keep this question set and rerun it whenever the material or the instructions change.
Picture a business like this one
The business below is a composite of the kind of company that writes to us, not a client. The numbers describe the shape of the problem, not a case study.
Picture a business like this one: an online retailer of specialty kitchen equipment with around 40 staff and a four-person support team. They get roughly 200 support emails and chats a day, and more than half are the same twelve questions: where is my order, how do I return this, does this fit that, what is the warranty. The team spends its mornings on those and its afternoons on the genuinely hard cases, and response times on the hard cases are poor.
What gets built:
- A knowledge base of about 80 short articles rewritten from the existing help pages and the team’s own canned replies, each one answering exactly one question.
- A chat widget on the site and in the order-status emails, opening with a disclosure and the three things it can do.
- Retrieval over the knowledge base, with the model instructed to answer only from what is found and to cite the article under every answer.
- Order lookup for customers who verify with the email on the order, read-only. No refunds, no address changes; those hand off with the conversation attached to the help desk queue.
- A handoff button on every message, and automatic handoff on any mention of a complaint, a chargeback, or a damaged item.
- A weekly review of fifty conversations and a running list of the top unanswered questions, owned by one named member of the support team.
What changes: the twelve repetitive questions mostly stop reaching the team. The hard cases get answered the same day. Customers checking order status at 10pm get a tracking link. And the top-unanswered list produces three new articles a week for the first two months, after which the bot’s admissions fall to a trickle.
What it costs to run
The model usage is priced per token, a token being roughly three quarters of a word of text going in or out of the model. A support conversation of a few turns, including the retrieved passages, might use a few thousand tokens. With the current mid-tier models from OpenAI, Anthropic or Google, that works out to well under a cent per conversation for most businesses. Check the current pricing pages; rates change and differ by model. A few thousand conversations a month is typically in the tens of dollars.
The retrieval side needs somewhere to store the searchable version of your material. Pinecone has a free tier and paid plans; pgvector inside a Postgres database on a $10 to $30 a month server is the cheaper route for a small knowledge base. The chat widget can be self-built or come from a help desk product (Intercom and Zendesk both sell AI answering as an add-on, priced per resolution or per seat; check their current pages). Add a small server if custom logic runs anywhere. The real ongoing cost is the weekly review and the writing that follows from it: a few hours a week of a support person’s time.
The mistakes we see most
-
Launching with the existing help center as-is. It was written for browsing, not for answering. Long pages with six topics each retrieve badly. Rewrite into one-question articles first.
-
No source under the answer. The customer cannot check, the team cannot trace wrong answers, and the bot’s drift goes unnoticed.
-
The handoff goes nowhere. A “contact us” form that emails a shared inbox nobody owns. Decide the queue and the owner before launch.
-
Letting it take actions without verification or limits. The first customer who talks it into a refund is a story you will hear about.
-
No weekly review. The bot is fine for two months, then the return policy changes, the material does not, and the bot spends a quarter quoting the old rule.
When to bring in help
If your help content is already decent and you use Intercom, Zendesk or a similar help desk, their built-in AI answering will get you a reasonable bot in a day or two with no developer. Turn on source citations, set up the handoff, and start the weekly review. For many businesses that is enough.
You need a developer when the bot has to read from your own systems (order status, account details, inventory), when you want it on your own site and in your own app rather than inside a help desk vendor’s widget, when you need control over which model is used and where your data goes, or when the actions it can take need proper verification and limits.
Levelbrook builds AI customer support chatbots to these rules, fixed price from a written scope, with the knowledge base, the retrieval and the handoff running in accounts you own. The form below is how a conversation starts.