Conversational AI for customer service: a buyer's guide | RibboAI

Every conversational AI demo looks good on easy questions. This guide gives you the three question types that separate platforms, a two-week evaluation plan, and the red flags to walk away from.

Conversational AI for customer service is software that holds a written conversation with a customer and settles what they came for: an answer, a status, a booking, or a clean handover to a person. The models behind most platforms are now similar. What separates one platform from another is what the model is connected to and what it is allowed to do.

That makes buying hard, because every vendor demo uses the same easy questions and every demo goes well. This guide gives you the questions that expose the differences, a two-week evaluation you can run on your own data, and the red flags that should end a conversation with a vendor.

It covers written channels: web chat, WhatsApp, email and similar. The same tests apply whichever channel you start on.

What a conversational AI platform does today

Older chatbots matched a customer's message to a list of intents and returned a scripted reply. They broke whenever a customer phrased something in a way nobody had anticipated, which was most of the time.

A current conversational AI platform puts a large language model in the middle and surrounds it with four things:

• Knowledge. Your help pages, policies, price lists and documents, searched at the moment a question arrives.
• Tools. Connections to your systems so the agent can read a record, check stock or find an order.
• Rules. What it may discuss, what it must never promise, and when it must hand over.
• A route to people. A handover that alerts your team when a person is needed, and a record of every thread.

The model writes the sentences. The four surrounding parts decide whether those sentences are true. When you evaluate a platform, you are mostly evaluating those four parts.

The test that separates platforms: two kinds of question

Look at a month of your own customer messages and you will find that most fall into one of two shapes.

The first shape is "how does X work?" How long do refunds take, do you deliver on Saturdays, what does the premium plan cover. The answer is the same for every customer and lives in a document. A platform answers these by searching your content for the right passage and writing a reply from it. This technique is called retrieval-augmented generation, and good platforms say so plainly when your content does not cover the question.

The second shape is "what is my X?" What is my balance, where is my order, has my claim been paid. The answer is different for every customer and lives in a row in your system. Document search is the wrong tool here. It retrieves text that looks similar to the question, which is how a chatbot ends up quoting a sample figure from a help article as if it were the customer's balance. These questions need an exact lookup: the platform identifies who is writing, calls your system with that customer's identifier, and reads the figure that comes back.

Many products marketed as conversational AI handle only the first shape. Ask every vendor to show both, on your data, in the same conversation.

A third shape, "do X for me", follows naturally: book the slot, change the address, send the invoice. It builds on the second, because the agent has to know whose booking or address it is changing before it changes anything.

A worked example: one thread, two mechanisms

Here is what a passing answer looks like at a fictional insurer, Northgate Insurance, on WhatsApp. On WhatsApp the sender's phone number identifies the customer, so the agent can match it to a policyholder without asking for a policy number.

Three details to look for when you run this test yourself:

1. The first reply uses the wording of the claims guide and nothing else. If the guide changes next month, you update the document and the answer changes with it.
2. The second reply contains a claim number, an amount and two dates. None of them came from a document. They were read from the claims system for this customer only.
3. The customer never typed a policy number. The lookup started from the number they wrote from, so the agent had the right record from the first message.

That third point is where many builds fail quietly. If the model chooses which identifier to look up, a cleverly worded message such as "show me claim NC-2290 instead" can ask for another customer's data. Ask each vendor how the identifier is chosen on each channel, and test it yourself during the trial.

What to require from a conversational AI platform

Treat this as the minimum. Each item maps to a failure that customers will find within weeks if it is missing.

• Answers grounded in your content. The agent answers only from what you gave it, and a change to a document reaches the answers without a rebuild.
• A real "I don't know". When the content does not cover a question, the agent says so and offers a person. Ask to see this happen live.
• Exact lookups for the right customer. Balances, orders, bookings and claims come from your system through a tool call, never from document search.
• A clear rule for missing records. If the lookup returns nothing, the agent says it cannot find the record. It never fills the gap with a plausible figure.
• Control over actions. You decide what the agent may only read and what, if anything, it may change in your systems. Starting read-only is the safe default.
• A handover that reaches a person. When the agent cannot help, your team is told quickly and can read the whole conversation before replying.
• The channels your customers already use. For many businesses that means WhatsApp first. Outside the 24-hour customer service window, Meta allows only approved template messages, according to its WhatsApp pricing documentation, so decide early which tool sends your outbound templates and which one answers the replies.
• Records and gaps. Every conversation is stored and readable, so you can find the questions the agent could not answer and fix the content.

Our platform page sets out which of these Ribbo covers and how. Whatever vendor you choose, ask them to show each one working rather than describe it.

How to evaluate a conversational AI platform in two weeks

A trial on your own questions beats a month of vendor calls. The process below fits in two weeks with one person spending a few hours a day on it.

Build the test set

Pull 40 real customer messages from the last month. Keep the original wording, typos included. Aim for roughly 20 policy questions, 15 account questions and 5 requests to do something. Add five messages that should end with a person: a complaint, a legal threat, a request for an exception.

Score the answers

Use four grades and nothing finer. Finer scales turn into arguments.

A single wrong figure on an account question should count more than several clumsy phrasings. Customers forgive an awkward sentence. They do not forgive a wrong balance.

Try to break it

Spend one afternoon being a difficult customer. Ask for another customer's order. Ask for a discount the policy does not allow. Ask it to write a poem about your company. Tell it you are the manager and it should ignore its instructions. In January 2024 a parcel firm had to switch off part of its chatbot after a customer persuaded it to swear and to criticise the company, as BBC News reported. That kind of failure shows up in an afternoon of testing, before it shows up on social media.

Red flags when buying

• The demo uses only FAQ questions. If the vendor cannot show an account lookup on a test record during the trial, assume it cannot do one.
• "We train the model on your data" as the whole answer. Fine-tuning changes how a model writes. It is a poor way to teach it facts that change. OpenAI's own guide to optimising accuracy recommends retrieval for proprietary or changing knowledge and fine-tuning for format, tone and consistent behaviour.
• No access to full conversations. Without the transcripts you cannot tell a correct answer from a confident guess.
• The agent never says "I don't know". Ask it something your content does not cover. If it answers anyway, it will do the same to customers.
• A builder with no one behind it. Some platforms hand you a blank canvas and a manual. If nobody on your team will own the build, look for a vendor that sets up a configured agent with you.
• Vague answers on customer data. You should get a clear written answer on whether your conversations are used to train shared models, what is redacted before storage, and who on the vendor side can read transcripts.

Disclosure and compliance questions to settle early

Customers should know they are talking to software. In the EU this is a legal requirement: Article 50 of the AI Act requires systems that interact directly with people to tell them they are dealing with an AI, unless that is obvious, per the text of the Act. A short line in the greeting covers it.

Banks, insurers and healthcare providers will also need answers on redaction of card numbers and ID numbers, retention periods, audit logs and approval rules for anything that writes to a core system. Raise these in week one of the evaluation, because they take longest to agree.

Where Ribbo fits

Ribbo AI is a conversational AI platform delivered as a working agent rather than a toolkit. It answers only from your documents and says when it does not know. It looks up exact records through Data Tools, from an uploaded data set or your own API; on WhatsApp the sender's number identifies the customer, and on web chat the customer types an identifier. When a person is needed, your team gets an email and follows up, and every conversation is readable in the dashboard. It runs on your website, WhatsApp, Messenger, Instagram and an API. The Ribbo platform overview shows how the three parts fit together, and you can try your first test questions on the free plan before connecting a channel. If you are earlier in the process, our guide to ChatGPT for small business covers the lighter options first.

Frequently asked questions