Chatbots that actually help: lessons from 70% automation

A conversation that ends without a handover is not necessarily resolved. The customer may have left because the answer was poor. Define the eligible request types, the evidence of resolution and the period over which you measure them. Keep excluded requests visible so that a favourable percentage cannot hide a difficult part of the workload.

If seventy of one hundred eligible requests are genuinely completed without staff action, the rate is 70%. It says nothing by itself about correctness, customer satisfaction or the effort spent on the other thirty. Track those measures separately.

Start with recurring questions that have clear answers: service coverage, preparation instructions or a documented process. Assign an owner to each source and remove contradictory versions. The assistant should retrieve the relevant material and be able to show where an answer came from when that helps the customer.

Do not use a knowledge base as a substitute for authenticated account access. General policy can be public; a customer's bookings, documents or payment details need a separate permission check. Server-side tools should return only the data that the current user is allowed to see.

Offer a clear way to ask for a person. When the assistant cannot resolve the request, hand over a concise summary, relevant answers already collected and the reason for escalation. Avoid making the customer repeat the whole story. Set realistic expectations about response times, especially outside staffed hours.

The assistant should not imply that a booking, refund or account change succeeded until the system confirms it. If an action has an uncertain result, show that state and check it before retrying. Confident wording is not evidence that the underlying operation completed.

Create a test set from representative request categories without copying unnecessary personal data. Include ambiguous wording, out-of-date information, unsupported requests and attempts to obtain another user's details. Re-run the set when instructions, tools or source material change.

Review a sample of apparently successful conversations, not just the escalations. Look for misleading answers that avoided handover but failed the user. Expand automation only where the evidence supports it. A lower percentage with dependable outcomes can be more valuable than a high percentage that leaves customers frustrated.