AI chatbot knowledge base: which content belongs in it and how to keep it current

On Friday evening a customer asks whether the business is open on Saturdays. The team has three answers: the website says “Mon to Fri”, a notice on the door promises “Sat by appointment”, and the colleague at reception knows that in summer it is closed on Saturdays anyway. All three consider their answer correct. This scenario is invented; the pattern is not.

This is exactly where an AI chatbot comes in, and exactly where it is decided whether it helps or harms. A language model does not know your business. It knows language, general relationships and plausible phrasings. If you give it nothing else, it answers with what usually applies in such cases. That sounds convincing and can still be wrong.

The knowledge base is therefore not an add-on but the foundation. It determines what the bot may say on behalf of your business. Neglect it and you get a polite machine with invented knowledge. Build it carefully and you get an assistant that passes on only what you have explicitly approved and that names the place where it is written.

This text is aimed at businesses facing exactly this decision: should an assistant give information on behalf of the house, and if so, on what basis? It explains how we would build a knowledge base, which trade-offs arise and where the limits lie. The examples are constructed so that nothing looks like a real project that never existed.

The core problem is called hallucination: the model produces text that fits the question without being backed by a source. The mechanism is mundane. A language model calculates which word is likely to come next. Likely is not the same as true. For “How much does an initial consultation cost?” there are countless plausible answers, and the model picks one when it has no concrete information.

Then there is the ageing problem. Even a knowledge base that is correct at first goes out of date. Prices change, an employee leaves, an offer expires, the practice closes between the years. A bot that repeats the state of spring seems just as sure in autumn as on day one. The customer cannot tell the difference, and the business often notices only when someone is standing at a locked door.

A third failure pattern concerns commitments. When someone asks “Can I cancel with you?”, it is not about information but about a binding statement. If the bot answers freely, it may formulate a rule that does not exist. A knowledge base therefore needs not only content but also boundaries: which topics does the bot answer, and for which does it hand over to a human?

Before anything is built technically, it is worth taking stock of the questions customers actually ask. The best sources for this are the inbox, notes from the phone, search queries on the website and the question of what the team answers for the fifth time in a week. These questions show which content the knowledge base has to carry. Not every piece of information that exists in the business belongs in it.

We would sort content by two criteria: how often is it asked, and how costly is a wrong answer? Opening hours are asked often, and a mistake is annoying but usually does little harm. Price ranges are asked frequently, and a mistake creates expectations that later have to be corrected. Contractual details are asked less often, but a mistake can have serious consequences. This determines how strict the approval has to be.

A counter-check is just as revealing: what should the bot explicitly not answer? Individual legal or health questions, binding prices for special cases, information about other customers. This boundary is part of the design and should be put in writing before the first line of content is entered. It later decides whether the bot hands over cleanly in critical moments.

Taking stock also includes a look at the form of the sources. A price list as a scanned image cannot be read by the bot, a table with merged cells is easily torn apart on import, and a photo of the notice is only good as a reminder. Such sources are converted into plain text beforehand, and it is exactly then that inconsistencies come to light that nobody had noticed before.

There are basically three ways to give a model business knowledge. The first is to write everything into the instruction placed in front of every request. This works for a handful of paragraphs and fails as soon as the material grows: the instruction becomes unwieldy, expensive on every request, and the model does not weight long texts reliably. For a business with ten pages of knowledge this may suffice; you should not build on it beyond that.

The second way is to retrain the model. That sounds thorough but suits business knowledge poorly. What has been learned is hard to correct when a price changes, and it cannot be shown where a statement comes from. Retraining is better suited to tone and format than to facts that change constantly. For opening hours it is the wrong lever.

The third way is called retrieval, in technical terms retrieval-augmented generation. For every question the bot first searches your own content, hands the passages it finds to the model and lets it answer only from them. If a price changes, you change one document, and the next answer is correct. In addition, every statement can be traced back to a source. This is our choice because it combines currency, traceability and control.

Retrieval is not a cure-all. If the search finds the wrong passage, the model reliably answers from the wrong passage. Quality therefore lies in two halves: in the search and in the instruction to answer only from what was found. Both halves have to be tested, and that is not a one-off task.

A knowledge base is not a dumping ground. Upload all PDFs, brochures and website texts and you get contradictory sources and a bot that quotes the old price list one time and the new one the next. Better is a small, deliberately built collection: one page per topic, short, unambiguous, with a date and a person responsible for it. Better forty maintained entries than four hundred unclear ones.

Writing style matters more than you might think. The search works by similarity of meaning, so entries worded the way customers ask help. An entry titled “Cancelling appointments” with the first sentence “Appointments can be cancelled free of charge up to 24 hours in advance” is found more easily than a paragraph from the terms and conditions that says the same in legal language. Excerpts from the terms stay as evidence; the plain-language version sits next to them.

Every entry carries metadata: topic, validity, date, contact person, visibility. The “valid until” note allows seasonal content to be hidden automatically. Visibility separates what the bot may say publicly from internal notes for the team. And prices are given as a range with conditions, not as a number without context, so that “from” does not accidentally become “exactly”.

Finally, what the bot does not know belongs in the knowledge base. An entry “Individual price quotes” with the content “The bot gives no information on this but refers to the team” is worth more than silence. It makes the boundary findable and ensures that the right reaction is chosen for related questions.

We resolve contradictions before entering content, not afterwards. If the website states different times than the price list, one person decides which version applies, and the other is corrected. A bot that knows two truths repeats one or the other at random. This reconciliation is uncomfortable but often uncovers errors that have also been sitting unnoticed on the website for months.

Technically, every request goes through four steps. First the content is prepared: texts are split into sections of a few sentences, and each section is translated into a series of numbers that represents its meaning. This translation, a so-called embedding, makes it possible to measure similarity. Two sentences with different words but the same meaning lie close together, such as “When are you open?” and “Opening hours”.

When a question arrives, it is translated in the same way, and the search returns the most similar sections. In practice we combine the meaning-based search with a classic keyword search, because it is more reliable for proper names, article numbers and technical terms. The best hits are ranked, and only hits above a minimum similarity are passed on. This threshold is the most important control for behaviour under uncertainty.

In the third step the language model receives the question, the sections found and an instruction: answer only on the basis of these sections, name the source, and if the sections do not answer the question, say so. In the fourth step a check verifies whether the answer is backed by the sections and blocks statements with prices or commitments that are not found there. This control stage costs a little time and saves a lot of trouble.

The most difficult quality of a chatbot is not knowledge but restraint. People forgive an honest statement that the bot doesn't know something far more readily than a wrong one they later have to deal with. Yet language models tend to seem helpful. This behaviour has to be steered deliberately: through the instruction, through the search threshold and through examples of good not-knowing.

A usable “I don't know” answer has three parts: it says clearly that the bot has nothing reliable on this. It offers the next step, such as a question to the team, a callback or the contact page. And it takes the question along, so the team knows what it was about. This third part turns the gap into a pointer for maintenance: questions without a hit are the most honest list of what is missing from the knowledge base.

Borderline cases are difficult, where the search finds something that fits only half. The question is “Is there a discount for clubs?”, the passage found talks about volume discounts. Here the bot must not extrapolate. We solve this by having the instruction distinguish between “answered”, “partly answered” and “not answered”, and for the middle level naming what was found without closing the gap.

Not-knowing also includes knowing one's own role. If someone asks for a commitment, a diagnosis or a legal question, the bot should not guess but name the responsible people. This is not a weakness but the reason why businesses can give such an assistant a window on their website at all.

A good knowledge base makes the bot traceable, not infallible. A realistic expectation is that the large majority of everyday questions are answered correctly from the approved content, that answers name their source and that the bot hands over when there are gaps. A figure for this can be given reliably only after a test phase with real questions, not before and not in general terms.

The benefit can be measured through figures close to the cause. These include the share of questions answered with a source, the share of handovers due to missing hits, spot checks in which a person checks answers against the source, and user feedback via thumbs. Only after that come measures such as fewer repeated enquiries in the inbox or shorter handling times, which depend on many other factors.

An example of interpretation: if the number of handovers due to missing hits rises in the first weeks, that is not a setback; the bot is showing where content is missing. If it then falls without the spot checks getting worse, coverage is growing. If it falls while the spot checks get worse, the bot is probably answering too liberally, and the threshold must become stricter.

Equally clear is what cannot be promised: no knowledge base guarantees error-free answers, especially for ambiguously worded questions or contradictory sources. A particular relief for the team also cannot be quantified in advance. It depends on how many enquiries your business actually receives, how standardised they are and how consistently maintenance is done.

The most common reason chatbots disappoint after a few months is not technology but missing responsibility. The rollout was a project; maintenance belongs to nobody. That is why we define from the start who is responsible for which area: prices with management, opening hours at reception, service descriptions with the specialist. Every entry has a name and a review date.

The rhythm has three levels. Event-driven, changes are made immediately when something changes: new price list, closure for holidays, a public holiday. Regularly, for example monthly, someone reviews the list of unanswered questions and negative feedback and adds entries. Twice a year the team goes through the entire knowledge base and deletes what no longer applies. Expiry dates help by automatically dropping entries with expired validity from the answers.

A simple way to make changes is important. If adjusting an entry requires coordination with the agency, it gets postponed. We therefore build an interface in which authorised people edit and approve entries, and we keep a history. With every change a small set of test questions runs along, so that a new wording does not silently shift the answers on other topics.

The knowledge base thus becomes a living document that is useful beyond the bot alone. Many businesses find that they are writing down for the first time what applies to cancellation, directions or payment. That also helps new employees and the website. This side effect is real but no substitute for the maintenance effort, which remains.

An AI chatbot pays off for businesses that answer similar questions often and whose answers can be captured in clear statements. Its quality depends on the knowledge base: on a few well-maintained entries, on a search that reliably finds the right one, on the source shown under every answer, and on a bot that admits gaps. The model behind it is interchangeable; the care that goes in beforehand is not.

The next step is small and doesn't need a bot yet. Write down the twenty questions your team has answered most often in recent weeks, and check whether the answers live in one single, up-to-date place. If that place is missing, that is the real task, and it is worthwhile regardless of any chatbot. We are happy to look together at how much of it already exists.

Honesty is part of this, too: for very small businesses with a few stable questions, a well-built FAQ page is enough. A chatbot plays to its strengths when questions are phrased in many different ways, when answers are expected around the clock, and when someone takes on the upkeep. If that last point is missing, you should postpone the launch.

If you do start, start small: with the five to ten most important topics, a supervised test phase, and a willingness to improve things based on real questions. The knowledge base then grows with demand instead of claiming completeness up front. That is slower than a large import, but the answers stay verifiable, and the team keeps track of what the bot says in the company's name.