How an AI assistant answers from your documents (and why it sometimes gets it wrong)
An assistant that answers from your own documents is only as good as those documents, and how it finds them. Here is how it works in plain language, and the fixes that make the biggest difference.
By Admin
"Train the AI on our documents" is one of the most common requests businesses make. In most cases, nothing is being trained at all. The assistant looks up the relevant parts of your documents when a question arrives, and writes its answer from what it finds.
Understanding that one idea explains almost every quality problem, and shows you what to fix.
How it works, in four steps
- Prepare. Your documents are split into short passages, and each keeps details about where it came from: the file, the date, the product it covers.
- Index. Each passage is turned into a numerical fingerprint of its meaning, and stored in a search index. Good systems also keep ordinary keyword search alongside it, because exact terms such as product codes matter.
- Retrieve. When someone asks a question, the system finds the passages that best match it.
- Answer. The question and those passages go to the language model, with an instruction to answer only from them and to say which source it used.
In short:
question -> search your documents -> best passages -> model writes an answer with sources
The model is not "remembering" your company. It is reading a handful of passages, every time, and doing its best with them. If those passages are wrong, missing or contradictory, so is the answer.
Why answers go wrong
| What you see | The usual cause | The fix |
|---|---|---|
| It quoted an old price | An outdated document is still in the index | Remove or expire old versions, and keep dates on documents |
| It missed an answer that is clearly in the documents | The passage was split badly, or the wording differs from how customers ask | Better splitting, keyword plus meaning search, and FAQs written in customers' own words |
| It made something up | Nothing relevant was found, and it answered anyway | Require a source for every answer, and let it say "I do not know" |
| It contradicts itself | Two documents disagree | Resolve the conflict at the source |
| It shows something it should not | Permissions were not applied when searching | Check access rights before retrieval, not after |
Fix your documents first
The cheapest improvement is rarely a better model. It is better documents.
- Keep one source of truth for each topic, instead of five slightly different copies.
- Give documents clear titles and dates.
- Write in plain language. If a page would confuse a new employee, it will confuse the assistant too.
- Turn tables and screenshots into text where the answer lives inside them.
This alone often fixes more than people expect, and it helps the human team as well.
Make it show its sources
Ask the assistant to name the document or page each answer came from. Citations let users check for themselves and let you find out why an answer was wrong. If the assistant cannot point to a source, it should say so rather than guess.
Keep private things private
Not everyone should see everything. The assistant should only retrieve what the person asking is allowed to read. Do not put every document in one shared index and hope for the best. Decide, too, what gets stored in conversation logs, who can read them and for how long.
Test it with real questions
Collect fifty to a hundred real questions from customers or staff, together with the correct answers. Run them again after every change to the documents, the settings or the model. Look most carefully for confident wrong answers, which do more damage than a missed answer. We describe the wider approach in what an AI support agent should and should not handle.
Keep it up to date
Documents change, and the index has to change with them. Re-index automatically when a document is updated, and give someone ownership of the content. An assistant answering from a stale handbook is worse than no assistant.
Want an assistant that answers from your own documents? Send us a sample of them and we will tell you how well it would work.
Related service: WhatsApp & AI Bots
Have a process eating your team's time?
Tell us about it and we'll say honestly whether automation is the right fix.
Book a discovery call
