How to Build an AI Knowledge Base from Your Documents and Conversations

Short answer

Most internal knowledge bases index documents and stop. That covers what your organisation decided, and misses why, and misses everything that changed since.

The rest lives in conversations: Slack threads, community forums, support tickets, meeting notes. Teams know this, which is why "just index Slack too" is such a common instinct. It is also why so many of these projects get worse when they try.

Documents and conversations fail in opposite directions. A document is authored, versioned and intended to be read later, so its main risk is being out of date. A conversation is none of those things, so its main risk is that most of it is wrong on purpose: guesses, half-answers and questions nobody replied to. Feed both into the same pipeline under the same rules and the conversations will drown the documents.

The rule that makes this work is short. Index conversations that reached an answer, and nothing else.

kapa.ai is an LLM-powered RAG platform purpose-built for technical documentation, and the specifics below are documentation of how one system handles this rather than a recommendation.

The two halves

A document tells you the state of the world as somebody intended to record it. An architecture decision record, a runbook, an API reference. Someone chose to write it down, which is itself a quality signal.

A conversation tells you the state of the world as it was actually encountered. The workaround nobody documented, the reason a decision was reversed, the config that only works on one cloud. Nobody chose to write it down. It exists because somebody had a problem at 4pm on a Tuesday.

You need both, and the reason teams index only documents is not that they do not know this. It is that indexing conversations naively makes retrieval measurably worse, people notice, and the project retreats to the safe half.

Why conversations break retrieval

Four differences, and they compound.

Most messages are not answers

In any support channel, the ratio of questions to answers is poor, and the ratio of answers to correct answers is worse. A retrieval system cannot tell them apart. It sees relevant text and returns it.

This is the same failure that makes unresolved support tickets dangerous, and it has the same fix: only ingest conversations that concluded. In practice that means requiring at least one reply, and preferably a reply from someone you trust.

kapa's Slack connector ingests threads, meaning a root message together with its replies. A standalone message with no reply is not a thread and is not ingested. That sounds like a limitation until you consider what a standalone message in a support channel usually is: an unanswered question. Indexing unanswered questions as though they were knowledge is how a knowledge base starts confidently returning the problem instead of the solution.

Nobody in the conversation is authoritative by default

A documentation page carries implicit authority because somebody reviewed and published it. A Slack message carries none. A confident wrong answer from a new user and a correct answer from your principal engineer are formatted identically.

The practical fix is to declare authority explicitly. Both the Slack and Discord connectors support a trusted users list, plus an option to only ingest threads where one of those users replied. That single setting does more for quality than any amount of retrieval tuning afterwards, because it changes what is in the index rather than how the index is searched.

Conversations have no canonical version

When documentation changes, the old version is replaced. When a Slack answer becomes wrong, it just sits there. The thread from eighteen months ago explaining the old authentication flow is still perfectly retrievable, still well written, and now actively harmful.

Both connectors expose a thread age filter for this. Discord defaults to the last six months. The right window depends on how fast your product moves, and it should be shorter than the window you would use for documentation.

Different systems can only be kept fresh in different ways

This is the part most architecture write-ups skip. Freshness is not one property of your knowledge base, it is a separate negotiation with every system you connect.

kapa's ingestion documentation is unusually candid about the consequences. Some APIs can report every change since a timestamp. Many cannot report deletions at all. Slack offers no way to ask what changed and enforces tight rate limits, so rather than pretend otherwise, the connector re-scans the last seven days of threads on a rolling basis.

The same document explains why deletions run as their own, less frequent pass: most upstream systems cannot be asked what was removed, so detecting it means comparing the whole upstream state against the index, which is expensive. New and updated content is cheap to query and gets picked up every few minutes. Deletion is not, and does not.

If you are building this yourself, that asymmetry is most of the work, and it is per connector.

When a document and a thread disagree

They will, and you should decide the policy before it happens rather than after a customer sees the wrong answer.

Three approaches, in increasing order of effort.

Scope by surface. Keep conversations out of anything customer-facing and let them serve internal assistants only. Source groups exist for exactly this: one knowledge base, one public assistant restricted to published documentation, one internal assistant that can also see threads and tickets. This is the cheapest correct answer for most teams.

Prefer recency over relevance where they conflict. A thread from last week describing behaviour that contradicts a docs page from last year is usually right, and the docs page is the thing that needs fixing. Treat the contradiction as a documentation bug rather than a retrieval bug.

Make the assistant decline. A system designed to say it does not know, rather than to pick a winner between contradictory sources, fails safely. In technical documentation a confidently wrong answer costs more than no answer. We covered the mechanics of this in how to make an AI assistant give source-backed answers.

A configuration that works

For a technical product, this is a defensible starting point.

The pattern across every row is the same. Documents are excluded when unfinished. Conversations are excluded when unanswered. Both are excluded when old.

Start narrower than feels comfortable. Widening filters after you can see what retrieval does is straightforward. Working out which of 200,000 indexed messages caused a bad answer is not.

How to test it

The failure this design prevents is subtle: adding conversations improves answers on questions documentation could not handle, while quietly degrading answers on questions it handled fine. A single average score hides that completely.

1. Build two question sets. Thirty questions your documentation answers well, and ten whose answer exists only in a thread or a ticket.

2. Run both sets before adding conversations, and keep the output.

3. Add conversational sources and run both again. The first set is your regression test, the second is your business case. You need both to move in the right direction.

4. Ask a question whose answer changed. Somewhere there is a docs page that is out of date and a thread that corrects it. See which one wins, and decide whether you are happy with that.

5. Measure refusals separately from correctness. A system that answers more by guessing from half-finished threads scores better and performs worse.

6. Re-run after two weeks without changing anything. Conversation sources drift in a way documentation does not, and this is the only way to see it.

The longer methodology, including how to build the test set and where LLM judges go wrong, is in how to evaluate RAG systems without relying on vibes. It is written to be used against us as much as for us.

The short version

Documents record decisions, conversations record reality, and you need both. The difference is that documents arrive pre-filtered by the act of someone choosing to write them, and conversations do not. So filter them yourself: threads with a reply, from people you trust, recent enough to still be true, scoped away from surfaces where a wrong answer is expensive.

If you are building this in-house, budget for the per-connector freshness work rather than the vector database. That is where the time actually goes. Building a RAG pipeline covers the build-versus-buy version of that decision.

This guide is written by kapa.ai, which makes one of the tools described. Kapa.ai is an LLM-powered RAG platform purpose-built for technical documentation, used in production by 200+ technical companies. Connector details are accurate as of September 2026.