6 common mistakes when building an AI Assistant in-house

SUMMARY

Building an AI assistant over your own documentation looks simple: connect an LLM to your docs and ship. The common mistakes are treating it as just RAG plus an API call, chasing a better model instead of fixing retrieval, shipping without evaluation, letting the index go stale, skipping an "I don't know" guardrail, and underestimating permanent maintenance. Together they are why roughly 70% of in-house builds never reach production. The fix for most teams is to buy a purpose-built platform like kapa.ai that has already solved these problems, rather than reinventing them.

A working AI assistant demo takes an afternoon. A production one that your users actually trust takes months, and most teams do not make it.

We have spent since early 2023 building kapa.ai to turn technical documentation into an AI assistant that works, and along the way we have watched dozens of teams attempt the internal build. Many of them became customers after the first attempt did not pan out. The failures are remarkably consistent. Below are the mistakes that turn a promising prototype into a stalled project, and why each one is harder to solve than it looks.

Here are the common mistakes that make an in-house build so hard:

1. Assuming it is just RAG plus an API call

2. Chasing a better model when retrieval is the problem

3. Shipping without evaluation

4. Letting your index go stale

5. Skipping the "I don't know" guardrail

6. Signing up for maintenance you will regret

Let's dig into each one.

Mistake 1: Assuming it is just RAG plus an API call

The pitch always sounds reasonable: "We already have the docs, we have engineers who know LLMs, and there are open-source RAG frameworks everywhere. How hard can it be?"

The demo really is that easy, and that is the trap. Connecting an LLM to a vector store and getting a plausible answer is the tip of an iceberg. Beneath the surface sit roughly 22 components you only discover you need once real users start asking real questions: data connectors for every source, chunking tuned per source type, hybrid retrieval and re-ranking, citation logic, evaluation, analytics, source freshness, LLM failover, PII handling, and SOC 2 controls, to name a few.

This is why the same story repeats. A team gets about 70% of the way to production, then finds the last 30% is where all the hard engineering lives. Industry-wide, roughly 70% of these in-house AI projects never reach production at all. The mistake is not building a prototype. It is assuming the prototype is 90% of the work when it is closer to 10%.

Mistake 2: Chasing a better model when retrieval is the problem

When answers come back wrong, the instinct is to reach for a bigger model or a higher-ranked embedding model. It feels productive. It rarely helps.

Once you have a reasonable balance of retrieval and model quality, swapping the LLM delivers marginal gains at best. Accuracy in a documentation assistant is set by retrieval, not by the model: if the system fetches the wrong passage, the answer is wrong no matter how capable the model is. The real work lives upstream, in data preprocessing, deliberate chunking, query understanding and routing, and hybrid retrieval that combines multiple retrievers. That work is unglamorous and easy to skip, which is exactly why so many builds plateau at mediocre and stay there while the team keeps swapping models hoping for a fix that never comes.

Mistake 3: Shipping without evaluation

Ask a team how they know their assistant is good and you will often hear some version of "we read a few answers and they looked right." That is a vibe check, and it does not scale, is not reproducible, and hides regressions.

Without a real evaluation harness you are flying blind. You cannot tell whether a new chunking strategy actually helped or whether upgrading the model made citations worse. One enterprise software team we spoke with burned six months and still could not get their hallucination rate below 7%, because they had no systematic way to measure and drive it down. And here is the part that is genuinely hard to replicate in-house: accuracy follows data volume. A single internal deployment produces a few thousand questions a month, which is nowhere near enough signal to A/B test frontier models or retrieval changes with confidence. By the time you have gathered enough data, three new models have shipped.

Mistake 4: Letting your index go stale

Documentation changes constantly. An assistant that answers from a snapshot taken three months ago will confidently mix old and new information, and users notice fast.

Keeping an index fresh is a real system, not a cron job. You need to detect when each source changes, re-ingest it, re-chunk and re-embed it, and invalidate stale answers, and you need to do that across every source type: a docs site, GitHub, Notion, Confluence, Zendesk, each with its own quirks. Skipping this is one of the most common and most invisible failures. One Fortune 500 team assigned a single engineer part-time to their RAG system, and a year later it had not been updated once. The assistant did not break loudly. It just quietly rotted until nobody trusted it.

Mistake 5: Skipping the "I don't know" guardrail

It is better to have no AI assistant than one that makes things up. A confident wrong answer about a parameter, a version, or a security setting does more damage than no answer at all, because it erodes trust in both the assistant and the product.

Most in-house builds answer everything, because getting a model to reliably say "I don't know" when the retrieved context does not support an answer is genuinely hard. It takes grounding, calibrated uncertainty, and a defense strategy layered on top of the raw model. Teams tend to treat this as a finishing touch and discover, after users start complaining, that it is foundational. The abstention behavior is not a nice-to-have; it is the difference between an assistant people rely on and one they learn to ignore.

Mistake 6: Signing up for maintenance you will regret

Even if you clear every hurdle above, you have not bought yourself a finished product. You have adopted a system that needs roughly two AI engineers maintaining it indefinitely: refreshing connectors, retuning retrieval, keeping up with new models, and chasing edge cases.

That is the cost that sinks most projects. Most teams that build an internal AI knowledge base abandon or replace it within 6 to 18 months. One large telecom company spent a year and a half building an AI documentation assistant before giving up entirely. The uncomfortable question is one of opportunity cost: for most companies the AI assistant supports the product but is not the product, so every sprint spent debugging retrieval is a sprint not spent on what actually differentiates you. Which modern company builds its own support-ticketing tool from scratch? Very few, because it is commodity infrastructure. Production RAG is heading the same way.

The hard part is not only building it, it is also keeping it good

None of this means building is always wrong. If an AI assistant is your core product and primary differentiator, you have deep ML expertise on staff, and you can absorb the timeline and the permanent maintenance, building can be the right call. For everyone else, the honest math points the other way.

That is the case for buying. kapa.ai is a purpose-built RAG platform that has already solved the six mistakes above so your team does not have to: 50+ source connectors with automatic freshness, tuned multi-stage retrieval, in-house evaluation on factuality and citation accuracy, coverage-gap analytics, an explicit hallucination guardrail, and SOC 2 Type II security, all deployed in days rather than months. Because it answers more than 500,000 questions a week across 200+ deployments, it has the data volume to keep improving in ways a single build never can.

The outcomes are the real argument. Netlify deployed kapa.ai in a week and now answers 200,000 developer questions a year with no maintenance overhead, and CTO Dana Lawson calls the value "irrefutable." Logitech evaluated several vendors, found most scored around 60% accuracy, and audited kapa.ai at over 99% accuracy when relevant content exists. As Lawson puts it, "Why waste time building from scratch and likely only make it 70% of the way to production? Instead, find somebody trusted you can work with."

If you are weighing an in-house build, the smart move is to validate on a platform first and build only if you uncover a genuinely unique requirement. Want to see it on your own docs? Start a free trial or read the full build vs buy iceberg.