Sep 12, 2025

Tutorial: Build an agentic AI system with knowledge of your product

Tutorial: Build an agentic AI system with knowledge of your product

Using Kapa + LangGraph to ground answers and trigger actions in tools like Zendesk, Salesforce and Pylon

by

by

by

David Karlsson

David Karlsson

David Karlsson

Overview

How Kapa fits in an agentic system
Why Kapa belongs in an agentic system
Example: Building a support agent with LangGraph
Practical considerations
Closing thoughts

kapa.ai is an LLM-powered AI assistant that answers technical questions about your products. We integrate with knowledge sources to answer user questions and support tickets through a RAG pipeline.

At the center of everything we do is answer accuracy. LLMs on their own are unreliable; they'll make stuff up because it sounds nice. Kapa exists to fix that by handling crawling, parsing, evaluation, guardrails, security (etc.) to give you and your users a specialized AI assistant that can be trusted.

Our customers and their end-users trust Kapa for accurate answers. Because of this, a natural question that often comes up is about ways to more deeply integrate Kapa into customer support workflows. For example:

“Kapa is great at answering questions — but can it also create tickets or escalate to a human?”

This use case, the desire for Kapa to handle more than just question-answering, is becoming increasingly common and resonates with the rising of interest in agentic AI systems in the industry in general.

The short answer is: not directly out-of-the-box. But Kapa is designed to act as a component inside a larger system, so you can extend it into those flows in a way that fits your business.

How Kapa fits in an agentic system

At its core, kapa.ai is a high-accuracy technical assistant for your product. We integrate your knowledge sources, run them through a retrieval-augmented generation pipeline, and deliver grounded answers with citations. Easy to set up and deploy anywhere your users are.

So why haven't we extended Kapa's ability to encompass more workflows, like ticket processing or billing?

We see real value in extended workflows, but our mission is to build the most accurate technical assistants. Staying focused on that ensures we deliver the reliability our customers count on.

Extended workflows are largely business-specific. A startup might use Zendesk, whereas a larger enterprise might use ServiceNow. We ourselves use Pylon. There's no one-size fits all. Not only in terms of which systems are being used, but also in terms of how a specific company wants their agent to act:

  • If a user reports a bug, should we create a ticket in Jira for engineering, or a case in Zendesk for support?

  • Should escalation rules differ for paying customers vs free-tier users?

  • If the question is account-related (billing, permissions, security), should it always bypass Kapa and go straight to human support?

Instead, we want Kapa to be a modular component: a tool you can drop into whatever agentic system makes sense for your company. That way, you get the best of both worlds: Kapa provides accurate answers, and your system handles the actions that are unique to your workflows.

Why Kapa belongs in an agentic system

If you're building an agentic tool or system without Kapa, you have two options:

  1. Build and maintain your own RAG pipeline, including all the bells and whistles that that entails.

  2. Live life at the mercy of a generic LLM trying to interpret the user's (or another agent's) question.

With Kapa, you can have the agent route technical questions to Kapa, and get an accurate response with sources, or a genuine "I don't know". Other questions (billing, bugs, access requests, ticket creation) can flow to other tools, either directly or via Kapa first.

The result is an agent that stays flexible but never sacrifices correctness.

Example: Building a support agent with LangGraph

To make this concrete, let's look at a small prototype we built with LangGraph.

Our support agent has three moving parts:

  1. query_kapa – a tool that calls Kapa's API.

  2. create_support_ticket – a tool that files a ticket (we wired Pylon, but you could swap Zendesk/ServiceNow/etc.).

  3. Reasoner node – a Claude-powered LLM that decides which tool to call based on the user's message.

That's essentially the whole loop: the reasoner looks at the user's question and decides the appropriate action (tool).

For testing, you could run this as a simple CLI, and for wider testing you may want to consider hooking it into a chat UI (like Slack). You may also want to consider extending it with more tools. The limits are where you decide them to be, and that's the kind of flexibility that we think agentic tools need and deserve in order to be useful.

The following sections show simplified code snippets of what each piece could look like.

1. Kapa as a tool

In this example, query_kapa is a tool that takes a natural-language question, calls Kapa's API, and returns a grounded answer.

@tool
def query_kapa(question: str) -> str:
    """Query Kapa API for documentation-based answers.

    Args:
        question: The search query to send to Kapa

    Returns:
        Text answer with optional sources and uncertainty note
    """
    resp = kapa_query(question)   # POST /query/v1/projects/{id}/chat
    if not resp:
        return "Docs unavailable: I couldn't access the documentation right now."

    answer = resp.get("answer", "")
    is_uncertain = response.get("is_uncertain", False)
    sources = resp.get("relevant_sources", [])[:2]

    out = [f"Technical assistant answer: "]
    if is_uncertain:
        out.append("\n[UNCERTAIN]: This information may be incomplete.")
    out.append(f"\n\n{answer}")
    if sources:
        out.append("\n\nSources:")
        for s in sources:
            out.append(f"- {s['title']}: {s['source_url']}")
    return "\n".join(out)
  1. Your own action tool

Business-specific action tool. In our case it files a Pylon ticket, but you might swap Zendesk or ServiceNow.

@tool
def create_support_ticket(title: str, body_html: str) -> str:
    """Create a support ticket with the provided title and body.

    Args:
        title: Ticket title
        body_html: HTML content

    Returns:
        Short confirmation message
    """
    ticket = pylon_create_issue(title=title, body_html=body_html)
    ticket_id = (ticket.get("data") or {}).get("id", "UNKNOWN")
    return f"Created support ticket #{ticket_id}. We'll follow up shortly."

3. Policy prompt for the reasoner

Here's the policy prompt that guides the reasoner. It's just text, but it encodes your routing logic.

SYSTEM_PROMPT = """
You are a support agent with two tools: query_kapa and create_support_ticket.

Policy:
- For technical/product questions → use query_kapa first and return its answer verbatim.
- If the answer is uncertain/insufficient, or request is account/billing/bug → offer create_support_ticket.
- Always confirm with the user before creating tickets.
"""

4. Putting it all together in LangGraph

Finally, we wire everything together with LangGraph. The loop is: reason → (maybe) tools → reason.

# Build the reasoner node
llm = ChatAnthropic(model_name="claude-4-sonnet-20250514", temperature=0)
llm_with_tools = llm.bind_tools([query_kapa, create_support_ticket])

def reasoner(state: MessagesState) -> dict:
    reply = llm_with_tools.invoke([SystemMessage(content=SYSTEM_PROMPT)] + state["messages"])
    return {"messages": [reply]}

# Wire the graph: reason → (maybe) tools → reason
builder = StateGraph(MessagesState)
builder.add_node("reasoner", reasoner)
builder.add_node("tools", ToolNode([query_kapa, create_support_ticket]))

builder.add_edge(START, "reasoner")
builder.add_conditional_edges("reasoner", tools_condition)
builder.add_edge("tools", "reasoner")

app = builder.compile()

Practical considerations

A few things to keep in mind if you're experimenting with this pattern:

  • Kapa queries inside your agent count the same as any other Kapa queries.

  • Kapa prioritizes accuracy, so queries typically take a few seconds.

  • Agentic systems are powerful but can be brittle; routing logic and tool definitions matter a lot.

  • In the examples provided here, tools return plaintext strings. In production, you may prefer to return structured JSON (e.g. {answer, sources, uncertain}) so the agent can make precise decisions about when to escalate or how to render results.

Closing thoughts

We're excited about the rise of agentic AI, and we're even more excited to see how customers experiment with Kapa inside these systems.

If you're building something, we'd love to hear about it! What tools are you combining Kapa with? What's working, what's missing? While our stance is that Kapa isn't trying to be the all-in-one agent, we want to make sure it's a trustworthy module you can plug in. Any product feedback to help us make this a better experience is incredibly valuable.

Our focus is on making Kapa the most accurate Q&A agent — we're excited to see how customers compose it into their own agentic workflows, and we'll keep improving developer ergonomics to make that easier.

Useful links:

Trusted by hundreds of COMPANIES to power production-ready AI assistants

Turn your knowledge base into a production-ready AI assistant

Request a demo to try kapa.ai on your data sources today.