RAG-Powered Chatbot

Build a retrieval-augmented generation chatbot with Brinicle as the knowledge base

Overview

Retrieval-Augmented Generation (RAG) combines a large language model with a retrieval system that fetches relevant context before generating a response. Brinicle serves as the high-performance retrieval layer: you index your documents as vectors, and at query time, you retrieve the most relevant chunks to feed into the LLM.

Brinicle's disk-first design means you can serve large knowledge bases with minimal RAM, making it practical for production RAG deployments on modest hardware.

Architecture

Chunk your documents into passages (e.g., 256-512 tokens each)
Embed each chunk using your preferred embedding model
Index the embeddings in Brinicle's VectorEngine
At query time, embed the user's question and search for the top-k relevant chunks
Pass the retrieved chunks as context to the LLM

Implementation

Step 1: Index Your Documents

import numpy as np
import brinicle

EMBED_DIM = 384

engine = brinicle.VectorEngine(
    "rag_index",
    dim=EMBED_DIM,
    M=48,
    ef_construction=1024,
    ef_search=512,
)

engine.init(mode="build")

# Example: index document chunks
documents = [
    {"id": "doc1_chunk1", "text": "Brinicle is a disk-first HNSW retrieval engine..."},
    {"id": "doc1_chunk2", "text": "VectorEngine supports search with distance..."},
    {"id": "doc2_chunk1", "text": "ItemSearchEngine combines lexical and semantic search..."},
    # ... thousands more chunks
]

for doc in documents:
    vector = embed(doc["text"])  # your embedding function
    engine.ingest(doc["id"], vector)

engine.finalize()

Step 2: Retrieve at Query Time

def retrieve_context(query: str, k: int = 5):
    query_vector = embed(query)
    results = engine.search_with_distance(query_vector, k=k)
    return results

# Example
context = retrieve_context("How does hybrid search work?")
print(context)
# [("doc2_chunk1", 0.2341), ("doc1_chunk1", 0.3012), ...]

Step 3: Generate with LLM

def rag_chat(query: str):
    # Retrieve relevant chunks
    results = retrieve_context(query, k=5)

    # Build context from retrieved chunks
    context_text = "\n\n".join(
        doc_lookup[doc_id]["text"]  # your document store
        for doc_id, _ in results
    )

    # Call LLM with context
    prompt = f"""Based on the following context, answer the question.

Context:
{context_text}

Question: {query}

Answer:"""

    response = llm.generate(prompt)  # your LLM call
    return response

Step 4: Batch Retrieval for Multiple Queries

queries = ["What is alpha?", "How to delete items?", "Hybrid search example"]
vectors = np.array([embed(q) for q in queries]).astype(np.float32)

results = engine.search_batch(vectors, k=5, n_jobs=4)

Streaming Ingest for Large Knowledge Bases

Brinicle ingests records one at a time, so the full dataset does not need to fit in memory:

engine.init(mode="build")

for chunk in stream_document_chunks():  # your streaming function
    vector = embed(chunk["text"])
    engine.ingest(chunk["id"], vector)

engine.finalize()

Updating the Knowledge Base

Insert New Documents

engine.init(mode="insert")

for chunk in new_documents:
    vector = embed(chunk["text"])
    engine.ingest(chunk["id"], vector)

engine.finalize()

Upsert Changed Documents

engine.init(mode="upsert")

for chunk in updated_documents:
    vector = embed(chunk["text"])
    engine.ingest(chunk["id"], vector)

engine.finalize()

Delete Outdated Documents

deleted_count, not_found = engine.delete_items(
    ["old_doc_1", "old_doc_2"],
    return_not_found=True,
)

RAG-Powered Chatbot

On this page