Project: A Private Document Q&A Assistant

Build the mental model

This is the flagship local-RAG project, and it exists to combine five separate ideas from the retrieval chapters into one pipeline that actually answers questions from a user's own documents.

Text extraction — turning an uploaded PDF or text file into plain text
Chunking — splitting text into overlapping pieces small enough to embed and retrieve individually
An embedding model — converting each chunk into a vector
A vector store — indexing those vectors for similarity search
Generation — a local chat model answering from retrieved chunks alone, with citations

Embedding an entire document as one vector loses the specificity needed to match a narrow question.

An embedding model converts each chunk into a vector, and a vector store indexes those vectors so a new question's own embedding can be compared against them by similarity search — this is the retrieval half of retrieval-augmented generation, and it is the reason the assistant can answer from documents the base model was never trained on.

The generation half is a local chat model that receives only the top-matching chunks plus the user's question as context, and is instructed to answer using that context alone.

Citations tie each claim back to a specific chunk and source document, which is what lets a user actually verify the answer rather than trust it blindly — a direct application of the honesty and verifiability principles the RAG chapter emphasized.

Nothing here trains or fine-tunes the model

It is worth restating plainly: nothing about this project trains or fine-tunes the model. The documents are never baked into any weights. Every question re-runs the same embed-search-generate sequence from scratch against the same static document index, so newly added documents are searchable immediately and nothing is ever "forgotten" by retraining.

text

PRIVATE DOCUMENT Q&A PIPELINE
-----------------------------
INGEST -- runs once per uploaded document
  document --> extract text --> chunk (with overlap) --> embed chunk
                                                              |
                                                              v
                                                      VECTOR STORE
                                        [ vector, chunk text, source, idx ]

QUERY -- runs on every question
  question --> embed question
                    |
                    v
        similarity search over VECTOR STORE
                    |
                    v
          top-K matching chunks (by score)
                    |
                    v
  prompt = system + top-K chunks + question  --->  LOCAL CHAT MODEL
                    |
                    v
         answer text + citations [source, chunk #]

Nothing here trains or fine-tunes the model -- every question re-runs
embed -> search -> generate fresh against the same static chunk index.

Connect it to a real scenario

Build this as an ingest path and a query path that share one vector store.

Extract and chunk documents

For each uploaded document, extract raw text and split it into chunks of a few hundred tokens with a small overlap, so an idea split across a chunk boundary is not lost.

Embed and store chunks

Call the local embedding endpoint for each chunk and store the resulting vector alongside its chunk text, source filename, and chunk index.

Embed the question

On every question, embed it with the same embedding model used for the chunks.

Retrieve top chunks

Run a similarity search against the vector store to pull back the top handful of matching chunks.

Generate a grounded answer

Build a prompt that includes only those chunks plus the question and an instruction to answer only from the provided context, then send that to the local chat model.

Render the answer together with a visible citation list — which document and chunk each part of the answer drew from — not as an afterthought but as a first-class part of the response.

"Done" for this project means:

asking a question the documents actually answer returns a correct answer with citations that check out when you open the source chunk
asking a question the documents do not cover makes the assistant say so instead of guessing
adding a new document makes it searchable on the very next question, with no restart or retraining step required

Chunking + embedding + vector search find only the relevant part of a document
Nothing here trains or fine-tunes the model -- it's retrieval, every time
Citations let the user verify an answer instead of trusting it blindly

Try the working example

typescript

// A private document Q&A pipeline: ingest documents into a vector store,
// then answer questions using only retrieved chunks -- never by retraining
// or fine-tuning anything.
const EMBED_URL = "http://localhost:11434/api/embeddings";
const CHAT_URL = "http://localhost:11434/api/chat";
const EMBED_MODEL = "nomic-embed-text";
const CHAT_MODEL = "llama3.1:8b";

type StoredChunk = {
  vector: number[];
  text: string;
  source: string;
  chunkIndex: number;
};

const vectorStore: StoredChunk[] = []; // in-memory for a single-user tool

function chunkText(text: string, size = 800, overlap = 150): string[] {
  const chunks: string[] = [];
  let start = 0;
  while (start < text.length) {
    const end = Math.min(start + size, text.length);
    chunks.push(text.slice(start, end));
    if (end === text.length) break;
    start = end - overlap; // overlap so an idea at a boundary isn't lost
  }
  return chunks;
}

async function embed(text: string): Promise<number[]> {
  const res = await fetch(EMBED_URL, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ model: EMBED_MODEL, prompt: text }),
  });
  if (!res.ok) throw new Error(`Embedding request failed: ${res.status}`);
  const data = (await res.json()) as { embedding: number[] };
  return data.embedding;
}

// --- ingest path: run once per uploaded document ---
export async function ingestDocument(source: string, rawText: string) {
  const chunks = chunkText(rawText);
  for (let i = 0; i < chunks.length; i++) {
    const vector = await embed(chunks[i]);
    vectorStore.push({ vector, text: chunks[i], source, chunkIndex: i });
  }
}

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0, normA = 0, normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

function searchChunks(queryVector: number[], topK = 4): StoredChunk[] {
  return [...vectorStore]
    .map((chunk) => ({ chunk, score: cosineSimilarity(queryVector, chunk.vector) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK)
    .map((r) => r.chunk);
}

// --- query path: run on every question ---
export async function answerFromDocuments(question: string) {
  const questionVector = await embed(question);
  const topChunks = searchChunks(questionVector);

  const context = topChunks
    .map((c, i) => `[${i + 1}] (${c.source}, chunk ${c.chunkIndex})\n${c.text}`)
    .join("\n\n");

  const res = await fetch(CHAT_URL, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: CHAT_MODEL,
      stream: false,
      messages: [
        {
          role: "system",
          content:
            "Answer only using the numbered context below. If the " +
            "context does not contain the answer, say so -- do not guess. " +
            "Cite context numbers like [1] for every claim.\n\n" + context,
        },
        { role: "user", content: question },
      ],
    }),
  });

  if (!res.ok) throw new Error(`Chat request failed: ${res.status}`);
  const data = (await res.json()) as { message: { content: string } };

  return {
    answer: data.message.content,
    citations: topChunks.map((c) => ({ source: c.source, chunk: c.chunkIndex })),
  };
}

Not runnable here

This code calls your own local AI server (e.g. Ollama) — this site's browser playground cannot reach a local server on your machine (network access is restricted there for safety). Run it yourself in a terminal or file where a local server is actually running.

You should see

When it runs correctly, a question that matches something in the uploaded documents gets a real, paragraph-length answer within a few seconds, with a citation list underneath showing which source document and chunk each part came from. A question the documents don't cover gets an honest "this isn't in the provided documents" response instead of a fabricated one. Adding a new document makes it show up in the very next query with no restart or re-setup needed.

5-minute try-it

Add a similarity-score threshold check: if every retrieved chunk's score falls below the threshold, skip calling the LLM entirely and return "no relevant content found in the documents" directly instead.

One important caution

Chunking too large or with no overlap -- retrieval pulls back irrelevant blocks of text and boundaries cut ideas in half

Feeding retrieved chunks as context without explicitly instructing the model to answer only from them -- it will fill gaps with its own general knowledge instead of admitting the documents don't cover it

Ollama API Reference -- Generate Embeddings — Local AI / Local LLM