Build the mental model
Retrieval-Augmented Generation (RAG) is the standard architecture for giving a local LLM access to information it was never trained on — your notes, a codebase, internal documentation — without retraining anything.
Indexing lane (offline)
Documents are split into chunks, each chunk is converted into a numeric vector by an embedding model, and those vectors are stored in a vector database alongside the original text. Runs once.
Query lane (every question)
The question itself is embedded with the same model, the vector store returns the closest chunks, and those chunks are inserted into the prompt sent to the LLM as context. The model then generates an answer grounded in that context.
The most persistent misconception about RAG is that it 'trains the model on your documents.' It does not. Nothing about the LLM's weights changes when you add a document to a RAG system — no gradient descent, no backpropagation, no fine-tuning happens. What actually happens is much simpler and much faster: relevant text is copied into the prompt at request time. This is why RAG updates are instant — add a new document and the next query can retrieve it immediately, with zero retraining cost.
RAG's Real Limitation
RAG has limits: the model only 'knows' what fits in the retrieved context for that one request, and if retrieval picks the wrong chunks, the model reasons over the wrong information. Operationally, RAG quality depends far more on chunking and retrieval accuracy than on the LLM itself.
- RAG
- Retrieval-Augmented Generation — an architecture where relevant text is retrieved from an external store and inserted into the prompt at request time, instead of being baked into the model's weights.
- Chunking
- Splitting a document into smaller pieces, usually with some overlap between adjacent pieces, so each piece is small enough to embed and retrieve meaningfully.
- Retrieval
- The step, run at query time, of searching a vector store for the chunks whose embeddings are most similar to the question's embedding.
LOCAL RAG: TWO LANES
--------------------
LOCAL RAG: TWO LANES
---------------------
INDEXING LANE (offline, runs once per document)
documents -> chunker -> embedding model -> vector store
QUERY LANE (runs on every user question)
question -> embed -> search vector store -> top chunks
|
v
chunks + question -> LLM -> answerConnect it to a real scenario
Building a local RAG pipeline requires four working pieces, and getting the first two right matters more than picking a fancy LLM.
Chunking
Split documents into overlapping pieces (a few hundred tokens each, with 10-20% overlap) so a fact near a chunk boundary is not orphaned.
Embeddings
Run a local embedding model (an all-MiniLM or BGE variant via sentence-transformers, or a GGUF embedding model via llama.cpp) over every chunk to get a fixed-length vector.
Vector store
Something as light as a local SQLite table with a vector extension, or a dedicated store like Chroma, Qdrant, or FAISS running entirely on your machine.
Retrieval at query time
Embed the incoming question with the same embedding model, run a nearest-neighbor search (cosine similarity is the common default) to get the top-k chunks, and paste them into the prompt template ahead of the question.
- Chunk size that is too large (dilutes relevance) or too small (loses context)
- A mismatch between the embedding model used for indexing vs querying — they must be the same model, or the vector spaces will not line up and retrieval quality collapses silently
Try the working example
def chunk_text(text, chunk_size=60, overlap=20):
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
if end >= len(text):
break
start = end - overlap
return chunks
document = (
"Local AI lets you run models on your own hardware. "
"This means no data leaves your machine during inference. "
"RAG adds a retrieval step before generation. "
"It does not change the model's weights at all."
)
chunks = chunk_text(document, chunk_size=60, overlap=20)
for i, c in enumerate(chunks):
print(f"chunk {i} (len={len(c)}): {c!r}")
print(f"total chunks: {len(chunks)}")Running the chunker over the sample document with chunk_size=60 and overlap=20 produces 5 overlapping chunks:
chunk 0 (len=60): 'Local AI lets you run models on your own hardware. This mean'
chunk 1 (len=60): ' hardware. This means no data leaves your machine during inf'
chunk 2 (len=60): 'r machine during inference. RAG adds a retrieval step before'
chunk 3 (len=60): 'etrieval step before generation. It does not change the mode'
chunk 4 (len=39): " not change the model's weights at all."
total chunks: 5
Each chunk after the first repeats the last 20 characters of the previous one (the overlap), which is why 'hardware' and 'inference' each appear split across two adjacent chunks instead of being lost at a boundary.5-minute try-it
Take a short text file (a few paragraphs) and write a script that splits it into overlapping chunks the way the code example does. Then write a second function that, given a question, does a naive 'search' by counting shared words between the question and each chunk, and returns the chunk with the most overlap. This is not real embedding-based search, but it demonstrates the same retrieve-then-inject pattern RAG uses with proper vector embeddings.
One important caution
Using a different embedding model (or model version) for indexing than for querying — the vector spaces are not compatible and retrieval quality silently collapses without any error.
Assuming RAG lets you skip data governance because 'the model isn't trained on it' — retrieved chunks are still sent to the LLM as plaintext context and can appear in its output or logs.
Quick Check
LangChain — RAG Concepts — Local AI / Local LLM