In the previous article, we learned what RAG (Retrieval-Augmented Generation) is and why it is useful for building AI applications.
Now let's go one level deeper and understand how a RAG system actually works.
A typical RAG system depends on several important technologies, including embeddings, vector databases, chunking, semantic search, retrieval, and Large Language Models (LLMs).
Understanding these components will make it much easier to build a RAG application using technologies such as C#, .NET, Ollama, and vector databases.
RAG Architecture at a Glance
A simplified RAG architecture looks like this:
Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
│
│
↓
User Question → Query Embedding
↓
Semantic Search
↓
Relevant Chunks
↓
LLM
↓
Final Answer
There are two major parts:
- Indexing Pipeline: Prepares documents for searching.
- Query Pipeline: Finds relevant information and generates an answer.
1. What is the RAG Indexing Pipeline?
The indexing pipeline prepares your documents before users start asking questions.
Documents
↓
Document Loader
↓
Text Extraction
↓
Chunking
↓
Embedding Model
↓
Vector Database
For example, imagine a company has the following documents:
Employee Handbook.pdf Leave Policy.pdf Travel Policy.pdf Insurance Policy.pdf IT Guidelines.pdf
These documents need to be processed and indexed before the AI can search them.
2. Step 1 – Document Loading
The first step is to load documents from your knowledge source.
A RAG application may load information from:
- PDF files
- Word documents
- Text files
- HTML pages
- Websites
- Databases
- APIs
- Code repositories
The application extracts the useful text from these sources.
PDF ↓ Text Extraction ↓ Plain Text
3. Step 2 – Document Chunking
Large documents are usually divided into smaller pieces called chunks.
Why?
Suppose you have a 200-page PDF containing multiple topics. A user may only need one paragraph from that document.
Sending the entire document to the LLM would be inefficient.
Instead, the document is split into smaller pieces.
Large Document
↓
┌────┼────┬────┬────┐
↓ ↓ ↓ ↓ ↓
C1 C2 C3 C4 C5
For example:
Employee Handbook Chunk 1 → Introduction Chunk 2 → Working Hours Chunk 3 → Leave Policy Chunk 4 → Benefits Chunk 5 → Travel Policy
What is Chunk Size?
Chunk size determines how much content is placed into each chunk.
For example, you might create chunks containing a certain number of tokens or characters.
Document ↓ 500 tokens ↓ Chunk 1 500 tokens ↓ Chunk 2 500 tokens ↓ Chunk 3
There is no single perfect chunk size for every RAG application.
The ideal size depends on the type of documents, the embedding model, retrieval strategy, and the LLM's context window.
What is Chunk Overlap?
Sometimes consecutive chunks share some content.
This is called chunk overlap.
Chunk 1
[ A B C D E ]
[ D E F G H ]
Chunk 2
[ G H I J K ]
Chunk 3
The overlapping content helps preserve context when important information is located near a chunk boundary.
4. Step 3 – What are Embeddings?
One of the most important concepts in RAG is embeddings.
An embedding converts text into a numerical vector that represents aspects of its meaning.
For example:
"How many vacation days do employees get?"
↓
Embedding Model
↓
[0.12, -0.42, 0.71, 0.08, ...]
The actual vector may contain hundreds or thousands of dimensions depending on the embedding model.
You normally do not need to understand every number. The important concept is that semantically related text can have similar vector representations.
5. Why Do We Need Embeddings?
Consider these two sentences:
Sentence 1: How many vacation days can I take? Sentence 2: Employees are entitled to 20 days of annual leave.
The words are different, but the meaning is related.
A semantic search system can use embeddings to recognize this relationship.
Sentence 1
↓
Vector A
Sentence 2
↓
Vector B
Vector A ≈ Vector B
↓
Similar Meaning
6. What is a Vector?
A vector is simply a collection of numbers.
[0.15, -0.22, 0.73, 0.08, -0.41, ...]
In AI applications, vectors can represent the semantic characteristics of text, images, audio, and other types of data.
In RAG, we usually create vectors from document chunks and user queries.
7. What is a Vector Database?
A Vector Database is a database designed to store and search vector representations efficiently.
A RAG system can store each document chunk together with its embedding and metadata.
Vector Database ┌──────────────────────────────────────────┐ │ Vector │ │ Original Text │ │ Document Name │ │ Page Number │ │ Metadata │ └──────────────────────────────────────────┘
For example:
Vector: [0.12, -0.42, 0.71, ...] Text: Employees are entitled to 20 days of annual leave. Document: Leave Policy.pdf Page: 12
8. Why Can't We Just Use SQL?
Traditional databases are excellent for structured data and exact queries.
For example:
SELECT * FROM Employees WHERE Department = 'IT';
But consider a question such as:
"What benefits are available to employees who have been with the company for several years?"
This is a semantic question rather than a simple exact-value lookup.
Vector search allows the system to search based on the meaning represented by the embeddings.
9. What is Semantic Search?
Semantic Search searches for information based on meaning rather than only matching exact words.
For example:
Query: How much annual vacation do I get? Document: Employees receive 20 days of paid annual leave.
A keyword search might not find a strong match because "vacation" and "annual leave" are different phrases.
Semantic search can recognize that the concepts are related.
10. How Does Similarity Search Work?
When the user asks a question, the application converts the question into an embedding.
User Question
↓
Embedding Model
↓
Query Vector
The query vector is then compared against vectors stored in the vector database.
Query Vector
↓
Vector Database
↓
Compare Vectors
↓
Find Similar Vectors
↓
Top Results
The system retrieves the chunks that are considered most relevant.
11. Cosine Similarity
One commonly used concept for measuring similarity between vectors is cosine similarity.
It measures the angle between two vectors rather than simply comparing their raw values.
Vector A
↘
↘
↘
→ Vector B
Small angle
↓
High similarity
A value closer to 1 generally indicates greater similarity, while a value closer to 0 indicates less similarity for common cosine-similarity use cases.
Other similarity or distance measures can also be used depending on the vector database and application.
12. What is Top-K Retrieval?
Instead of returning every matching document, a RAG system normally retrieves a limited number of results.
This is commonly called Top-K retrieval.
For example, if K = 5:
User Question
↓
Vector Search
↓
Top 5 Relevant Chunks
↓
LLM
The retrieved chunks are then added to the prompt or context sent to the LLM.
13. The RAG Query Pipeline
Now let's look at what happens when a user asks a question.
User Question
↓
Create Query Embedding
↓
Search Vector Database
↓
Retrieve Top-K Chunks
↓
Build Prompt
↓
Send to LLM
↓
Generate Answer
For example:
User:
How many vacation days can I take?
↓
Query Embedding
↓
Vector Search
↓
Relevant Chunk:
Employees receive 20 days
of annual leave.
↓
LLM
↓
Answer:
Employees receive 20 days
of annual leave.
14. What is Context in RAG?
The retrieved information provided to the LLM is commonly referred to as context.
The application might construct a prompt like:
System: Answer the question using the provided context. Context: Employees receive 20 days of paid annual leave. Question: How many vacation days can I take?
The LLM uses the context to generate the response.
15. RAG Does Not Train the LLM
This is one of the most common misunderstandings about RAG.
When you add a new document to a RAG system, you generally do not retrain the LLM.
Instead, the new document is processed and added to the retrieval system.
New Document
↓
Chunk
↓
Embedding
↓
Vector Database
↓
Available for Retrieval
The LLM itself remains unchanged.
16. RAG vs Fine-Tuning
| RAG | Fine-Tuning |
|---|---|
| Adds external context at runtime | Modifies model behavior through additional training |
| Useful for changing knowledge | Useful for specialized behavior |
| Documents can be updated independently | Training is required for model updates |
| Can retrieve specific source information | Does not inherently provide source retrieval |
For frequently changing business information, RAG can often be more practical than retraining a model every time information changes.
17. What is Hybrid Search?
Semantic search is powerful, but exact keyword matching can also be important.
Hybrid Search combines keyword-based search with semantic/vector search.
User Query
↓
┌─────────┴─────────┐
↓ ↓
Keyword Search Vector Search
↓ ↓
└─────────┬─────────┘
↓
Combined Results
↓
LLM
This can be useful when a query contains important exact terms such as product codes, error codes, API names, or technical identifiers.
18. What is Metadata Filtering?
RAG systems often store metadata along with document chunks.
For example:
Document: LeavePolicy.pdf Department: HR Year: 2026 Country: India Page: 10
The application can use this metadata to restrict retrieval.
For example:
Question ↓ Filter: Country = India Year = 2026 ↓ Vector Search ↓ Relevant Results
Metadata filtering can be especially useful in enterprise applications where users should only retrieve information relevant to their department, region, product, or access level.
19. Reranking
The first retrieval step may return several potentially relevant chunks. A reranker can then evaluate those results and reorder them based on relevance.
User Question
↓
Initial Retrieval
↓
20 Candidate Chunks
↓
Reranker
↓
Top 5 Best Chunks
↓
LLM
Reranking can improve retrieval quality in some applications, especially when the initial search returns many similar results.
20. What is a Context Window?
LLMs have a limited amount of information they can process in a single request. This available input capacity is commonly called the context window.
If a RAG system retrieves too much information, it may consume a large portion of the available context.
User Question
+
Retrieved Chunks
+
Instructions
↓
LLM Context Window
This is one reason why good retrieval is important. The goal is not to retrieve everything. The goal is to retrieve the most useful information.
21. RAG Quality Depends on Retrieval Quality
A powerful LLM cannot completely compensate for poor retrieval.
Consider:
User Question
↓
Poor Retrieval
↓
Wrong Context
↓
LLM
↓
Poor Answer
Compare that with:
User Question
↓
Good Retrieval
↓
Relevant Context
↓
LLM
↓
Better Answer
This is why RAG development requires attention not only to the LLM but also to document processing and retrieval.
22. Common RAG Problems
Problem 1: Poor Chunking
Important information may be split across chunks in an inconvenient way.
Problem 2: Wrong Retrieval
The vector database may return information that is similar but not actually relevant.
Problem 3: Too Much Context
Retrieving too many chunks can introduce irrelevant information and consume context.
Problem 4: Poor Documents
Badly formatted PDFs, scanned documents, tables, and duplicated content can reduce retrieval quality.
Problem 5: Access Control
A RAG application must ensure that users cannot retrieve documents they are not authorized to access.
23. Complete RAG Architecture
INDEXING PIPELINE
Documents
↓
Document Loader
↓
Text Extraction
↓
Chunking
↓
Embedding Model
↓
Vector Database
│
│
│
│
└──────────────────────────────┐
│
↓
QUERY PIPELINE
User Question
↓
Query Embedding
↓
Metadata Filter
↓
Vector / Hybrid Search
↓
Reranking
↓
Top Relevant Chunks
↓
Prompt + Context
↓
LLM
↓
Final Answer
24. Example: Company Knowledge Assistant
Let's put everything together using a real-world example.
Imagine you are building an AI assistant for a company.
The company has:
500 PDF documents 100 Word documents 1000 HTML pages Internal technical documentation
The indexing process could be:
Documents
↓
Extract Text
↓
Clean Text
↓
Chunk
↓
Generate Embeddings
↓
Store in Vector Database
A developer asks:
"How do I resolve error XYZ-102?"
The query pipeline could be:
Question ↓ Embedding ↓ Vector Search ↓ Retrieve Technical Documentation ↓ Rerank Results ↓ Top Relevant Sections ↓ LLM ↓ Troubleshooting Steps
25. Where Does the LLM Fit?
The LLM is only one component of a RAG system.
RAG Application ┌─────────────────────────────────────────┐ │ │ │ Document Processing │ │ ↓ │ │ Embeddings │ │ ↓ │ │ Vector Database │ │ ↓ │ │ Retrieval │ │ ↓ │ │ Context │ │ ↓ │ │ LLM │ │ ↓ │ │ Answer │ │ │ └─────────────────────────────────────────┘
This is an important concept: RAG is an application architecture, not a type of LLM.
26. RAG Technology Stack
A RAG application can be built using many different technologies.
| Layer | Examples |
|---|---|
| Application | C#, .NET, Python, JavaScript |
| LLM | GPT, Claude, Gemini, Llama, Qwen |
| Embedding Model | Various text embedding models |
| Vector Database | Various vector database technologies |
| Document Storage | Files, object storage, databases, websites |
| Retrieval | Vector search, keyword search, hybrid search |
The exact technology choices depend on your application requirements, scale, cost, security, and infrastructure.
27. RAG with Local LLMs
RAG does not require the LLM to be hosted in the cloud.
You can also build a RAG application using a locally hosted model.
Documents
↓
Embeddings
↓
Vector Database
↓
Retriever
↓
Local LLM
↓
Answer
Tools such as Ollama can be used to run supported LLMs locally, while your application handles document processing and retrieval.
This can be useful when experimenting with AI locally or when certain data-handling requirements favor local processing.
28. RAG with C# and .NET
RAG is not limited to Python.
Developers can build RAG applications using C# and .NET.
A simplified architecture could look like this:
ASP.NET Core
↓
RAG Service
↓
┌──────────┼──────────┐
↓ ↓ ↓
LLM Embeddings Retriever
↓
Vector DB
The application can expose the RAG functionality through a Web API, web application, desktop application, or background service.
29. RAG vs Traditional Database Query
| Traditional Database | RAG |
|---|---|
| Structured queries | Natural-language questions |
| Exact conditions | Semantic retrieval |
| Structured data | Often unstructured or semi-structured information |
| Returns records | Retrieves context and generates an answer |
In real applications, these approaches can also be combined.
30. RAG in Simple Terms
If you remember only one thing from this article, remember this:
Documents
↓
Break into Chunks
↓
Convert Chunks into Embeddings
↓
Store in Vector Database
↓
User Asks Question
↓
Convert Question into Embedding
↓
Search Similar Vectors
↓
Retrieve Relevant Chunks
↓
Send Context + Question to LLM
↓
Generate Answer
Conclusion
A RAG system is much more than simply connecting a chatbot to a vector database.
A well-designed RAG application typically involves document processing, chunking, embeddings, vector storage, semantic retrieval, metadata filtering, reranking, context management, and an LLM.
The most important concepts to understand are:
- Chunking: Break large documents into useful pieces.
- Embeddings: Represent text as numerical vectors.
- Vector Database: Store and search those vectors.
- Semantic Search: Find information based on meaning.
- Retrieval: Select relevant information for a question.
- Context: Provide retrieved information to the LLM.
- Generation: Let the LLM produce the final response.
Once you understand these components, you are ready to build a real RAG application.
No comments:
Post a Comment