Build a RAG Application with C# and .NET 8

In the previous articles, we explored RAG, embeddings, vector databases, semantic search, and fine-tuning.

Now let's build something practical.

In this tutorial, we will build a simple RAG (Retrieval-Augmented Generation) application using C# and .NET 8.

The application will allow an AI model to answer questions using information from our own documents instead of relying only on the model's built-in knowledge.

We will use Ollama to run an LLM locally and build the RAG pipeline using C#.

What We Are Going to Build

Our application will follow this architecture:

                Document
                    ↓
              Text Extraction
                    ↓
                 Chunking
                    ↓
             Embedding Model
                    ↓
              Vector Storage
                    ↓
              Semantic Search
                    ↓
             Relevant Chunks
                    ↓
                    LLM
                    ↓
                 Answer

For example, imagine we have a company document containing:

Employees are entitled to 20 days
of paid annual leave every year.

Unused leave can be carried forward
according to company policy.

The user can ask:

How many annual leave days do employees get?

The RAG application retrieves the relevant document content and provides it to the LLM.

Technologies Used

• C#
• .NET 8
• Ollama
• LLM
• Embedding Model
• Vector Search

To keep the example easy to understand, we will implement a lightweight vector store directly in C# rather than introducing a full production vector database.

The same architecture can later be extended to databases such as Qdrant, PostgreSQL with pgvector, Elasticsearch, Azure AI Search, or other vector databases.

Prerequisites

Before starting, install:

• .NET 8 SDK
• Ollama
• A local LLM model
• An embedding model

Verify the .NET installation:

dotnet --version

You should see a .NET SDK version beginning with 8 or a compatible newer SDK.

1. Install Ollama

Ollama allows you to run supported AI models locally.

After installing Ollama, verify that it is available:

ollama --version

Start an LLM model. For example:

ollama pull llama3.2

Then run it:

ollama run llama3.2

You can use another compatible model depending on your hardware.

2. Download an Embedding Model

RAG requires an embedding model to convert documents and user questions into vectors.

For example:

ollama pull nomic-embed-text

The embedding model and chat model serve different purposes.

Model	Purpose
Embedding Model	Converts text into vectors
LLM	Generates the final answer

3. Create a .NET 8 Project

Create a new console application:

dotnet new console -n SimpleRag
cd SimpleRag

Open the project in Visual Studio or your preferred IDE.

4. Understand the RAG Pipeline

Before writing code, let's understand the complete flow.

             INGESTION

Documents
    ↓
Split into Chunks
    ↓
Generate Embeddings
    ↓
Store Vectors


             QUERY

User Question
    ↓
Generate Query Embedding
    ↓
Similarity Search
    ↓
Retrieve Relevant Chunks
    ↓
Send Context to LLM
    ↓
Generate Answer

There are therefore two major stages:

• Ingestion: Prepare and index the documents.
• Query: Retrieve relevant information and generate an answer.

5. Create Sample Documents

For this tutorial, create a folder called Documents.

Create a file called company-policy.txt:

Company Leave Policy

Employees receive 20 days of paid annual leave every year.

Employees should submit leave requests through the employee portal.

Unused annual leave can be carried forward according to the company's leave policy.

Managers are responsible for approving leave requests.

Emergency leave should be reported to the manager as soon as possible.

6. Create a Document Chunk

A document is usually divided into smaller pieces before generating embeddings.

Create a simple class:

public class DocumentChunk
{
    public string Text { get; set; } = string.Empty;

    public float[] Embedding { get; set; } = Array.Empty<float>();
}

Each chunk contains:

• The original text
• The embedding vector

7. Create the Ollama Embedding Method

Ollama exposes an HTTP API that can be called from C#.

Create a method to generate an embedding:

using System.Net.Http.Json;

public class OllamaEmbeddingService
{
    private readonly HttpClient _httpClient;

    public OllamaEmbeddingService(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<float[]> GenerateEmbeddingAsync(string text)
    {
        var request = new
        {
            model = "nomic-embed-text",
            prompt = text
        };

        var response = await _httpClient.PostAsJsonAsync(
            "http://localhost:11434/api/embeddings",
            request);

        response.EnsureSuccessStatusCode();

        var result =
            await response.Content.ReadFromJsonAsync<EmbeddingResponse>();

        return result?.Embedding ?? Array.Empty<float>();
    }

    private class EmbeddingResponse
    {
        public float[] Embedding { get; set; } = Array.Empty<float>();
    }
}

The embedding model converts the supplied text into a numerical vector.

8. What Does an Embedding Look Like?

You don't normally need to understand every number in an embedding.

Conceptually, the result looks like:

"Employees receive 20 days of annual leave."

        ↓

[0.023, -0.451, 0.812, 0.117, ...]

The vector can contain hundreds or thousands of dimensions depending on the embedding model.

9. Create a Simple Text Chunker

For a production application, chunking requires more sophisticated rules.

For this tutorial, we can use paragraphs as chunks:

public static List<string> SplitIntoChunks(string text)
{
    return text
        .Split(
            new[] { "\r\n\r\n", "\n\n" },
            StringSplitOptions.RemoveEmptyEntries)
        .Select(x => x.Trim())
        .Where(x => !string.IsNullOrWhiteSpace(x))
        .ToList();
}

This keeps the example simple.

10. Create a Simple Vector Store

For demonstration purposes, we can keep the vectors in memory.

public class VectorStore
{
    private readonly List<DocumentChunk> _chunks = new();

    public void Add(DocumentChunk chunk)
    {
        _chunks.Add(chunk);
    }

    public List<DocumentChunk> GetAll()
    {
        return _chunks;
    }
}

In a real application, the vectors would usually be persisted in a vector database.

11. Calculate Cosine Similarity

To determine how similar two vectors are, we can use cosine similarity.

The basic idea is:

Query Vector
     ↓
Compare with Document Vector
     ↓
Similarity Score
     ↓
Rank Results

Create a helper method:

public static double CosineSimilarity(
    float[] vectorA,
    float[] vectorB)
{
    if (vectorA.Length != vectorB.Length)
        throw new ArgumentException("Vector dimensions must match.");

    double dotProduct = 0;
    double magnitudeA = 0;
    double magnitudeB = 0;

    for (int i = 0; i < vectorA.Length; i++)
    {
        dotProduct += vectorA[i] * vectorB[i];

        magnitudeA += vectorA[i] * vectorA[i];
        magnitudeB += vectorB[i] * vectorB[i];
    }

    if (magnitudeA == 0 || magnitudeB == 0)
        return 0;

    return dotProduct /
           (Math.Sqrt(magnitudeA) * Math.Sqrt(magnitudeB));
}

A higher similarity score generally indicates that the vectors are more closely aligned according to this metric.

12. Implement Semantic Search

Now we can search our vector store.

public List<(DocumentChunk Chunk, double Score)> Search(
    float[] queryEmbedding,
    int topK = 3)
{
    return _chunks
        .Select(chunk => (
            Chunk: chunk,
            Score: CosineSimilarity(
                queryEmbedding,
                chunk.Embedding)))
        .OrderByDescending(x => x.Score)
        .Take(topK)
        .ToList();
}

This is the core of our semantic retrieval process.

13. Index the Documents

Now let's read the document and create embeddings for each chunk.

var httpClient = new HttpClient();

var embeddingService =
    new OllamaEmbeddingService(httpClient);

var vectorStore = new VectorStore();

var documentText =
    await File.ReadAllTextAsync(
        "Documents/company-policy.txt");

var chunks =
    SplitIntoChunks(documentText);

foreach (var chunkText in chunks)
{
    var embedding =
        await embeddingService
            .GenerateEmbeddingAsync(chunkText);

    vectorStore.Add(new DocumentChunk
    {
        Text = chunkText,
        Embedding = embedding
    });
}

At this point, our document has been converted into searchable vectors.

14. Generate an Embedding for the User Question

Suppose the user asks:

How many vacation days do employees get?

Generate an embedding for the question:

var question =
    "How many vacation days do employees get?";

var queryEmbedding =
    await embeddingService
        .GenerateEmbeddingAsync(question);

Now both the question and documents exist in the same embedding space.

15. Search for Relevant Documents

Perform a similarity search:

var results =
    vectorStore.Search(
        queryEmbedding,
        topK: 3);

foreach (var result in results)
{
    Console.WriteLine(
        $"Score: {result.Score:F4}");

    Console.WriteLine(
        result.Chunk.Text);

    Console.WriteLine();
}

The result might look conceptually like:

Score: 0.91

Employees receive 20 days of paid annual leave every year.

--------------------------------

Score: 0.74

Unused annual leave can be carried forward according to the company's leave policy.

--------------------------------

Score: 0.42

Managers are responsible for approving leave requests.

The highest-ranked chunks are the most relevant candidates for the answer.

16. Send the Retrieved Context to the LLM

Now we have the relevant information.

We need to provide that information to the LLM.

Create a prompt such as:

You are a helpful company policy assistant.

Answer the question using only the context provided below.

Context:
Employees receive 20 days of paid annual leave every year.

Question:
How many vacation days do employees get?

Answer:

The LLM can now generate an answer based on the retrieved context.

17. Call the Ollama Chat API

We can call the Ollama API using HttpClient.

public class OllamaChatService
{
    private readonly HttpClient _httpClient;

    public OllamaChatService(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<string> GenerateAsync(
        string prompt)
    {
        var request = new
        {
            model = "llama3.2",
            prompt = prompt,
            stream = false
        };

        var response = await _httpClient.PostAsJsonAsync(
            "http://localhost:11434/api/generate",
            request);

        response.EnsureSuccessStatusCode();

        var result =
            await response.Content.ReadFromJsonAsync<OllamaResponse>();

        return result?.Response ?? string.Empty;
    }

    private class OllamaResponse
    {
        public string Response { get; set; } =
            string.Empty;
    }
}

18. Build the Final RAG Prompt

Combine the retrieved chunks:

var context = string.Join(
    "\n\n",
    results.Select(x => x.Chunk.Text));

var prompt = $"""
You are a helpful company policy assistant.

Answer the question using only the supplied context.

If the answer cannot be found in the context,
say that the information is not available.

Context:
{context}

Question:
{question}

Answer:
""";

This gives the LLM the information retrieved from our document.

19. Generate the Final Answer

Create the chat service:

var chatService =
    new OllamaChatService(httpClient);

var answer =
    await chatService.GenerateAsync(prompt);

Console.WriteLine("Question:");
Console.WriteLine(question);

Console.WriteLine();

Console.WriteLine("Answer:");
Console.WriteLine(answer);

The application should produce an answer similar to:

Employees receive 20 days of paid annual leave every year.

20. Complete RAG Flow

We have now implemented the complete basic RAG pipeline.

company-policy.txt
        ↓
Read Document
        ↓
Split into Chunks
        ↓
Generate Embeddings
        ↓
Store Vectors
        ↓
       ┌──────────────────┐
       │                  │
       │   User Question  │
       │                  │
       └────────┬─────────┘
                ↓
        Generate Embedding
                ↓
        Cosine Similarity
                ↓
          Top-K Chunks
                ↓
        Build RAG Prompt
                ↓
              Ollama
                ↓
          Generated Answer

21. Complete Example

The following example puts the main pieces together:

using System.Net.Http.Json;

var httpClient = new HttpClient();

var embeddingService =
    new OllamaEmbeddingService(httpClient);

var chatService =
    new OllamaChatService(httpClient);

var vectorStore =
    new VectorStore();

var documentText =
    await File.ReadAllTextAsync(
        "Documents/company-policy.txt");

var chunks =
    SplitIntoChunks(documentText);

foreach (var chunkText in chunks)
{
    var embedding =
        await embeddingService
            .GenerateEmbeddingAsync(chunkText);

    vectorStore.Add(new DocumentChunk
    {
        Text = chunkText,
        Embedding = embedding
    });
}

Console.Write("Ask a question: ");

var question =
    Console.ReadLine() ?? string.Empty;

var queryEmbedding =
    await embeddingService
        .GenerateEmbeddingAsync(question);

var results =
    vectorStore.Search(
        queryEmbedding,
        topK: 3);

var context =
    string.Join(
        "\n\n",
        results.Select(x => x.Chunk.Text));

var prompt = $"""
You are a helpful company policy assistant.

Answer the question using only the supplied context.

If the answer cannot be found in the context,
say that the information is not available.

Context:
{context}

Question:
{question}

Answer:
""";

var answer =
    await chatService.GenerateAsync(prompt);

Console.WriteLine();
Console.WriteLine("Answer:");
Console.WriteLine(answer);

static List<string> SplitIntoChunks(string text)
{
    return text
        .Split(
            new[] { "\r\n\r\n", "\n\n" },
            StringSplitOptions.RemoveEmptyEntries)
        .Select(x => x.Trim())
        .Where(x => !string.IsNullOrWhiteSpace(x))
        .ToList();
}

22. What Happens When the User Asks a Question?

Suppose the user asks:

Can I carry unused leave to the next year?

The application does not simply send the question directly to the LLM.

Instead:

User Question
      ↓
Embedding
      ↓
Vector Similarity Search
      ↓
Relevant Leave Policy Chunk
      ↓
RAG Prompt
      ↓
LLM
      ↓
Answer

This is the fundamental idea behind RAG.

23. Why Not Just Send the Entire Document to the LLM?

You might ask:

Why do we need embeddings and vector search?
Why not send the entire document to the LLM?

For small documents, that may be perfectly reasonable.

However, real applications can contain:

• Thousands of documents
• Millions of paragraphs
• Large technical manuals
• Customer records
• Frequently updated information

Sending everything to the LLM for every question is inefficient and may exceed the model's context capacity.

RAG retrieves only the information that is likely to be relevant.

24. Why Chunking Matters

Imagine a document contains 100 pages.

If the entire document is treated as one vector, the search may not be precise enough.

Instead:

100 Page Document
       ↓
Chunk 1
Chunk 2
Chunk 3
...
Chunk 500
       ↓
Embedding for each chunk
       ↓
Precise Retrieval

Good chunking can significantly affect the quality of a RAG system.

25. Improving the Chunking Strategy

The simple paragraph-based chunking used in this tutorial is only a starting point.

Production systems may use:

• Fixed token-size chunks
• Sentence-based chunks
• Paragraph-based chunks
• Heading-aware chunking
• Recursive chunking
• Overlapping chunks

For example:

Chunk 1:
Employees receive 20 days of annual leave...

Chunk 2:
...annual leave can be carried forward...

Overlap:
Some content appears in both chunks.

Overlap can help preserve context between chunks, although the appropriate strategy depends on the document type and retrieval requirements.

26. Why Use a Vector Database?

Our example keeps vectors in memory:

List<DocumentChunk>
```

This is fine for learning, but it is not a production-ready storage solution.

A real application may use a vector database to provide:


  
Persistent storage
  
Fast similarity search
  
Metadata filtering
  
Scalability
  
Indexing
  
Access control





27. Production RAG Architecture

A production architecture could look like this:

              Documents
                   ↓
            Document Processor
                   ↓
               Chunking
                   ↓
            Embedding Model
                   ↓
             Vector Database
                   │
                   │
                   ↓
             Retrieval API
                   ↑
                   │
              User Query
                   ↓
             Query Embedding
                   ↓
           Similarity Search
                   ↓
             Top-K Results
                   ↓
               Reranking
                   ↓
            Prompt Construction
                   ↓
                  LLM
                   ↓
                Answer




28. RAG With a Vector Database

Instead of:

C# List
   ↓
Cosine Similarity
   ↓
Results

A production system could use:

C#
 ↓
Vector Database
 ↓
Vector Search
 ↓
Top-K Results

The application code remains conceptually similar even when the underlying storage technology changes.




29. RAG Does Not Mean Only Vector Search

A mature RAG system can combine multiple retrieval techniques.

User Query
     ↓
 ┌───┴────────────┐
 ↓                ↓
Keyword       Semantic
Search        Search
 ↓                ↓
 └───────┬────────┘
         ↓
      Ranking
         ↓
      Reranking
         ↓
    Relevant Context
         ↓
        LLM

This approach is often called Hybrid Search.




30. Common RAG Problems

A RAG application can still produce poor answers.

Common causes include:


  
Poor document extraction
  
Bad chunking
  
Weak embeddings
  
Irrelevant retrieval results
  
Incorrect Top-K value
  
Missing metadata filters
  
Poor prompts
  
Insufficient context
  
Too much irrelevant context


Building a good RAG system is therefore more than simply adding a vector database.




31. Add a Relevance Threshold

You can prevent obviously irrelevant documents from being passed to the LLM.

var results = vectorStore.Search(
    queryEmbedding,
    topK: 5);

var relevantResults =
    results
        .Where(x => x.Score >= 0.70)
        .ToList();

The exact threshold should be determined through testing. A score such as 0.70 is only an example and should not be treated as a universal cutoff.




32. Add Source Information

A useful improvement is to store metadata along with each chunk.

public class DocumentChunk
{
    public string Text { get; set; } = string.Empty;

    public float[] Embedding { get; set; } =
        Array.Empty<float>();

    public string Source { get; set; } =
        string.Empty;

    public int ChunkNumber { get; set; }
}

This allows the application to tell the user where the information came from.




33. Example with Source Citations

The final answer could be displayed as:

Employees receive 20 days of paid annual leave every year.

Source:
company-policy.txt

For enterprise applications, source references can improve transparency and make it easier for users to verify the answer.




34. Security Considerations

Production RAG systems should also consider security.

For example, if a user does not have permission to view a document, the retrieval system should not return that document to the LLM.

User
 ↓
Authentication
 ↓
Authorization
 ↓
Metadata Filtering
 ↓
Vector Search
 ↓
Allowed Documents
 ↓
LLM

Access control should therefore be part of the retrieval architecture rather than added as an afterthought.




35. RAG vs Normal LLM Application


  

    
Normal LLM
    
RAG Application
  
  

    
User question
    
User question
  
  

    
Prompt
    
Generate query embedding
  
  

    
LLM
    
Search knowledge base
  
  

    
Answer
    
Provide retrieved context to LLM
  
  

    

    
Generate answer
  





36. Key Takeaways


  
RAG combines information retrieval with LLM generation.
  
Documents are normally divided into smaller chunks.
  
Embedding models convert chunks into vectors.
  
Vectors can be stored in a vector database or other vector index.
  
User questions are also converted into embeddings.
  
Similarity search retrieves relevant information.
  
The retrieved information is added to the LLM prompt.
  
The LLM generates the final response using the retrieved context.
  
Production systems usually require better chunking, persistent storage, filtering, evaluation, and security.





Conclusion

In this tutorial, we built a simple RAG application using C# and .NET 8.

We started with a document, divided it into chunks, generated embeddings using Ollama, stored the vectors in memory, performed semantic search using cosine similarity, and finally provided the retrieved context to an LLM.

The complete concept can be summarized as:

Documents
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector Storage
    ↓
Semantic Search
    ↓
Relevant Context
    ↓
LLM
    ↓
Answer

This simple application demonstrates the core principles behind much larger production RAG systems.

Once you understand this pipeline, you can replace the in-memory vector store with a real vector database, add PDF processing, introduce hybrid search and reranking, and expose the RAG pipeline through an ASP.NET Core API.


  Next: In the next article, we will explore What Is MCP (Model Context Protocol)? and understand how AI models can interact with external tools, applications, and data sources through a standardized protocol.