Case: RAG App on .NET

Problem

Design a RAG (Retrieval-Augmented Generation) app: users ask questions; system retrieves relevant docs from a knowledge base (1M+ documents, 100M+ chunks); injects into LLM prompt; returns grounded answer. Sub-2s latency; citations; multi-tenant.

Walkthrough

Clarify

1M docs, ~100M chunks (text + embeddings).
100K queries/day; up to 10/s peak.
Multi-tenant: docs per tenant; strict isolation.
Citations: which docs answered.
Streaming response (token-by-token).
Update freshness: docs added/updated continuously; sync within 1h.

Architecture

[Client]
  │ /chat with question
  ▼
[Chat API (.NET)]
  │
  ├──→ [Embedding] (Microsoft.Extensions.AI: IEmbeddingGenerator)
  │       └─ generates query vector
  │
  ├──→ [Vector store search]
  │       └─ Cosmos DB / Qdrant / Azure AI Search → top-k chunks
  │
  ├──→ [Reranker] (optional; Cohere Rerank / cross-encoder)
  │       └─ refines top-k → top-3
  │
  ├──→ [Build prompt] with retrieved context + citations
  │
  ├──→ [LLM stream] (IChatClient → OpenAI / Anthropic / ...)
  │       └─ streamed answer
  │
  └──→ [Audit log] (query, retrieved IDs, response, tokens, cost)

Ingestion (offline):
[Source docs] → [Chunker] → [Embedder] → [Vector store + metadata DB]

Microsoft.Extensions.AI

builder.Services.AddSingleton<IChatClient>(sp =>
    new OpenAIClient(apiKey).AsChatClient("gpt-4o-mini"));

builder.Services.AddSingleton<IEmbeddingGenerator<string, Embedding<float>>>(sp =>
    new OpenAIClient(apiKey).AsEmbeddingGenerator("text-embedding-3-small"));

Vendor-neutral abstractions. Swap provider with config.

Ingestion pipeline

public async Task IngestAsync(Document doc, CancellationToken ct)
{
    var chunks = _chunker.Chunk(doc.Content, maxTokens: 500, overlap: 50);

    var embeddings = await _embedGen.GenerateAsync(chunks.Select(c => c.Text).ToArray(), ct);

    var records = chunks.Zip(embeddings, (c, e) => new VectorRecord
    {
        Id = $"{doc.Id}:{c.Index}",
        TenantId = doc.TenantId,
        DocumentId = doc.Id,
        ChunkIndex = c.Index,
        Text = c.Text,
        Embedding = e.Vector
    });

    await _vectorStore.UpsertBatchAsync(records, ct);
}

Chunkers: simple text split, semantic, sentence-window. SemanticSimilarityChunker in MS.Extensions.AI for content-aware chunking.

Chunking strategy

Doc → split by section/headings → split paragraphs → 500-token windows
       ↑                                              ↑
     primary boundaries                             with 50-token overlap

Trade-offs: - Smaller chunks: more precise retrieval; more rows; more embeddings cost. - Larger chunks: more context; less precise. - 300-700 tokens typical.

Vector store choice

Store	Pros	Cons
Azure AI Search	Hybrid (vector + keyword); managed	Azure-only
Cosmos DB	Multi-model; .NET 9 vector support	Newer
Qdrant	Open-source; performant	Self-hosted or cloud
pgvector (Postgres)	Same DB as relational	Limited filtering
Pinecone	Managed; mature	Vendor

For .NET + Azure: Azure AI Search (hybrid) or Cosmos DB (collocated with app data).

Hybrid search

Vector + keyword for best results:

score = α * vector_similarity + β * BM25

Azure AI Search does this natively.

Reranking

var topK = await _vectorStore.SearchAsync(queryEmbedding, k: 50);
var reranked = await _reranker.RerankAsync(query, topK.Select(x => x.Text), top_n: 5);

Reranker (Cohere Rerank, cross-encoder model) refines initial top-k. Improves citation quality.

Prompt construction

var prompt = $$"""
You are an assistant. Answer the question using ONLY the provided context.
If the context doesn't contain the answer, say "I don't know".
Cite each fact with [Doc-N].

Context:
{{string.Join("\n\n", reranked.Select((c, i) => $"[Doc-{i+1}] {c.Text}"))}}

Question: {{query}}

Answer:
""";

var response = _chatClient.GetStreamingResponseAsync(prompt, cancellationToken: ct);
await foreach (var chunk in response)
    yield return chunk.Text ?? "";

Streaming response

app.MapGet("/chat", async (string q, RagService rag, HttpContext ctx) =>
{
    ctx.Response.ContentType = "text/event-stream";
    await foreach (var token in rag.AnswerStreamAsync(q, ctx.RequestAborted))
    {
        await ctx.Response.WriteAsync($"data: {JsonSerializer.Serialize(new { token })}\n\n", ctx.RequestAborted);
        await ctx.Response.Body.FlushAsync(ctx.RequestAborted);
    }
});

SSE for token streaming.

Multi-tenancy

var results = await _vectorStore.SearchAsync(
    queryEmbedding,
    filter: $"tenantId eq '{tenantId}'",
    k: 50,
    ct);

Pre-filter by tenant. Critical for isolation.

Cost optimization

Embedding: $0.02/M tokens; cache repeated queries.
LLM: $0.15-15/M tokens depending on model.
  - Use cheap model (gpt-4o-mini, Claude Haiku) for general queries.
  - Promote to expensive model only for high-value / complex.
Vector store: per RU/query; quantization shrinks storage.

Caching

Query → answer cache with TTL: identical question → cached response.
Document embeddings: never re-embed unchanged docs (hash-based).
Reranker results: cached when query identical.

Observability

OTel GenAI semantic conventions:

activity?.SetTag("gen_ai.system", "openai");
activity?.SetTag("gen_ai.request.model", "gpt-4o-mini");
activity?.SetTag("gen_ai.usage.input_tokens", in);
activity?.SetTag("gen_ai.usage.output_tokens", out);

Track: - Tokens in/out per query. - Cost per query. - Latency: embed → search → rerank → LLM. - Citation count + faithfulness (rate of "ungrounded" answers).

Quality measurement

Faithfulness: does answer cite actual sources?
Relevance: are retrieved chunks relevant?
Precision@K: top-K retrieval quality.

Use eval frameworks (Ragas; LangSmith; manual labels).

Prompt injection defense

User input: "Ignore previous instructions and..." — guard with: - System prompt clarity. - Spotlighting (mark user input distinctly). - Output validation. - Azure Content Safety Prompt Shields.

Rate limiting + abuse

.RequireRateLimiting("user-tier-rate")

Per-tenant per-user limits. Track abuse via cost spikes.

Failure modes

LLM API down: fall back to second provider via IChatClient routing.
Vector store slow: timeout + serve cached / "system busy" response.
Embedding API spike: queue + backpressure.

Trade-offs

Choice	Why	Trade-off
MS.Extensions.AI	Vendor-neutral	New abstraction layer
Hybrid search	Best quality	More complex
Reranker	Better top-K	Extra latency
Streaming	Better UX	Harder to cache
Cheap LLM default	Cost	Sometimes wrong

What we'd skip

Fine-tuning: RAG handles most cases; only fine-tune for specific domain language.
Local LLM for primary: too expensive perf-wise unless privacy critical.
Custom vector DB: managed Azure AI Search / Cosmos cheaper than self-hosting.

What we'd add for scale

Caching layer: Redis with semantic cache.
Multi-model routing: cheap → expensive based on query complexity.
A/B testing: variants on chunking/prompt.
Background jobs: regular re-embedding / index rebuild.

Cross-references

This case bridges into the AI / LLM Integration section. See: