Skip to content

AI Layer Overview

Key Points

  • 2026 .NET AI stack crystallized around three Microsoft pillars: Microsoft.Extensions.AI (abstractions), Microsoft Agent Framework (orchestration; GA April 2026), MCP C# SDK (tool / data exchange; v1.0 March 2026).
  • Vendor-neutral by design: same IChatClient interface works against OpenAI, Azure OpenAI, Anthropic, Google, Mistral, Ollama, Foundry-hosted models.
  • Patterns: simple chat, RAG, function calling, agents (single + orchestrated), workflows, local inference.
  • Where things live: extensions handle the call; agent framework adds memory/orchestration; MCP standardizes tool servers; vector stores (Microsoft.Extensions.VectorData) handle retrieval.
  • Senior responsibilities: cost governance, observability (GenAI OTel), prompt injection defense, evaluation pipelines, multi-vendor strategy.

The 2026 stack map

                  ┌─────────────────────────────────────────────┐
                  │           User-facing app                   │
                  │  (ASP.NET, Blazor, console, agents)         │
                  └───────────────┬─────────────────────────────┘
                                  │
            ┌─────────────────────┴──────────────────────┐
            │                                            │
            ▼                                            ▼
  ┌─────────────────┐                         ┌──────────────────┐
  │ Microsoft Agent │                         │     RAG          │
  │   Framework     │                         │  (retrieval +    │
  │ (orchestration, │                         │   generation)    │
  │  memory, tools) │                         └──────────┬───────┘
  └────────┬────────┘                                    │
           │                                             ▼
           ▼                              ┌────────────────────────┐
  ┌──────────────────┐                    │  Microsoft.Extensions  │
  │  Tool servers    │                    │      .VectorData       │
  │  (via MCP)       │◀──────────────┐    │  (Cosmos, Qdrant, ...) │
  └──────────────────┘               │    └────────────────────────┘
                                     │
            ┌────────────────────────┼─────────────────────────┐
            │                        │                         │
            ▼                        ▼                         ▼
   ┌──────────────────────────────────────────────────────────────┐
   │            Microsoft.Extensions.AI                            │
   │  IChatClient · IEmbeddingGenerator · IImageGenerator          │
   │  Function calling · Telemetry · Caching · Resilience         │
   └────────────────────────┬─────────────────────────────────────┘
                            │
       ┌────────────────────┼─────────────────────────┐
       ▼                    ▼                         ▼
  ┌─────────┐        ┌────────────┐        ┌────────────────┐
  │ Azure   │        │  OpenAI    │        │ Local (Ollama, │
  │ OpenAI  │        │  Anthropic │        │  ONNX, Phi-X)  │
  │ Foundry │        │  Google    │        │                │
  └─────────┘        └────────────┘        └────────────────┘

Pillar 1: Microsoft.Extensions.AI

The contract layer. Every higher-level feature builds on IChatClient, IEmbeddingGenerator, IImageGenerator.

builder.Services.AddSingleton<IChatClient>(sp =>
    new OpenAIClient(apiKey).AsChatClient("gpt-4o-mini")
        .AsBuilder()
        .UseFunctionInvocation()
        .UseLogging(loggerFactory)
        .UseDistributedCache(cache)
        .UseOpenTelemetry()
        .Build());

Pipeline pattern: each Use* wraps the next handler.

Pillar 2: Microsoft Agent Framework

GA April 2026. Replaces Semantic Kernel for new orchestration work and AutoGen for multi-agent.

Capabilities: - Single agents with personas, instructions, tools. - Orchestration patterns: sequential, concurrent, group chat, handoff, magentic. - Memory: persistent conversation state. - Workflows: more structured than agents (deterministic where possible). - MCP integration for tools. - A2A protocol for cross-agent communication.

Pillar 3: Model Context Protocol (MCP)

Open standard for AI tool servers. C# SDK v1.0 March 2026.

[Agent / IDE / chat] ←─ MCP protocol ─→ [Server: filesystem / git / GitHub / Slack / SQL / custom]

Build server once → reuse across Claude, ChatGPT, Cursor, your custom .NET agent. Decouples tool implementations from agents.

Pillar 4: RAG + Vector data

Microsoft.Extensions.VectorData — uniform abstraction over vector stores (Azure AI Search, Cosmos, Qdrant, pgvector, in-memory).

public class MyDoc
{
    [VectorStoreKey] public string Id { get; set; } = "";
    [VectorStoreData] public string Title { get; set; } = "";
    [VectorStoreVector(Dimensions: 1536)] public ReadOnlyMemory<float>? Embedding { get; set; }
}

Same DTO; swap providers via DI.

Vendor strategy

Default: Azure OpenAI for production (compliance, regions, integration).
Fallback / alternative: OpenAI direct, Anthropic, Google.
Edge / privacy: Ollama, Phi-X via ONNX.
Specialty: embeddings (text-embedding-3), rerankers (Cohere), vision (Claude/GPT-4o), audio (Whisper).

Multi-provider via IChatClient-based routing. See Vendor Trade-offs & Lock-In.

Cost engineering

A typical RAG response: ~3K input tokens (context) + ~500 output. At GPT-4o pricing: ~$0.01/query. At 1M queries/mo: $10K/mo.

Levers: - Cheap model default (gpt-4o-mini). - Prompt caching (Anthropic, OpenAI auto). - Response caching for identical queries. - Aggressive token budget on context. - Quantize embeddings; smaller dim (text-embedding-3-small). - Batch inference for non-realtime.

See Pricing & Cost Engineering.

Observability

OpenTelemetry GenAI semantic conventions. Microsoft.Extensions.AI emits:

gen_ai.system: "openai"
gen_ai.request.model: "gpt-4o-mini"
gen_ai.usage.input_tokens: 1230
gen_ai.usage.output_tokens: 412

Plus span events for messages.

Security

  • Prompt injection: spotlighting, output validation, Azure Content Safety Prompt Shields.
  • PII: redaction before LLM call; output scrubbing.
  • API keys: managed identity for Azure OpenAI; Key Vault otherwise.

Evaluation

Production AI needs eval pipelines:

  • Faithfulness: does answer match retrieved sources?
  • Relevance: are retrieved sources relevant?
  • Correctness: domain-specific evals.
  • Cost / latency: regression tracking.

Tools: Ragas, LangSmith, Prompt Flow, custom evals.

Common patterns

Pattern When
Chat completion Q&A, generation
RAG Grounded answers in your data
Function calling Structured tool use
Agent Multi-step reasoning + tool use
Workflow Deterministic multi-step pipeline
Multi-agent Coordinated specialists

What's deprecated

See Deprecated & Outdated.

Senior responsibilities

  1. Pick model wisely (cost / capability / region).
  2. Build vendor-portable code (IChatClient).
  3. Implement caching, retries, fallback.
  4. Structure prompts (system/user/assistant; few-shot if needed).
  5. Defend against injection.
  6. Observe — token usage, errors, latency.
  7. Eval and measure quality.
  8. Govern cost (budgets, alerts, rate limits).

Cross-references