Local & Edge Inference
Running models locally — Phi-3/Phi-4 SLMs via ONNX Runtime GenAI for latency-sensitive workloads, Ollama for developer ergonomics, plus the perf trade-offs between them.
Topics (canonical order)
-
Running Phi-3/Phi-4 with ORT GenAI from .NET, DirectML/CUDA, token throughput
-
Ollama from
Microsoft.Extensions.AI.Ollama, Aspire integration, local development -
Perf Trade-offs — ONNX vs Ollama
When ONNX beats Ollama (latency-sensitive prod), when Ollama wins (DX), benchmarking guidance
Why this order
ONNX first (production path), Ollama next (dev path), trade-offs last (decision capstone).
Cross-references
- Phi models discussed in Microsoft Phi & Small Models.
- IChatClient over Ollama in IChatClient & Pipeline.
- Aspire integration in Aspire — Cloud-Native.