Azure Monitor & Application Insights — Deep

Companion to Azure Monitor & Application Insights. This topic goes deeper on sampling internals, KQL for incident analysis, profiler/snapshot debugger, distributed tracing semantics, and cost control — the operational levers seniors actually pull.

Key Points

• Workspace-based App Insights is the modern path. Resources unify under a Log Analytics workspace; cross-resource KQL is a first-class capability.
• Three sampling layers: SDK-side adaptive (default — drops at source to save cost), SDK-side fixed-rate (deterministic), and server-side ingestion sampling (after data lands — most flexible, but you've already paid).
• Tables are the API: requests, dependencies, traces, exceptions, pageViews, customEvents, customMetrics, availabilityResults, browserTimings, performanceCounters. Joins on operation_Id are the workhorse.
• Distributed tracing uses W3C Trace Context (traceparent + tracestate headers). The legacy Request-Id / Request-Context headers are deprecated.
• Connection string > instrumentation key. CS supports regional ingest endpoints, sovereign clouds, and Live Metrics auth.
• Cost = ingestion volume × retention. Daily cap, sampling, and basic-logs/archive tiers are the three knobs.
• Profiler + Snapshot Debugger give prod-grade post-mortem: sampled CPU profiles + call stack & locals on uncaught exceptions.
• Workbooks > legacy Dashboards for parameterized, share-able reports.

Concepts (deep dive)

The base Azure Monitor & Application Insights topic covers the "what does it do" surface. This file is the operations layer: how sampling actually decides which spans get dropped (and what that costs you in lost outliers), how distributed tracing propagates across services via W3C Trace Context, how to find the slow query when an incident is live, and how to keep ingestion costs under control once volume becomes real. Every section below answers a question a senior gets asked at 2 AM during an incident.

Workspace-based App Insights

Old model: each AI resource had its own data store. Cross-resource queries were painful.

Modern model:

┌─────────────── Log Analytics Workspace ───────────────┐
│                                                       │
│  ┌──────────────┐  ┌──────────────┐  ┌─────────────┐  │
│  │ AppInsights  │  │ AppInsights  │  │ Diagnostics │  │
│  │ (Web API)    │  │ (Worker)     │  │ (KeyVault,  │  │
│  │              │  │              │  │  Cosmos…)   │  │
│  └──────────────┘  └──────────────┘  └─────────────┘  │
│                                                       │
│  Cross-resource KQL queries work across all of these  │
└───────────────────────────────────────────────────────┘

When you create AI in the portal, "workspace-based" is now the only option. Migrate classic resources via the portal (one-click).

Sampling — three layers, three trade-offs

[App Process]                    [Ingest Endpoint]            [Workspace]
   │                                  │                          │
   │ Adaptive sampling (SDK)          │ Ingestion sampling       │
   │ Fixed-rate sampling (SDK)        │ (server-side)            │
   │                                  │                          │
   ●──────── send (or drop) ─────────▶●───── store (or drop) ───▶●
   ▲                                  ▲
   │ Saves cost AND egress            │ Saves cost only
   │ Loses fidelity for outliers      │ Full sample available pre-decision

// Adaptive (OTel distro default)
builder.Services.AddOpenTelemetry().UseAzureMonitor();

// Fixed-rate at 10%
builder.Services.AddOpenTelemetry().UseAzureMonitor(o =>
{
    o.SamplingRatio = 0.1f;
});

💡 Errors and dependencies-with-failures are usually preserved by adaptive sampling — it correlates by operation_Id so you don't get half-traces.

Tables — what's where

KQL deep — incident-grade queries

Joins across tables by operation_Id:

// Failed requests with their exceptions and slowest dependency
let failures =
    requests
    | where timestamp > ago(1h) and success == false
    | project timestamp, name, operation_Id, duration;
let exns =
    exceptions
    | project operation_Id, exType=type, exMsg=outerMessage;
let slowDeps =
    dependencies
    | summarize maxDepDur=max(duration), depTarget=any(target) by operation_Id;
failures
| join kind=leftouter exns on operation_Id
| join kind=leftouter slowDeps on operation_Id
| project timestamp, name, duration, exType, exMsg, depTarget, maxDepDur
| order by duration desc

make-series for visualization-friendly metrics:

requests
| where timestamp > ago(24h)
| make-series p95=percentile(duration, 95) default=0
    on timestamp from ago(24h) to now() step 5m
    by name
| render timechart

summarize percentiles():

requests
| where timestamp > ago(1h)
| summarize percentiles(duration, 50, 95, 99, 99.9) by name
| order by percentile_duration_99 desc

mv-expand to fan out arrays / nested fields:

exceptions
| where timestamp > ago(1d)
| extend frames = parse_json(details)
| mv-expand frame=frames
| extend method = tostring(frame.parsedStack[0].method)
| summarize count() by method
| top 10 by count_

parse_json for customDimensions:

traces
| where message has "OrderPlaced"
| extend props = todynamic(customDimensions)
| extend orderId = tostring(props.OrderId), amount = todouble(props.Amount)
| summarize sum(amount) by bin(timestamp, 1h)

Live Metrics — sub-second observability

.UseAzureMonitor()   // Live Metrics on by default

Live Metrics streams un-sampled telemetry directly from the SDK to the portal pane — bypasses ingestion entirely. Use during deploys, incidents, load tests. Latency: ~1 second.

⚠️ Live Metrics requires the SDK to authenticate to the portal session. Connection string with regional endpoints handles this; bare instrumentation key may not.

Application Map

Auto-built from operation_Id correlation across services:

[Frontend SPA]
     │
     ▼
[Web API (200ms p95, 0.2% errors)]
     │
     ├──▶ [SQL: orders DB (45ms p95)]
     ├──▶ [Cosmos: catalog (12ms p95)]
     └──▶ [Service Bus: orders queue]
                │
                ▼
            [Worker (worker-prod)]
                │
                └──▶ [SQL: orders DB]

The map is generated from dependencies + cross-component requests data — no manual config.

Profiler & Snapshot Debugger

Profiler = sampling CPU profiler in production.

// Web app: enable in App Service blade
// Self-host: install Microsoft.ApplicationInsights.Profiler.AspNetCore
builder.Services.AddServiceProfiler();

Triggers: scheduled (2 min every hour), CPU spike, memory spike, manual. Output: function-level CPU breakdown, ETW-based, low overhead (~5%).

Snapshot Debugger captures the call stack + local variables when an uncaught exception fires.

builder.Services.AddSnapshotCollector();

Inspect the snapshot in Visual Studio — like a post-mortem dump but with locals. Limited captures per app per day.

✅ Use both in prod: profiler for perf regression hunts; snapshot for "we can't reproduce this exception".

Workbooks vs Dashboards

Workbook structure:
  ─ Parameters (TimeRange, Service, Severity)
  ─ Markdown header
  ─ KQL: requests overview (chart)
  ─ KQL: top errors (grid → click → drill to next workbook)
  ─ KQL: dependencies map

Alerts deep

Multi-resource log alerts: one rule scans many AppInsights / Log Analytics resources at once — essential for fleet ops.

// Multi-resource alert query
union workspace("ws-prod-*").requests
| where success == false
| summarize fails=count() by _ResourceId
| where fails > 100

Action groups: email, SMS, voice, webhook, Logic App, ITSM, Azure Function, Event Hubs.

Cost management

The two cost drivers:

Cost = Ingestion (GB) × $/GB  +  Retention (days beyond 30) × $/GB-day

Knobs:

• Sampling (SDK) — biggest single lever. 25% sampling = 75% cost cut.
• Daily cap — hard ceiling on workspace ingestion. Telemetry over the cap is dropped, not queued. Set with alarms.
• Retention — 30 days included; archive tier is ~10× cheaper for long-term.
• Basic Logs / Auxiliary Logs — cheaper ingestion for noisy tables (with query restrictions).
• Per-table retention — set short retention for chatty dependencies, long for exceptions.

⚠️ A daily cap protects against runaway cost but creates an observability blackout when hit. Always alert at 80% of cap.

Distributed tracing — W3C Trace Context

Modern default. Two HTTP headers:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ││ ────────────trace-id───────────── ─parent-span-id── ││
             │└─ trace flags (sampled)            
             └── version

tracestate: vendor1=value1,vendor2=value2

operation_Id = trace-id. operation_ParentId = parent span. AppInsights joins everything by these.

Legacy Request-Id / Request-Context (|guid.span_) is still understood for backward-compat but deprecated. New SDKs emit W3C only.

// OTel auto-propagates W3C across HttpClient, gRPC, Service Bus, EF, etc.
builder.Services.AddOpenTelemetry().UseAzureMonitor();

Connection string vs instrumentation key

Connection string (modern):

InstrumentationKey=<guid>;
IngestionEndpoint=https://<region>.in.applicationinsights.azure.com/;
LiveEndpoint=https://<region>.livediagnostics.monitor.azure.com/;
ApplicationId=<app-id>

Why CS over IKey-only: - ✅ Regional ingest endpoint (sovereign clouds, data residency). - ✅ Live Metrics endpoint baked in. - ✅ Application ID for app-level auth scenarios. - ✅ Azure roadmap — IKey-only is being phased out.

// Pull from Key Vault / config; never hardcode
.UseAzureMonitor(o => o.ConnectionString = builder.Configuration["AppInsights:ConnectionString"]);

How it works under the hood

┌──── App Process ────┐
│ OTel SDK            │
│  - sampling         │
│  - context propagat │
│  - exporters        │
└──────────┬──────────┘
           │ OTLP / AppInsights ingestion API
           ▼
┌──── Regional Ingest Endpoint ────┐
│  - auth / rate limit             │
│  - server-side sampling (opt)    │
└──────────┬───────────────────────┘
           ▼
┌──── Log Analytics Workspace ─────┐
│  Tables: requests / deps / etc   │
│  Indexed for KQL                 │
└──────┬─────────────────┬─────────┘
       │                 │
       ▼                 ▼
┌── KQL queries ──┐  ┌── Alerts ──┐
│  Workbooks      │  │  rules     │
│  Dashboards     │  │  → Action  │
│  Live Metrics ─►│  │    Groups  │
└─────────────────┘  └────────────┘

Live Metrics is a separate channel — the SDK opens a long-poll to the Live endpoint and streams un-sampled samples directly. Bypasses Log Analytics ingestion entirely.

Code: correct vs wrong

❌ Wrong: instrumentation key only

.UseAzureMonitor(o => o.ConnectionString = "InstrumentationKey=xxxx");

Works, but no regional endpoint, no Live Metrics auth.

✅ Correct: full connection string

.UseAzureMonitor(o => o.ConnectionString = builder.Configuration["AppInsights:ConnectionString"]);

❌ Wrong: high-cardinality custom dimension

_logger.LogInformation("Request {RequestId} for {UserId}", requestId, userId);
// userId × requestId → millions of unique series in customMetrics if used as a metric tag

✅ Correct: bounded cardinality

_logger.LogInformation("Request for tier {Tier}", tier);   // tier ∈ {free, basic, premium}

❌ Wrong: no sampling at high traffic

Burst from 100 RPS to 10K RPS → 100× ingestion → bill spike → daily cap → blackout.

✅ Correct: adaptive sampling on

.UseAzureMonitor();  // adaptive sampling default-on

❌ Wrong: KQL without time bound on big workspace

requests | where success == false | count

Scans everything. Slow + expensive.

✅ Correct: time-bounded

requests | where timestamp > ago(1h) and success == false | count

Design patterns for this topic

Pattern 1 — "Workspace-based AI + diagnostic settings"

• Intent: unified KQL across app + Azure resources.
• How: route Cosmos/KV/Storage diag logs into the same workspace.

Pattern 2 — "Adaptive sampling + Live Metrics for incidents"

• Intent: save cost during normal ops; full fidelity during deploys.
• How: adaptive default; watch Live Metrics during rollouts.

Pattern 3 — "Multi-resource log alerts for fleets"

• Intent: one rule for many services.
• How: union workspace("…").requests query.

Pattern 4 — "Workbooks as runbooks"

• Intent: parameterized incident-response notebook.
• How: workbook with TimeRange + Service params; version-controlled JSON.

Pattern 5 — "Profiler + Snapshot for hard-to-repro"

• Intent: prod post-mortems with locals.
• How: enable both; review snapshots in Visual Studio.

Pros & cons / trade-offs

When to use / when to avoid

• Use workspace-based AI for new apps. Always.
• Use adaptive sampling unless you need 100% fidelity (compliance, low traffic).
• Use Workbooks for any dashboard that has parameters or drill-down.
• Use Profiler in prod for ongoing perf tracking.
• Avoid classic AI (non-workspace) for greenfield.
• Avoid tagging customMetrics with high-cardinality fields.
• Avoid unbounded KQL queries — always include timestamp > ago(...).

Interview Q&A

Q1. Workspace-based vs classic App Insights? Workspace-based is the modern path: data lands in Log Analytics, enabling cross-resource KQL. Classic is single-resource and being deprecated.

Q2. Sampling layers? Adaptive (SDK, default, target rate, drops at source), fixed-rate (SDK, deterministic), ingestion (server-side, after data lands — most flexible but already paid egress).

Q3. Why connection string over instrumentation key? Regional ingest endpoints, Live Metrics endpoint, sovereign cloud support. IKey-only is legacy.

Q4. W3C Trace Context vs Request-Id? W3C is the modern open standard (traceparent/tracestate). Request-Id/Request-Context is the legacy AppInsights format. New SDKs emit W3C; both are accepted on ingest.

Q5. KQL join across tables? Common: requests | join (exceptions) on operation_Id — all signals for one logical operation correlate by operation_Id.

Q6. summarize vs make-series? summarize aggregates into rows; make-series produces evenly-spaced time-series suitable for render timechart.

Q7. Profiler vs Snapshot Debugger? Profiler = sampled CPU profile in prod. Snapshot = call stack + locals on uncaught exception. Different tools for different questions.

Q8. How to control AppInsights cost? Sampling (biggest), daily cap, retention tier (basic logs / archive), per-table retention.

Q9. Daily cap risk? Telemetry over cap is dropped — observability blackout. Alert at 80% of cap.

Q10. Live Metrics — how is it different from logs? Separate channel; un-sampled; sub-second; bypasses ingestion. Used during deploys/incidents.

Q11. Multi-resource alerts? One log alert can scan many workspaces / AppInsights resources via union workspace(...). Essential at scale.

Q12. Workbooks vs Dashboards? Workbooks: parameterized, drill-down, version-controlled JSON. Dashboards: pinned tiles, no parameters. Use Workbooks.

Gotchas / common mistakes

• ⚠️ Classic (non-workspace) AI — migrate; cross-resource KQL is worth it.
• ⚠️ High-cardinality dimensions — series explosion; metric becomes useless and expensive.
• ⚠️ No daily cap — runaway bill on traffic burst.
• ⚠️ Daily cap with no headroom alert — silent blackout.
• ⚠️ KQL without time bound — scans entire retention; slow + costs query units.
• ⚠️ Live Metrics off — you lose the best deploy-time tool.
• ⚠️ IKey-only connection — no regional endpoint, no Live Metrics auth.
• ⚠️ Sampling rate misunderstood — adaptive ≠ fixed. Don't assume "10%".

Further reading

• MS Learn: Workspace-based AI
• MS Learn: Sampling in App Insights
• MS Learn: KQL reference
• MS Learn: Profiler
• MS Learn: Snapshot Debugger
• MS Learn: Workbooks
• W3C Trace Context spec
• MS Learn: Cost optimization for Azure Monitor