Workflow Engines

Key Points

• Workflow engines run long-running, durable, versioned workflows. They handle persistence, retries, timers, versioning automatically.
• .NET options: Azure Durable Functions (DTFx-based, Azure-native), Temporal.io (open-source; cross-language), Workflow Core (lightweight .NET library), Elsa Workflows (visual designer).
• vs sagas: workflows are richer — branching, async waits for human input, scheduled actions days later, fan-out/fan-in. Sagas in MassTransit cover state transitions; workflows cover programs.
• Key trick: workflow code is replayable. Engine records every decision; on restart, replays to reach current state. Code must be deterministic.
• Use when: human approvals, scheduled actions, complex branches, multi-day flows, audit requirements.

Concepts (deep dive)

What's a workflow engine?

A platform that runs your workflow code as durable orchestration — surviving crashes, machine restarts, scaling. The engine persists every step's input/output.

Azure Durable Functions

[Function(nameof(OrderOrchestrator))]
public static async Task<OrderResult> OrderOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext ctx)
{
    var order = ctx.GetInput<Order>()!;

    // 1. Reserve inventory (durable activity call)
    var reserved = await ctx.CallActivityAsync<bool>(nameof(ReserveInventory), order);
    if (!reserved) return new OrderResult { Failed = "no-stock" };

    // 2. Wait for human approval (or timeout)
    var approved = await ctx.WaitForExternalEvent<bool>("Approved",
        TimeSpan.FromHours(24), defaultValue: false);
    if (!approved)
    {
        await ctx.CallActivityAsync(nameof(ReleaseInventory), order);
        return new OrderResult { Failed = "not-approved" };
    }

    // 3. Charge & ship
    await ctx.CallActivityAsync(nameof(ChargePayment), order);
    await ctx.CallActivityAsync(nameof(ScheduleShipping), order);
    return new OrderResult { Success = true };
}

[Function(nameof(ReserveInventory))]
public static Task<bool> ReserveInventory([ActivityTrigger] Order order)
    => Task.FromResult(true);   // real impl

The orchestrator runs on TaskOrchestrationContext. Every await is a checkpoint — the engine persists state, can be replayed.

Replay model

Run 1: do A → record result → crash
Run 2: replay (recall result of A) → do B → record → wait

The orchestrator's code re-executes from the top. Every previous decision is recalled from history. Side effects only happen via CallActivityAsync — those are recorded.

Implication: orchestrator code must be deterministic: - ❌ DateTime.UtcNow — different on each replay. - ❌ Random — non-deterministic. - ❌ await new HttpClient().GetAsync(...) — bypasses engine. - ✅ await ctx.CallActivityAsync(...) — recorded. - ✅ ctx.CurrentUtcDateTime — deterministic; same on replay.

Temporal.io

Open-source, cross-language (Go, Java, Python, .NET, TypeScript). Same replay model as Durable Functions; richer features.

[Workflow]
public class OrderWorkflow
{
    [WorkflowRun]
    public async Task<OrderResult> RunAsync(Order order)
    {
        await Workflow.ExecuteActivityAsync(
            (Activities a) => a.ReserveInventoryAsync(order),
            new ActivityOptions { StartToCloseTimeout = TimeSpan.FromMinutes(5) });

        var approved = await Workflow.WaitConditionAsync(() => _approved,
            TimeSpan.FromHours(24));

        // ...
    }

    [WorkflowSignal]
    public Task ApproveAsync() { _approved = true; return Task.CompletedTask; }
}

Temporal has signals (events into a workflow), queries (read state), child workflows, continue-as-new (restart with fresh history when too long).

Workflow Core

Lightweight, no engine — just a .NET library. State persisted to your DB. Smaller than Temporal/DF.

public class OrderWorkflow : IWorkflow<OrderData>
{
    public string Id => "order-flow";
    public int Version => 1;

    public void Build(IWorkflowBuilder<OrderData> b)
    {
        b.StartWith<ReserveInventory>()
            .Input(s => s.Order, d => d.Order)
            .Then<ChargePayment>()
            .Then<ScheduleShipping>();
    }
}

Good for simpler needs without an external engine.

Elsa Workflows

Visual designer + .NET runtime. JSON-defined or code-first. Aimed at low-code scenarios; product teams design flows.

When workflow engine vs saga

Versioning

Long-running workflows started on v1 of code may need to finish under v2.

Strategies: - Side-by-side: keep v1 code; new instances use v2. - Patch points: if (Workflow.Patched("v2-fix")) { newPath } else { oldPath } — Temporal pattern. - continue-as-new: long workflows split into chunks; restart with fresh history at boundaries.

This is the hard part of workflow engines.

Activities

Where side effects happen. The orchestrator calls activities; activities are normal code.

[Function(nameof(ChargePayment))]
public static async Task ChargePayment([ActivityTrigger] Order order)
{
    // Real HTTP call to Stripe — non-deterministic; that's OK; orchestrator records the result.
    await _stripe.ChargeAsync(...);
}

Activities can fail; the engine retries with policy.

Retries

new ActivityOptions
{
    RetryPolicy = new RetryPolicy
    {
        MaxAttempts = 5,
        InitialInterval = TimeSpan.FromSeconds(1),
        BackoffCoefficient = 2.0
    }
}

Engine handles retry; orchestrator code is unaware.

Sub-workflows / fan-out

var tasks = orderIds.Select(id => ctx.CallSubOrchestratorAsync<bool>(nameof(ProcessOne), id));
var results = await Task.WhenAll(tasks);

Workflow engines optimized for parallel sub-workflows.

When NOT a workflow engine

• Simple request-response: just RPC.
• Pure event-driven, no state: choreography.
• Short-lived (seconds): may be overkill.
• Tight latency requirements: engines add overhead.

Cost / operational

• Durable Functions: managed in Azure; pay per execution. Scales auto.
• Temporal: self-hosted (Postgres + Cassandra) or cloud (Temporal Cloud). Operational complexity.
• Workflow Core / Elsa: lightweight; runs in your app.

Observability

Workflow engines have first-class workflow inspection — see every step, input/output, retries, timing. Vastly better than reverse-engineering from logs.

Idempotency

Activities can be retried. Use idempotency keys (workflow ID + step number) to prevent duplicate side effects.

Code: correct vs wrong

❌ Wrong: non-deterministic in orchestrator

public async Task RunAsync(Order order, TaskOrchestrationContext ctx)
{
    var now = DateTime.UtcNow;   // changes on replay!
    if (now > order.Deadline) ...
}

✅ Correct: use the context's clock

var now = ctx.CurrentUtcDateTime;   // deterministic

❌ Wrong: HTTP call in orchestrator

public async Task RunAsync(...) { var r = await _http.GetAsync(...); }

Bypasses replay; results not recorded.

✅ Correct: call activity

var r = await ctx.CallActivityAsync<Response>(nameof(MakeHttpCall), input);

Design patterns for this topic

Pattern 1 — "Replay-safe orchestrator"

• Intent: deterministic; side effects via activities.

Pattern 2 — "Wait for external event with timeout"

• Intent: human approvals, async signals.

Pattern 3 — "Sub-workflows for fan-out"

• Intent: parallel work units.

Pattern 4 — "continue-as-new for long workflows"

• Intent: bounded history.

Pattern 5 — "Versioning via patch points"

• Intent: evolve workflow code safely.

Pros & cons / trade-offs

When to use / when to avoid

• Use for human approvals, scheduled, multi-day flows.
• Use when audit + observability of workflow critical.
• Avoid for short-lived, simple flows.
• Avoid for hot-path low-latency.

Interview Q&A

Q1. What's a workflow engine? Platform running durable orchestrations — survives crashes; persists state; replays from history.

Q2. Why must orchestrator code be deterministic? Engine replays the orchestrator on restart. Non-deterministic code (DateTime.Now, Random) gives different results on replay → state corruption.

Q3. Saga vs workflow? Saga: state transitions on events. Workflow: programs with branches/loops/timers/human waits. Workflows are richer.

Q4. Why .NET activities can use anything? Activities are recorded — engine remembers their result. Non-deterministic code OK there.

Q5. continue-as-new? Long workflow's history grows. continue-as-new restarts the workflow with fresh history, carrying state.

Q6. Versioning workflows? Patch points (Workflow.Patched); side-by-side versions; continue-as-new at safe boundaries.

Q7. Wait for external event? ctx.WaitForExternalEvent with timeout. Used for human approvals, async signals.

Q8. Durable Functions vs Temporal? DF: managed in Azure; integrated with .NET. Temporal: self-hosted or cloud; cross-language; richer features.

Q9. Activity retry policy? Configured in ActivityOptions — max attempts, backoff. Engine handles automatically.

Q10. Why workflow over saga? Branching, conditional, multi-day, human input. Sagas are more constrained.

Q11. Workflow engine cost? Operational (running engine), per-execution (managed), state storage. Significant.

Q12. Side effect in orchestrator? Don't. Side effects only in activities — recorded for replay.

Gotchas / common mistakes

• ⚠️ Non-deterministic in orchestrator — replay corruption.
• ⚠️ HTTP / DB call in orchestrator — bypass engine.
• ⚠️ No versioning strategy — old workflows break.
• ⚠️ Unbounded history — without continue-as-new, performance degrades.
• ⚠️ Treating engine as message bus — wrong tool.

Further reading

• Azure Durable Functions
• Temporal.io docs
• Workflow Core
• Elsa Workflows