Binary Serialization

Key Points

• Choosing wire formats: JSON for human-readable; Protobuf for cross-language strict schemas; MessagePack for JSON-like with binary perf; Avro for schema-evolution-heavy streaming.
• Don't use BinaryFormatter — banned in .NET 9+. Insecure deserialization.
• Performance ranking (rough): MessagePack (with code-gen) ≈ Protobuf > MemoryPack > Bond > Avro > JSON > XML > BinaryFormatter.
• MemoryPack (Cysharp) — .NET-only; fastest in raw .NET benchmarks; zero-encoding for blittable types.
• System.Text.Json with source-gen is fast enough for most APIs — don't optimize prematurely.

Concepts (deep dive)

Format comparison

Protobuf

message Order {
    int64 id = 1;
    string customer = 2;
    repeated Line lines = 3;
}
message Line { int32 sku = 1; int32 quantity = 2; }

// Generated code:
var order = new Order { Id = 1, Customer = "Alice" };
order.Lines.Add(new Line { Sku = 100, Quantity = 2 });
byte[] bytes = order.ToByteArray();
var clone = Order.Parser.ParseFrom(bytes);

Strengths: tiny, fast, schema-driven, polyglot, mature ecosystem.

Use cases: gRPC, cross-language messaging, persistent stores.

MessagePack

[MessagePackObject]
public class Order
{
    [Key(0)] public long Id { get; set; }
    [Key(1)] public string Customer { get; set; } = "";
    [Key(2)] public List<Line> Lines { get; set; } = new();
}

byte[] bytes = MessagePackSerializer.Serialize(order);
var clone = MessagePackSerializer.Deserialize<Order>(bytes);

Strengths: JSON-like model; smaller; blazing fast with ContractlessStandardResolver.

Use cases: SignalR (default for new projects), Redis caching, RPC.

MemoryPack (Cysharp, 2022)

[MemoryPackable]
public partial class Order
{
    public long Id;
    public string Customer = "";
    public List<Line> Lines = new();
}

byte[] bytes = MemoryPackSerializer.Serialize(order);
var clone = MemoryPackSerializer.Deserialize<Order>(bytes);

Strengths: source-gen; fastest .NET serializer; zero-encoding (memcpy) for blittable types; IBufferWriter<byte> integration.

Use cases: high-perf .NET-to-.NET. Not for cross-language.

Avro

// Apache.Avro
var schema = Schema.Parse("...");
var writer = new GenericDatumWriter<GenericRecord>(schema);

Strengths: schema evolution (rich rules: add fields with defaults, alias, etc.); Kafka-friendly; Confluent Schema Registry integration.

Use cases: Kafka streams, event sourcing.

When each

System.Text.Json source-gen

[JsonSerializable(typeof(Order))]
public partial class AppJsonContext : JsonSerializerContext { }

JsonSerializer.Serialize(order, AppJsonContext.Default.Order);

Compile-time codegen; AOT-friendly; faster; no reflection.

For most APIs: STJ source-gen is plenty fast. Don't switch to MessagePack just for perf.

Schema evolution

Protobuf

Add fields with new numbers; old code ignores. Don't reuse numbers. Mark as optional (default in proto3) — missing field = default value.

Avro

Reader/writer schema both. Fields with default values can be added. Aliases for renaming. Schema registry coordinates evolution.

JSON

No schema → no enforcement → easy to break clients. JSON Schema or OpenAPI helps but isn't enforced at parse time.

Versioning strategies

1. Version in payload: {"v":2, ...}. Code branches.
2. Version in URL/topic: /api/v2/orders, orders.placed.v2.
3. Version in content type: application/vnd.myapp.v2+json.
4. Schema-driven evolution: Protobuf field numbers; Avro defaults.

Compression

For large payloads, compression helps regardless of format:

// SignalR
.AddMessagePackProtocol(options =>
{
    options.SerializerOptions = MessagePackSerializerOptions.Standard
        .WithCompression(MessagePackCompression.Lz4BlockArray);
});

Why NOT BinaryFormatter

• Insecure deserialization — attacker-crafted blob can run arbitrary code.
• Bloated output.
• Reflection-based; slow.
• Removed/banned in .NET 9+.

If you find legacy BinaryFormatter, replace with JSON / Protobuf / MessagePack.

ProtoBuf-net (Marc Gravell)

Protobuf for .NET that maps directly to .NET classes:

[ProtoContract]
public class Order
{
    [ProtoMember(1)] public long Id { get; set; }
    [ProtoMember(2)] public string Customer { get; set; } = "";
}

Serializer.Serialize(stream, order);

Mature; convenient for code-first.

Binary in HTTP

var content = new ByteArrayContent(MessagePackSerializer.Serialize(order));
content.Headers.ContentType = new("application/x-msgpack");
await http.PostAsync(url, content);

Server reads from request body stream; deserializes.

Custom serialization in EF / cache

EF Core 8+ supports JSON columns. Cache (Redis) commonly uses JSON or MessagePack. Pick once per project; consistent.

Performance anecdotes

Stephen Toub benchmarks (rough): - JSON: 100µs/object (baseline). - STJ source-gen: 30–50µs. - MessagePack: 10–20µs. - MemoryPack: 5µs.

For most APIs, even JSON is fine — bandwidth/IO dominates serialization.

Code: correct vs wrong

❌ Wrong: BinaryFormatter

new BinaryFormatter().Serialize(stream, obj);

✅ Correct: JSON or MessagePack

JsonSerializer.Serialize(stream, obj);

❌ Wrong: untyped JSON in hot path

JsonSerializer.Serialize(obj);   // reflection per call

✅ Correct: source-gen

JsonSerializer.Serialize(obj, AppJsonContext.Default.MyType);

❌ Wrong: Protobuf field renumbering

message Order {
    int64 customer_id = 1;   // was id
}

Field number 1 was id previously — old serialized data corrupts.

Design patterns for this topic

Pattern 1 — "STJ source-gen for APIs"

• Intent: fast enough; AOT.

Pattern 2 — "Protobuf for gRPC"

• Intent: native fit.

Pattern 3 — "MessagePack for SignalR/Redis"

• Intent: compact; fast.

Pattern 4 — "Avro for Kafka with registry"

• Intent: schema evolution.

Pattern 5 — "MemoryPack for .NET-only hot paths"

• Intent: ultimate perf.

Pros & cons / trade-offs

When to use / when to avoid

• Use STJ for public APIs.
• Use Protobuf for gRPC.
• Use MessagePack for SignalR / cache.
• Avoid BinaryFormatter.
• Avoid changing serializer for premature perf.

Interview Q&A

Q1. Why is BinaryFormatter banned? Insecure deserialization; attacker-controlled blob can execute arbitrary types. Removed in .NET 9+.

Q2. JSON vs Protobuf? JSON: universal, human-readable, larger. Protobuf: schema-driven, smaller, faster, cross-lang.

Q3. MessagePack use cases? SignalR, Redis caching, RPC. JSON-like model in binary form.

Q4. MemoryPack? Cysharp .NET serializer; source-gen; fastest in .NET benchmarks; not cross-language.

Q5. Avro vs Protobuf? Both schema-driven, binary. Avro: richer evolution rules; Kafka standard with Schema Registry. Protobuf: gRPC standard.

Q6. STJ source-gen? Compile-time codegen for serializers. AOT-friendly; faster; no reflection.

Q7. Schema evolution best practices? Protobuf: don't reuse field numbers. Avro: defaults for new fields. JSON: version in payload/URL.

Q8. Compression in MessagePack? LZ4 block. Big bandwidth saver.

Q9. Performance tier? MemoryPack > MessagePack ≈ Protobuf > STJ source-gen > STJ reflection > JSON.NET > BinaryFormatter.

Q10. When premature optimization? Switching from JSON for perf without measuring. Most APIs aren't bottlenecked by serializer.

Q11. Cross-language gotchas? Float precision; date formats; default values. Protobuf/Avro handle; ad-hoc binary doesn't.

Q12. JSON column in EF + which serializer? EF uses STJ by default. Configure converter for custom format.

Gotchas / common mistakes

• ⚠️ BinaryFormatter anywhere.
• ⚠️ Protobuf field renumbering.
• ⚠️ JSON without versioning — break clients silently.
• ⚠️ MemoryPack across .NET versions — incompatibility.
• ⚠️ Premature perf switch without measuring.

Further reading

• MS Learn: System.Text.Json source generation
• MessagePack for C#
• MemoryPack
• Protobuf C# guide
• Confluent Avro