With a rich ecosystem and support for multiple languages, it’s no surprise that Protobuf has emerged as a challenger to Avro’s crown as the de-facto serialization format for Kafka. Helped by first class support from Confluent, RedHat and others, Protobuf has finally arrived as viable choice for enterprise wide use cases. During this talk we will tackle how we have used Protobuf successfully with Kafka: from clients to connectors; streams to schema registry; and gitops to governance. We will go over our learnings, including how we have improved the developer experience through the use of linting and early breaking change detection. Expect to leave this talk knowing more about Protobuf and how it is supported across the Kafka ecosystem. We will cover thorny topics such as field presence, reusability, the value of a registry and when schema-less is the right answer. Finally, we will share the pitfalls and challenges, how we’ve made Protobuf work seamlessly for us at scale.

1. Reigning in Protobuf
Graham Stirling David Navalho

2. Saxo adopted a ‘data mesh’ architecture in 2019

5. << Data Governance

8. ▪ Schema Registry is a “serving layer for your metadata”
▪ Storage, versioning and retrieval of schemas
▪ Enables data discovery
▪ Allows for safe schema evolution

12. Before we can talk about Protobuf,
we first need to talk about …

14. common/identifiers.avdl
trading/transaction.avdl instrument/base.avdl
@namespace("domain.common")
protocol Identifiers {
record InstrumentIdentifier {
int id;
}
}
@namespace("domain.trading")
protocol TransactionProtocol {
import idl "common/identifiers.avdl";
record OpenPosition {
domain.common.InstrumentIdentifier id;
}
}
@namespace("domain.instrument")
protocol BaseProtocol {
import idl "common/identifiers.avdl";
record InstrumentBase {
domain.common.InstrumentIdentifier id;
}
}

15. {
"protocol" : "TransactionProtocol",
"namespace" : "domain.trading",
"types" : [ {
"type" : "record",
"name" : "InstrumentIdentifier",
"namespace" : "domain.common",
"fields" : [ { "name" : "id", "type" : "int" } ]
}, {
"type" : "record",
"name" : "OpenPosition",
"fields" : [ { "name" : "id", "type" : "domain.common.InstrumentIdentifier"
} ]
} ],
"messages" : { }
}
public static final org.apache.avro.Schema SCHEMA$ =
new org.apache.avro.Schema.Parser().parse("{"type":"record"," +
""name":"OpenPosition","namespace":"domain" +
".trading","fields":[{"name":"id"," +
""type":{"type":"record"," +
""name":"InstrumentIdentifier","namespace":"domain" +
".common","fields":[{"name":"id"," +
""type":"int"}]}}]}");
> java –jar avro-tools-1.11.0.jar idl transaction.avdl transaction.avsc
> java –jar avro-tools-1.11.0.jar compile protocol transaction.avsc
lookup

16. Caused by:
io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
Schema not found; error code: 40403

17. Not all language bindings are created equal

18. RESOLVED
@matt_howlett
OPEN
OPEN
OPEN
OPEN
OPEN
OPEN OPEN
OPEN

19. Optimistically tweet Matt Howlett?

21. A serialization format with no gaps?

23. Avro Proto3
Schema
Evolution
▪ Required with defaults ▪ Everything is optional
Annotations ▪ Loosely typed e.g.
@info-class(“personal")
▪ Strongly typed e.g.
option (info_class) = PERSONAL
Official Releases
(Since Jan 21)
▪ 2 ▪ > 20
Language
Support (Official)
▪ C#
▪ Java
▪ Python
▪ C++
▪ JS/Perl/PHP/Ruby/Rust
▪ C#
▪ Java/Kotlin
▪ Python
▪ C++
▪ PHP/Ruby/PHP/Go/Dart/Objective-C
Data Lake
Support
▪ Yes ▪ No (mostly)
3rd Party
Ecosystem
▪ Limited (outside of Hadoop) ▪ Rich

24. lookup
syntax = "proto3";
package domain.trading;
import "common/identifiers.proto";
message OpenPosition {
domain.common.InstrumentIdentifier id = 1;
}
syntax = "proto3";
package domain.common;
message InstrumentIdentifier {
int64 id = 1;
}
static {
java.lang.String[] descriptorData = {
"n031trading/transaction.proto022016domain.trad" +
"ing032030common/identifiers.proto"?n014OpenPosition022/n002id” +
“030001 001(0132#.domain.common.InstrumentIdentifierb006proto3"
};
static {
java.lang.String[] descriptorData = {
"n030common/identifiers.proto022rdomain.common""n024Instr” +
“umentIdentifier022nn002id030001 001(003b006proto3"
};

25. import "common/v1/change_type.proto";
import "common/v1/client_identifier.proto";
import "common/v1/currency_identifier.proto";
import "common/v1/date.proto";
import "common/v1/uuid.proto";
import "common/v1/employee_identifier.proto";
import "common/v1/quantity.proto";
import "common/v1/timestamp_ticks.proto";
import "common/v1/user_role_type_identifer.proto";
import "common/v1/market_identifier_code.proto";
import "common/instruments/v1/instrument_identifier.proto";
import "common/instruments/v1/instrument_price.proto";
import "common/instruments/v1/instrument_symbol_code.proto";
import "common/trading/v1/commission_group_identifier.proto";
import "common/trading/v1/currency_cut_profile_identifier.proto";
import "common/trading/v1/order_identifier.proto";
import "common/trading/v1/tool_identifier.proto";
import "common/trading/v1/trade_identifier.proto";
import "common/trading/v1/trading_account_public_identifier.proto";
import "common/trading/v1/external_references.proto";
import "trading/v1/open_position_fx_option.proto";
import "trading/v1/open_position_bond.proto";
import "trading/v1/open_position_cfd.proto";
import "trading/v1/open_position_cfd_on_future.proto";
import "trading/v1/open_position_contract_future.proto";
import "trading/v1/open_position_eto.proto";
import "trading/v1/open_position_external_charge.proto";
import "trading/v1/open_position_tax.proto";
import "trading/v1/open_position_transaction_cost.proto";
import "trading/v1/trade_registration.proto";
import "trading/v1/instrument_to_account_rate.proto";
import "trading/v1/open_position_fx.proto";
import "trading/v1/correlation.proto";
[truncated]

26. 404(03)

27. Not all language bindings are created equal !!!

28. And as adoption increased, versions changed,
and boundaries pushed, we started to see a lot
more issues cropping up in our support channel …

30. * To expedite matters, one might consider
skipping making choices entirely, and proceed
straight to regret.

31. Let’s revisit the original design driver for a
Kafka Schema Registry …

32. AVRO-1124 (“RESTful service for holding schemas”)
Jay Kreps

33. “Motivation: Pass around data in serialized form but still
know the exact schema that was used to serialize it. The
overhead of storing the schema with each record is too high
unless the individual records are very large”

35. Protobuf is different …

37. Let’s look at how the ID is used by the client …

38. C# consumer doesn’t care …
Confluent.SchemaRegistry.Serdes.Protobuf/ProtobufDeserializer.cs

39. The C# producer does care ..
Confluent.SchemaRegistry.Serdes.Protobuf/ProtobufSerializer.cs
… but only because the consumer might

40. But the latest version might be good enough …
Confluent.SchemaRegistry.Serdes.Protobuf/ProtobufSerializer.cs
… more on that later

41. Other benefits mentioned in AVRO-1124 …
▪ A graphical browser
▪ Automatic enforcement of compatibility rules

43. Let’s talk about compatibility rules

44. Avro
Backward ▪ Delete Mandatory Fields
▪ Add Optional fields
Backward Transitive ▪ Delete Mandatory Fields
▪ Add Optional fields
Forward ▪ Add Mandatory Fields
▪ Delete Optional fields
Forward Transitive ▪ Add Mandatory Fields
▪ Delete Optional fields
Full ▪ Add optional fields
▪ Delete optional fields
Full Transitive ▪ Add optional fields
▪ Delete optional fields
None ▪ All changes accepted

45. Avro
Backward ▪ Delete Mandatory Fields
▪ Add Optional fields
Backward Transitive ▪ Delete Mandatory Fields
▪ Add Optional fields
Forward ▪ Add Mandatory Fields
▪ Delete Optional fields
Forward Transitive ▪ Add Mandatory Fields
▪ Delete Optional fields
Full ▪ Add optional fields
▪ Delete optional fields
Full Transitive ▪ Add optional fields
▪ Delete optional fields
None ▪ All changes accepted

46. Avro Proto3 (aka “everything is optional”)
Backward ▪ Delete Mandatory Fields
▪ Add Optional fields
▪ Add Optional fields
▪ Delete Optional fields
Backward Transitive ▪ Delete Mandatory Fields
▪ Add Optional fields
Forward ▪ Add Mandatory Fields
▪ Delete Optional fields
Forward Transitive ▪ Add Mandatory Fields
▪ Delete Optional fields
Full ▪ Add optional fields
▪ Delete optional fields
Full Transitive ▪ Add optional fields
▪ Delete optional fields
None ▪ All changes accepted ▪ All changes accepted

47. Evolving schemas over time

51. How does Schema Registry handle it?

56. 409

57. Intuitively, this looks wrong…

58. Reserved is your friend…
but not enforced!

59. Reserved is your friend…
but not enforced!

60. Multiple event types in the same topic

67. oneof: data loss scenario

68. oneof: data loss scenario

69. oneof: data loss scenario

70. How does Schema Registry handle it?

72. 409

74. 409

76. Id: N

78. Id: N

79. ▪ ONEOF compatibility doesn’t work for us
▪ Field deletion/reserved validation not handled
▪ Need more control over schema evolution

80. Buf CLI: breaking change detector

81. Fine grained control of the exact breaking rules we want

82. Fine grained control of the exact breaking rules we want

83. Fine grained control of the exact breaking rules we want

84. Documentation enforcement
Opinionated approach to major versioning (v1, v2, etc)
Naming standardisation
Enforcement of package cohesion
Bonus: Buf CLI linting!

85. Overall approach

90. What about Schema Registry?

93. Future Direction

94. ▪ Use a Schema Registry?
▪ Don’t use a Schema Registry?
▪ Consider what problem you’re trying to solve

95. Many improvements to Schema Registry e.g.
▪ Compare canonical form (normalize=true)
▪ Better errors (verbose=true)

96. thanks to a patient Saxo development community

98. Tak
Graham Stirling David Navalho