Evolving Schemas Without Schema Evolution With Andreas Evers | Current 2022 Schema Registry allows us to control how schemas can be evolved through time without breaking compatibility. However, when using high retention, you could end up with a convoluted “dirty” schema depending on the selected compatibility level, as well as many different models in your application logic to support all the schema versions. Upcaster chains allow you to read an old version of a message and bring it to what your logic needs today. The upcasters in the chain describe how to jump from one version to the next. They describe what your logic expects instead of covering all the possible variations that were ever published. We created an upcaster registry to allow you to declare upcasters and their chains in a declarative manner allowing them to be used across all your streaming apps. It is a great combo with Schema Registry; one providing the schemas themselves, the other expressing how to evolve from one schema version to the next.

1. Evolving Schemas
Without Schema Evolution

2. Abstract
Schema Registry allows us to control how schemas can be evolved through time without breaking compatibility.
However, when using high retention, you could end up with a convoluted “dirty” schema depending on the selected
compatibility level, as well as many different models in your application logic to support all the schema versions.
Upcaster chains allow you to read an old version of a message and bring it to what your logic needs today.
The upcasters in the chain describe how to jump from one version to the next. They describe what your logic expects instead of
covering all the possible variations that were ever published.
We created an upcaster registry to allow you to declare upcasters and their chains in a declarative manner allowing them to be
used across all your streaming apps. It is a great combo with Schema Registry; one providing the schemas themselves, the other
expressing how to evolve from one schema version to the next.

3. Andreas Evers
CTO @ KOR Financial
Previously Pivotal / VMware
➢ Staff solution architect consulting for some of the biggest European firms
➢ Focused on application transformation & system design
➢ Worked on Spinnaker, together with the Spring team
Who is this guy?
@andreasevers

5. Mission
Avoid another financial crisis
by mapping out risk in financial
markets

6. Selena Gomez
explains “CDOs”, one of the
financial products that were
the cause of the 2008 crash

8. KOR Solution
Real-time insights based on
regulation to make financial
products like CDOs
transparent to the market and
regulators.

11. Most companies use Kafka with a
low retention setting, such as 7 days
This works great for messaging use cases

12. Kafka can mean so much more
for your company

13. The typical use case offloads the streamed events into an external database
Those databases typically use commit logs or sequential files internally
Such commit logs and sequential files are in fact very similar
to Kafka topics and segment files 🤔
Why not use Kafka as the commit log for your data, and
become the system of record instead of merely a tunnel

14. KSQL: Turning the database inside-out
Source of truth
stream
Continuous
query
Derived table

15. High retention offers a lot of benefits,
but also comes with its own unique challenges
You can’t just wait 7 days
for old formats to leave your cluster

16. How do we deal with
Evolution?

17. Let’s start with
short retention

19. Class People {
- Rick
- Morty
}

20. Class People {
- Rick
- Morty
}

21. - Rick
- Morty
Class People {
- Rick
- Morty
}
0

22. - Rick
- Morty
Class People {
- Rick
- Morty
}
0
Class People {
- Rick
- Morty
}

23. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
- Summer
}

24. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
- Summer
}
- Rick
- Morty
- Summer
1

25. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
- Summer
}
- Rick
- Morty
- Summer
1

26. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
- Summer
}
- Rick
- Morty
- Summer
1

27. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
}
1

28. Class People {
- Rick
- Morty
}
- Rick
- Morty
0 1
Class People {
- Rick
- Morty
}

29. Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
0 1
Class People {
- Rick
- Morty
}

30. Class People {
- Rick
- Morty
- Summer
}
Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
0 1

31. Class People {
- Rick
- Morty
- Summer
}
Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
0 1
- Rick
- Morty
- Summer

32. Class People {
- Rick
- Morty
- Summer
}
Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
- Rick
- Morty
- Summer
0 1

33. Class People {
- Rick
- Morty
- Summer
}
Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
- Rick
- Morty
- Summer
0 1

34. Class People {
- Rick
- Morty
- Summer
}
Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
- Rick
- Morty
- Summer
0 1

35. 1
Class People {
- Rick
- Morty
- Summer
}
Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
- Summer

36. 1
Class People {
- Rick
- Morty
- Summer
}
Class People {
- Rick
- Morty
- Summer
}
- Rick
- Morty
- Summer

37. Isn’t this easier
in HTTP APIs?

38. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
}

39. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
}

40. - Rick
- Morty
- Summer
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
- Summer
}

41. Tolerant Reader
Postel’s Law,
a.k.a. the Robustness Principle:
Be conservative in what you do,
be liberal in what you accept from others.

42. Client Functions
Client Stub
Server Functions
Server Stub
(skeleton)
Network Routines Network Routines
kernel
client
process
kernel
server
process
network
client server

43. Client Objects
Client SOAP Handler
Server Objects
Server SOAP Handler
Network Routines Network Routines
network
kernel
client
process
kernel
server
process
client server

44. SpecificRecord
Avro Serdes
SpecificRecord
Avro Serdes
Network Routines Network Routines
network
kernel
client
process
kernel
server
process
client server

45. mvn generate-sources
Look ma, no programming!

46. - Rick
- Morty
- Summer
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
- Summer
}
Be conservative in what you do,
be liberal in what you accept from others.
@JsonIgnoreProperties
(ignoreUnknown = true)

47. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
- Summer
}
- Rick
- Morty
- Summer
1
Be conservative in what you do,
be liberal in what you accept from others.
Tolerant
GenericRecordToDto
Deserializer

48. What about
some other changes?

49. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
}
- Rick
1

50. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
}
- Rick
1

51. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- PickleRick
- Morty
}
- PickleRick
- Morty
1

52. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- PickleRick
- Morty
}
- PickleRick
- Morty
1

53. - Rick
- Morty
0
Class People {
- PickleRick
- Morty
}
Class People {
- PickleRick
- Morty
}
- PickleRick
- Morty
1

54. - Rick
- Morty
0
Class People {
- PickleRick
- Morty
}
Class People {
- PickleRick
- Morty
}
- PickleRick
- Morty
1

55. - Rick: String
- Morty
0
Class People {
- Rick: String
- Morty
}
Class People {
- Rick: boolean
- Morty
}
- Rick: boolean
- Morty
1

56. - Rick: String
- Morty
0
Class People {
- Rick: String
- Morty
}
Class People {
- Rick: boolean
- Morty
}
- Rick: boolean
- Morty
1

57. How can we prevent
breaking consumers?

58. - Rick
- Morty
0
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
}
1

59. Class People {
- Rick
- Morty
}
- Rick
- Morty
0 1
Class People {
- Rick
- Morty
}

60. Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
0 1
Class People {
- Rick
- Morty
}

61. Class People {
- Rick
- Morty
- Summer
}
Class People {
- Rick
- Morty
- Optional<Summer>
}
- Rick
- Morty
0 1

62. Consumer-driven
Contract Testing
Field-by-field compatibility testing

63. Consumer-driven
Contract Testing
Field-by-field compatibility testing

64. V2

65. V2
Consumer A is on version 2,
only uses PickleRick

66. V2
Consumer A is on version 2,
only uses PickleRick

67. V2
Consumer A is on version 2,
only uses PickleRick

68. V2
Consumer A is on version 2,
only uses PickleRick

69. V2
Consumer-Driven Contract Testing
Consumer A is on version 2,
only uses PickleRick

70. Consumer-Driven Contract Testing
❯ pact-broker can-i-deploy
--pacticipant Foo
--version 23
--to production

71. Schema Registry
Automatic schema management
and validation

72. Schema Evolution through Schema Registry
Schema Registry
• Versions every schema
• Every schema has a unique ID
• Uses subjects with Topic, Record, or TopicRecord names
• For each subject, each schema change gets a version increase
Schema
ID: foo
Schema
ID: bar
Subject
topic-People
- Rick
- Morty
- Rick
- Morty
- Summer
V1
V2

73. Compatibility Types
Compatibility Type Changes allowed Check against which schemas Upgrade first
BACKWARD Delete fields
Add optional fields
Last version Consumers
BACKWARD_TRANSITIVE Delete fields
Add optional fields
All previous versions Consumers
FORWARD Add fields
Delete optional fields
Last version Producers
FORWARD_TRANSITIVE Add fields
Delete optional fields
All previous versions Producers
FULL Add optional fields
Delete optional fields
Last version Any order
FULL_TRANSITIVE
Add optional fields
Delete optional fields
All previous versions Any order
NONE All changes are accepted Compatibility checking disabled Depends

74. Backward

75. Class People {
- Rick
- Optional<Summer>
}
Backward
0 1
- Rick
- Morty
- Rick
- Optional<Summer>

76. Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
Backward
- Rick
- Morty
0
- Rick
- Optional<Summer>
1

77. Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
Backward
0 1
- Rick
- Morty
- Rick
- Optional<Summer>

78. Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
Backward
0 1
- Rick
- Morty
- Rick
- Optional<Summer>

79. Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
- Rick
- Optional<Summer>
Backward

80. Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
- Rick
- Optional<Summer>
Backward

81. Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Morty
}
- Rick
- Morty
Backward

82. Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Morty
}
- Rick
- Morty
Backward

83. - Rick
- Morty
0
Class People {
- Rick
- Optional<Summer>
}
- Rick
- Optional<Summer>
1
- Rick
2
Backward
Transitive

84. - Rick
- Morty
0
Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
- Rick
- Optional<Summer>
1
- Rick
2
Backward
Transitive

85. - Rick
- Morty
0
Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
- Rick
- Optional<Summer>
1
- Rick
2
Backward
Transitive

86. - Rick
- Morty
0
Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
- Rick
- Optional<Summer>
1
- Rick
2
Backward
Transitive

87. - Rick
- Morty
0
Class People {
- Rick
- Optional<Summer>
}
Class People {
- Rick
- Optional<Summer>
}
- Rick
- Optional<Summer>
1
- Rick
2
Backward
Transitive

88. Forward

89. Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
- Optional<Morty>
}
Forward
- Rick
- Optional<Morty>

90. Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
- Optional<Morty>
}
Forward
- Rick
- Optional<Morty>

91. Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
- Summer
}
Forward
- Rick
- Optional<Morty>
- Rick
- Summer

92. Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
- Optional<Morty>
}
- Rick
- Optional<Morty>
Forward

93. Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
- Summer
}
- Rick
- Summer
Forward

94. - Rick
- Optional<Morty>
0
Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
- Optional<Morty>
}
1 2
Forward
Transitive

95. - Rick
- Optional<Morty>
0
Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
- Optional<Morty>
}
1 2
Forward
Transitive

96. - Rick
- Optional<Morty>
0
Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
}
1
- Rick
2
Forward
Transitive

97. - Rick
- Optional<Morty>
0
Class People {
- Rick
- Optional<Morty>
}
Class People {
- Rick
- Summer
}
- Rick
- Summer
1
- Rick
2
Forward
Transitive

98. Short Retention = Full?
Backward when
• you want to allow the producer to roll back to a previous version,
writing a previous schema, without breaking consumers (provided they
are tolerant readers)
Forward when
• you want to allow consumers to read new versions without needing to
be updated first (provided they are tolerant readers and non-compatible
upgrades are controlled, e.g. using CDCT)

99. High Retention = Full Transitive?
Backward transitive when
• you want to allow new consumers which will start at offset 0,
to be added at any time in the future
• or you want to allow producers to roll back to any previous version,
writing a previous schema, without breaking consumers (provided they
are tolerant readers)
Forward transitive when
• you want to allow consumers to read new versions without needing to
be updated first (provided they are tolerant readers)
regardless from where they start consuming

100. High Retention = Full Transitive forever?
Class People {
- Optional<Rick>
- Optional<Morty>
- Optional<Summer>
- Optional<PickleRick>
- Optional<Rick:boolean>
}

101. Evolution strategies
• Tolerant Reader
• Versioned Events
• Copy-Transform
• Upcasting

102. Copy-Transform
A.k.a. Copy & Replace
A.k.a. The Parallel Universe

103. Copy-Transform

104. Copy-Transform

105. Copy-Transform

106. Copy-Transform

107. Copy-Transform

108. Copy-Transform

109. Copy-Transform

110. Copy-Transform
Benefits Drawbacks
Relatively simple to
implement
Complete freedom to do
anything you want
Data preservation
One-time operation
Performance

111. Upcasting
A.k.a. Message Translators

113. Avro defaults give you wire-compatibility
Upcasting gives you business-compatibility

114. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick

115. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick

116. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick

117. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V3

118. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V3
V1 → V2
V2 → V3
V2 → V3

119. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V3

120. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V3
V2 → V3

121. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V3
V2 → V3

122. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V3
V1 → V2
V2 → V3
V2 → V3

123. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V3
V1 → V2
V2 → V3
V2 → V3

124. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2

125. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2
V1 → V2 V3 → V2

126. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2
V1 → V2 V3 → V2

127. What if some consumers
are missing upcasters?

128. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2
V1 → V2 V3 → V2

129. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2
V3 → V2

130. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2
V3 → V2

131. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2

132. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2

133. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2

134. - Rick
- Morty
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2
V3 V2
V1
V3
V1
V2
V1

135. Upcaster Registry
Centralized repository for up- and
downcasters

136. Class People {
- Rick
- Morty
}
V1

137. Class People {
- Rick
- Morty
}
V1
- Rick
- Morty
V1

138. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
}
V1
V1
V1

139. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
}
V1
V1
V1

140. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
- Morty
}
V1
V1
V1

141. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
}
V1
V1
V2

142. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
}
V1
V1
V2
V1 → V2
V2 → V1

143. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
}
V1
V1
V2
V1 → V2
V2 → V1
- Rick
V2

144. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
}
V1
V1
V2
V1 → V2
V2 → V1
- Rick
V2

145. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
}
V1
V1
V2
V1 → V2
V2 → V1
- Rick
V2

146. - Rick
- Morty
Class People {
- Rick
- Morty
}
Class People {
- Rick
}
V1
V1
V2
V1 → V2
V2 → V1
- Rick
V2
V2 → V1

147. - Rick
- Morty
Class People {
- Rick
}
Class People {
- Rick
}
V1
V2
V2
V1 → V2
V2 → V1
- Rick
V2

148. - Rick
- Morty
Class People {
- Rick
}
Class People {
- Rick
}
V1
V2
V2
V1 → V2
V2 → V1
- Rick
V2

149. - Rick
- Morty
Class People {
- Rick
}
Class People {
- Rick
}
V1
V2
V2
V1 → V2
V2 → V1
- Rick
V2
V1 → V2

150. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
V1
V2
V3
V1 → V2
V2 → V1
- Rick
V2

151. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
V1
V2
V3
V1 → V2
V2 → V1
- Rick
V2
V3 → V2
V2 → V3

152. - Rick
- Morty
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
- Rick
V1 V2 V3
V2
V1 → V2 V3 → V2
V1 → V2
V3 → V2
V2 → V1
V2 → V3
V3

153. https://github.com/KOR-Financial/kopper

154. - Rick
- Morty
0
Class People {
- Rick
}
Class People {
- PickleRick
- Summer
}
- PickleRick
- Summer
1
- Rick
2
V1 V2 V3
V2
V1 → V2 V3 → V2
V1 → V2
V3 → V2
V2 → V1
V2 → V3
V3
Consumer A is on V2
Supports V1 and V3

155. ❯ pact-broker can-i-deploy
--pacticipant Foo
--version 23
--to production

156. Can we combine these
evolution strategies?

157. Combining strategies for short retention
• Versioned Events
• Tolerant Reader
• Consumer-driven Contract Testing

158. Combining strategies for short retention
This allows you to:
• Avoid transitive
• Avoid the optional dance
• Chase consumers to upgrade
• Prevent producers from mistakenly releasing incompatible version
after releasing compatible version
• Keep your consumers tolerant to change
• Keep your consumers ignorant of information they don’t need

159. Combining strategies for high retention
• Versioned Events
• Upcasting
• Upcaster registry
• Tolerant Reader
• Copy-Transform

160. Combining strategies for high retention
This allows you to:
• Avoid Full Transitive Forever
• Avoid every field being optional
• Avoid having to write to countless deprecated fields
• Keep your consumers tolerant to change
• Keep your consumers ignorant of information they don’t need

161. Thank you!
Accreditations:
Graphics by Arthur Shlain, Creative Stal, Siddharth Dasari, Webtechops LLP, Icons Producer, SBTS from Noun Project
Andreas Evers
@andreasevers
aevers@korfinancial.com
KOR Financial
www.korfinancial.com