Will talk about Avro, schema evolution, and its compatibility with hands on approach – mostly in Java

1. Avro

4. Topic <=> DB Table
Schema <=> DB Column metadata
Model

5. • Data safety
• Data compressed automatically
• Documentation
• Cross language portability
• Evolution
• Many tools and moving parts
• Learning curve
• Difficult to debug
• Complexity (comes with Kafka)
+
-
Avro Schema

6. Demo

9. Compatibility Type Changes allowed Check against which schemas Upgrade first
BACKWARD
•Delete fields
•Add optional fields
Last version Consumers
BACKWARD_TRANSITIVE
•Delete fields
•Add optional fields
All previous versions Consumers
FORWARD
•Add fields
•Delete optional fields
Last version Producers
FORWARD_TRANSITIVE
•Add fields
•Delete optional fields
All previous versions Producers
FULL
•Add optional fields
•Delete optional fields
Last version Any order
FULL_TRANSITIVE
•Add optional fields
•Delete optional fields
All previous versions Any order
NONE •All changes are accepted Compatibility checking disabled Depends

10. C1 P1
Schema Evolutions
V1

11. C1 P1
Schema-Forward
V1 V2

12. C1 P1
Schema-Forward(Broken)
C2 C2
V1 V2

13. C1 P1
Schema-Forward(Fixed by Backward)
C2 C2
V1 V2

14. C1 P1
Schema-Forward(Delete)
V1

15. C1 P1
Schema-Forward(Delete)
V1 V2

16. C1 P1
Schema-Forward(Delete)
V1 V2
C2

17. C1 P1
Schema
V1
C2 P2

18. V2
C1 P1
Schema-Backward
V1
C2 P2

19. V2
C1 P1
Schema-Backward(fixed)
V1
C2 P2

20. V2
Schema-Transitive(Forward)
V1
V3

21. • Java-Avro codegen
• Reflect
• And why reflect?