Download as PDF, PPTX
![@bbejeck
Build a Schema
{
"type":"record",
"namespace": "io.confluent.developer.avro",
"name":"Purchase",
"fields": [
{"name": "item", "type":"string"},
{"name": "amount", "type": "double”, ”default”:0.0},
{"name": "customer_id", "type": "string"}
]
}
21](https://image.slidesharecdn.com/bs40billbejeckconfluent-220503181755/75/Schema-Registry-101-with-Bill-Bejeck-Kafka-Summit-London-2022-21-2048.jpg)
![@bbejeck
Build a Schema
Avro
{"name": ”list", "type":{ "type": "array", "items" : "string",
"default": [] }}
{"name": ”numbers", "type": {"type": "map", "values": ”long”,
“default” : {}}}
Protobuf
repeated string strings = 1;
map<string, string> projects = 2;
map<string, Message> myOtherMap = 3;
23](https://image.slidesharecdn.com/bs40billbejeckconfluent-220503181755/75/Schema-Registry-101-with-Bill-Bejeck-Kafka-Summit-London-2022-23-2048.jpg)
Uploaded byHostedbyConfluent
Schema Registry 101 with Bill Bejeck | Kafka Summit London 2022
The document provides an overview of the Schema Registry in Apache Kafka, detailing its importance in managing schema evolution and compatibility for event objects. It outlines the process for building, registering, and using schemas with various formats like Avro, Protocol Buffers, and JSON Schema, as well as configuration options for producers and consumers. Additionally, it discusses schema compatibility checks, the lifecycle of schemas, and includes resources for further learning.
More Related Content
Similar to Schema Registry 101 with Bill Bejeck | Kafka Summit London 2022
Schema Registry 101 with Bill Bejeck | Kafka Summit London 2022
- 1. Schema Registry 101 BillBejeck @bbejeck
- 2. @bbejeck Nice to meetyou! • Member of the DevX team • Prior to DevX ~3 years as engineer on Kafka Streams team • Apache Kafka® Committer and PMC member • Author of “Kafka Streams in Action” - 2nd edition underway! 2
- 3. Why Schema Registry? Aquick Apache Kafka® review
- 4. @bbejeck Kafka – anappend only distributed log 4
- 5. @bbejeck Kafka brokers workwith bytes only 5
- 6. @bbejeck Clients produce andconsume 6
- 7. Event objects forma contract
- 8.
- 9. @bbejeck Expectation of objectstructure 9
- 10. @bbejeck Potentially make unexpectedchanges 10
- 11. @bbejeck Use Schema RegistryFTW! 11
- 12. @bbejeck Use Schema RegistryFTW! 12
- 13. Working with SchemaRegistry
- 14. @bbejeck Working with SchemaRegistry 1. Write/download a schema 2. Test and upload the schema 3. Generate the objects 4. Configure clients (Producer, Consumer, Kafka Streams) 5. Write your application! 14
- 15. @bbejeck Working with SchemaRegistry • Confluent Cloud UI • Schema Registry REST API / Confluent CLI • Producer and Consumer client • Schema Registry console producer and consumer • Tools • Gradle and Maven plugins 15
- 16.
- 17. @bbejeck Build a Schema •Avro • Protocol Buffers • JSON Schema 17
- 18.
- 19. @bbejeck Build a Schema { "type":"record", "namespace":"io.confluent.developer.avro", } 19
- 20. @bbejeck Build a Schema { "type":"record", "namespace":"io.confluent.developer.avro", "name":"Purchase", } 20
- 21. @bbejeck Build a Schema { "type":"record", "namespace":"io.confluent.developer.avro", "name":"Purchase", "fields": [ {"name": "item", "type":"string"}, {"name": "amount", "type": "double”, ”default”:0.0}, {"name": "customer_id", "type": "string"} ] } 21
- 22. @bbejeck Build a Schema(Protobuf) syntax = "proto3"; package io.confluent.developer.proto; option java_outer_classname = "PurchaseProto"; message Purchase { string item = 1; double amount = 2; string customer_id = 3; } 22
- 23. @bbejeck Build a Schema Avro {"name":”list", "type":{ "type": "array", "items" : "string", "default": [] }} {"name": ”numbers", "type": {"type": "map", "values": ”long”, “default” : {}}} Protobuf repeated string strings = 1; map<string, string> projects = 2; map<string, Message> myOtherMap = 3; 23
- 24.
- 25.
- 26.
- 27. @bbejeck Register 27 Producer clients can“auto-register” producerConfigs.put(“auto.register.schemas”, true) Not for production!!!
- 28.
- 29.
- 30.
- 31.
- 32. @bbejeck Clients 32 • basic.auth.credentials.source=USER_INFO • schema.registry.url= https://<CLUSTER>.us-east-2.aws.confluent.cloud •basic.auth.user.info=API_KEY:API_SECRET
- 33. @bbejeck Clients - Producer 33 producerConfigs.put("value.serializer",? ); • KafkaAvroSerializer.class • KafkaProtobufSerializer.class • KafkaJsonSchemaSerializer.class
- 34. @bbejeck Clients – Consumer 34 consumerConfigs.put("value.deserializer",? ); • KafkaAvroDeserializer.class • KafkaProtobufDeserializer.class • KafkaJsonSchemaDeserializer.class
- 35. @bbejeck Clients – Consumer 35 •specific.avro.reader = true|false • specific.protobuf.value.type = proto class name • json.value.type = class name
- 36. @bbejeck Clients – ConsumerReturned Types Avro • SpecificRecord – myObj.getFoo() • GenericRecord - generic.get(“foo”) Protobuf • Specific • Dynamic 36
- 37. @bbejeck Clients – KafkaStreams 37 • SpecificAvroSerde • GenericAvroSerde • KafkaProtobufSerde • KafkaJsonSchemaSerde
- 38. @bbejeck Clients – KafkaStreams 38 Serde<Generated> serde = new SpecificAvroSerde<>(); Map<String, String> configs = Map.of(“schema.registry.url”, “https..”, …) serde.configure(configs)
- 39. @bbejeck Clients – CommandLine Produce 39 confluent kafka topic produce purchases --value-format protobuf --schema src/main/proto/purchase.proto --sr-endpoint https://.... --sr-api-key xxxyyyy --sr-api-secret abc123 --cluster lkc-45687 > {"item":"pizza", "amount":17.99, "customer_id":”lombardi"}
- 40. @bbejeck Clients – CommandLine Produce 40 ./bin/kafka-protobuf-console-producer --topic purchases --bootstrap-server localhost:9092 --property schema.registry.url = http://... --property value.schema ="$(<src/main/proto/purchase.proto)" > {"item":"pizza", "amount":17.99, "customer_id":”lombardi"}
- 41. @bbejeck Clients – CommandLine Consume 41 confluent kafka topic consume purchases --from-beginning --value-format protobuf --schema src/main/proto/purchase.proto --sr-endpoint https://.... --sr-api-key xxxyyyy --sr-api-secret abc123 --cluster lkc-45687
- 42. @bbejeck Clients – CommandLine Consume 42 ./bin/kafka-protobuf-console-consumer --from-beginning --topic purchases --bootstrap-server localhost:9092 --property schema.registry.url = http://...
- 43. @bbejeck Clients –Testing 43 For testingyou can use a mock schema registry • schema.registry.url= “mock://scope-name”
- 44. What is aSubject?
- 45. @bbejeck What is aSubject ? • Defines a namespace for a schema • Compatibility checks are per subject • Different approaches – subject name strategies 45
- 46.
- 47.
- 48.
- 49.
- 50. @bbejeck Schema Compatibility • SchemaRegistry provides a mechanism for safe changes • Evolving a schema • Compatibility checks are per subject • When a schema evolves, the subject remains, but the schema gets a new ID and version 50
- 51. @bbejeck Schema Compatibility • Backward- default • Forward • Full Latest version will work with previous version • But not necessarily with versions beyond that 51
- 52. @bbejeck Schema Compatibility • Backwardtransitive • Forward transitive • Full transitive Latest version will work with all pervious versions 52
- 53. @bbejeck Schema Compatibility Backward ForwardFull Delete fields Add fields with default values Delete fields with default values Add fields Delete or add fields either must have default values. Update consumer clients first Update producer clients first Order of update doesn’t matter 53
- 54. @bbejeck Testing a Schemafor compatiblity 54
- 55. @bbejeck Summary • Kafka brokerworks with bytes only • Event objects form an implicit contract • Using Schema Registry allows for an explicit contract • Schema Registry provides for keeping domain objects in sync 55
- 56. @bbejeck Resources • Documentation -https://docs.confluent.io/platform/current/schema- registry/index.html#sr-overview • Multiple Event Types Presentation - https://www.confluent.io/en- gb/events/kafka-summit-europe-2021/managing-multiple-event-types-in- a-single-topic-with-schema-registry/ • Multiple Events tutorial - https://developer.confluent.io/tutorials/multiple- event-type-topics/confluent.html • Multiple Event Type code - https://github.com/bbejeck/multiple-events- kafka-summit-europe-2021 56
- 57. @bbejeck Resources • Martin Kleppmann- https://www.confluent.io/blog/put-several-event- types-kafka-topic/ • Robert Yokota - https://www.confluent.io/blog/multiple-event-types-in- the-same-kafka-topic/ • Avro - https://avro.apache.org/docs/current/ • Protocol Buffers - https://developers.google.com/protocol-buffers • JSON Schema - https://json-schema.org/ 57
- 58. @bbejeck Build tools 58 • SchemaRegistry Maven • https://docs.confluent.io/platform/current/schema- registry/develop/maven-plugin.html#sr-maven-plugin • Schema Registry Gradle • https://github.com/ImFlog/schema-registry-plugin • Protobuf Gradle • https://github.com/google/protobuf-gradle-plugin • Avro Gradle • https://github.com/davidmc24/gradle-avro-plugin
- 59. @bbejeck Build tools 59 • JSONSchema • https://github.com/jsonschema2dataclass/js2d-gradle • https://github.com/joelittlejohn/jsonschema2pojo
- 60.
- 61. Your Apache Kafka® journeybegins here developer.confluent.io 61