![19
Connectors
"config": {
[...]
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:postgresql://postgres:5432/",
"topics": "asgard.demo.orders",
}](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-19-2048.jpg)
![21
Converters
Source Kafka®
Connect Kafka®
Connector
Native Data Connect Record
Converter
Bytes[ ]](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-21-2048.jpg)
![32
Single Message Transforms
"config": {
[...]
"transforms": "addDateToTopic,labelFooBar",
"transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.addDateToTopic.topic.format": "${topic}-${timestamp}",
"transforms.addDateToTopic.timestamp.format": "YYYYMM",
"transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.labelFooBar.renames": "delivery_address:shipping_address",
}](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-32-2048.jpg)
![33
Single Message Transforms
"config": {
[...]
"transforms": "addDateToTopic,labelFooBar",
"transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.addDateToTopic.topic.format": "${topic}-${timestamp}",
"transforms.addDateToTopic.timestamp.format": "YYYYMM",
"transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.labelFooBar.renames": "delivery_address:shipping_address",
}
Transforms config](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-33-2048.jpg)
![34
Single Message Transforms
"config": {
[...]
"transforms": "addDateToTopic,labelFooBar",
"transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.addDateToTopic.topic.format": "${topic}-${timestamp}",
"transforms.addDateToTopic.timestamp.format": "YYYYMM",
"transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.labelFooBar.renames": "delivery_address:shipping_address",
}
Apply these transforms
Transforms config](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-34-2048.jpg)
![35
Single Message Transforms
"config": {
[...]
"transforms": "addDateToTopic,labelFooBar",
"transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter",
"transforms.addDateToTopic.topic.format": "${topic}-${timestamp}",
"transforms.addDateToTopic.timestamp.format": "YYYYMM",
"transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.labelFooBar.renames": "delivery_address:shipping_address",
}
Apply these transforms
Transforms config Config per Transform](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-35-2048.jpg)
![51
Troubleshooting Kafka®
Connect
Task
Connector
RUNNING
FAILED
$ curl -s "http://localhost:8083/connectors/source-debezium-orders/status" |
jq '.connector.state'
"RUNNING"
$ curl -s "http://localhost:8083/connectors/source-debezium-orders/status" |
jq '.tasks[0].state'
"FAILED"](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-51-2048.jpg)
![52
Troubleshooting Kafka®
Connect
$curl -s "http://localhost:8083/connectors/source-debezium-orders-00/status" | jq '.tasks[0].trace'
"org.apache.kafka.connect.errors.ConnectExceptionntat
io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)ntat
io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:197)ntat
io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogReader.java:
1018)ntat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:950)ntat
com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)ntat
com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)ntat
java.lang.Thread.run(Thread.java:748)nCaused by: java.io.EOFExceptionntat
com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:190)ntat
com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:46)ntat
com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserial
izer.java:35)ntat
com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserial
izer.java:27)ntat
com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:212)nt
at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:224)ntat
com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:922)nt... 3 moren"](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-52-2048.jpg)
![61
Error Handling & Dead Letter Queues (DLQ)
https://cnfl.io/connect-dlq
Handled
● Convert (read/write from
Kafka®, [de]-serialisation)
● Transform
Not Handled
● Start (connections to a Data
Store)
● Poll / Put (read/write from/to
Data Store)*
* Can be retried by Kafka® Connect](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-61-2048.jpg)
![69
Schema where are You...
$ http localhost:8081/subjects[...]
jq '.schema|fromjson'
{
"type": "record",
"name": "Value",
"namespace": "asgard.demo.ORDERS",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "order_id",
"type": [
"null",
"int"
],
"default": null
},
{
"name": "customer_id",
"type": [
"null",
"int"
],
"default": null
},
$ kafkacat -b localhost:9092 -C -t mysql-debezium-asgard.demo.ORDERS
QF@Land RoverDefender 90SheffieldSwift LLC,54258 Michigan
Parkway(2019-05-09T13:42:28Z(2019-05-09T13:42:28Z
Schema
(from Schema Registry)
Messages
(AVRO)](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-69-2048.jpg)
![72
Schema where are You...
Schema
(Embedded per
JSON message)
{
"schema": {
"type": "struct",
"fields": [
{
"type": "int32",
"optional": false,
"field": "id"
},
[...]
],
"optional": true,
"name": "asgard.demo.ORDERS.Value"
},
"payload": {
"id": 611,
"order_id": 111,
"customer_id": 851,
"order_total_usd": 182190.93,
"make": "Kia",
"model": "Sorento",
"delivery_city": "Swindon",
"delivery_company": "Lehner,
Kuvalis and Schaefer",
"delivery_address": "31 Cardinal
Junction",
"CREATE_TS":
"2019-05-09T13:54:59Z",
"UPDATE_TS": "2019-05-09T13:54:59Z"
}
Messages
(JSON)
JsonDeserializer with schemas.enable requires "schema" and "payload" fields and may not contain additional fields](https://image.slidesharecdn.com/apachekafkameetupzurichatswissrefromzerotoherowithkafkaconnect20190826v01-190827124450/75/Apache-kafka-meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01-72-2048.jpg)
Uploaded byconfluent
Apache kafka meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01
This document provides an overview of Kafka Connect and how it can be used to stream data between Kafka and other data systems. It discusses key Kafka Connect concepts like connectors, converters, transforms, deployment modes, and troubleshooting. The document contains configuration examples for connectors and transforms.
More Related Content
Similar to Apache kafka meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Apache kafka meet_up_zurich_at_swissre_from_zero_to_hero_with_kafka_connect_20190826v01
- 1. 1 From Zero toHero with Kafka® Connect Perry Krol perry@confluent.io @perkrol
- 2.
- 3. 3 Streaming Integration withKafka® Connect Kafka® Connect Kafka® Brokers Sources
- 4. 4 Streaming Integration withKafka® Connect Kafka® Connect Kafka® Brokers Sinks
- 5. 5 Streaming Integration withKafka® Connect Kafka® Connect Kafka® Brokers Sources Sinks
- 6.
- 7. 7 Streaming Integration withKafka® Connect Kafka® Connect Kafka ® Connect AWS S3 HDFS RDBMS
- 8. 8 Streaming Integration withKafka® Connect Kafka® Connect Kafka ® Connect Kafka ® Connect AWS S3 HDFS RDBMS
- 9. 9 Writing to DataStores from Kafka® Kafka ® Connect Apps DATA STORE
- 10. 10 Evolve Processing fromOld Systems to New Kafka® Connect RDBMS Existing App
- 11. 11 Evolve Processing fromOld Systems to New Kafka® Connect RDBMS New App <x> Existing App
- 12.
- 13.
- 14. 14 Kafka® Connect REST API- Tricks http://go.rmoff.net/connector-status
- 15. 15 Kafka® Connect REST API- Tricks http://go.rmoff.net/selectively-delete-connectors (h/t to @madewithtea)
- 16. 1616 Configuring Kafka® Connect Inside theAPI - connectors - transforms - converters
- 17.
- 18.
- 19. 19 Connectors "config": { [...] "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "connection.url":"jdbc:postgresql://postgres:5432/", "topics": "asgard.demo.orders", }
- 20. 20 Connectors Source Kafka® Connect Kafka® Connector NativeData Connect Record
- 21. 21 Converters Source Kafka® Connect Kafka® Connector NativeData Connect Record Converter Bytes[ ]
- 22. 22 Serialisation & Schemas Confluent SchemaRegistry AVRO JSONProtobuf CSV https://qconnewyork.com/system/files/presentation-slides/qcon_17_-_schemas_and_apis.pdf
- 23. 23 The Confluent SchemaRegistry Source Kafka® Connect Kafka® ConnectKafka® Target Schema Registry AVRO message AVRO message AVRO schema AVRO schema
- 24. 24 The Confluent SchemaRegistry Source Kafka® Connect Kafka® ConnectKafka® Target Schema Registry AVRO message AVRO message AVRO schema AVRO schema
- 25. 25 The Confluent SchemaRegistry Source Kafka® Connect Kafka® ConnectKafka® Target Schema Registry AVRO message AVRO message AVRO schema AVRO schema
- 26. 26 Converters key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://localhost:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://localhost:8081 Set as aglobal default per-worker; optionally can be overridden per-connector
- 27. 27 Converters key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://localhost:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://localhost:8081 Set as aglobal default per-worker; optionally can be overridden per-connector
- 28. 28 Converters key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://localhost:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://localhost:8081 Set as aglobal default per-worker; optionally can be overridden per-connector
- 29. 29 What About InternalConverters value.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter key.internal.value.converter=org.apache.kafka.connect.json.JsonConverter value.internal.value.converter=org.apache.kafka.connect.json.JsonConverter key.internal.value.converter.bork.bork.bork=org.apache.kafka.connect.json.JsonConverter key.internal.value.please.just.work.converter=org.apache.kafka.connect.json.JsonConverter
- 30. 30 What About InternalConverters value.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter key.internal.value.converter=org.apache.kafka.connect.json.JsonConverter value.internal.value.converter=org.apache.kafka.connect.json.JsonConverter key.internal.value.converter.bork.bork.bork=org.apache.kafka.connect.json.JsonConverter key.internal.value.please.just.work.converter=org.apache.kafka.connect.json.JsonConverter Internal converters are for internal use by Kafka Connect only, and have been deprecated as of Apache Kafka 2.0. You should not change these, and you will get warnings from Apache Kafka 2.0 onwards, if you do try to configure them.
- 31. 31 Single Message Transforms SourceKafka® Connect Kafka® Connector ConverterTransform(s)
- 32. 32 Single Message Transforms "config":{ [...] "transforms": "addDateToTopic,labelFooBar", "transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter", "transforms.addDateToTopic.topic.format": "${topic}-${timestamp}", "transforms.addDateToTopic.timestamp.format": "YYYYMM", "transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.labelFooBar.renames": "delivery_address:shipping_address", }
- 33. 33 Single Message Transforms "config":{ [...] "transforms": "addDateToTopic,labelFooBar", "transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter", "transforms.addDateToTopic.topic.format": "${topic}-${timestamp}", "transforms.addDateToTopic.timestamp.format": "YYYYMM", "transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.labelFooBar.renames": "delivery_address:shipping_address", } Transforms config
- 34. 34 Single Message Transforms "config":{ [...] "transforms": "addDateToTopic,labelFooBar", "transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter", "transforms.addDateToTopic.topic.format": "${topic}-${timestamp}", "transforms.addDateToTopic.timestamp.format": "YYYYMM", "transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.labelFooBar.renames": "delivery_address:shipping_address", } Apply these transforms Transforms config
- 35. 35 Single Message Transforms "config":{ [...] "transforms": "addDateToTopic,labelFooBar", "transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter", "transforms.addDateToTopic.topic.format": "${topic}-${timestamp}", "transforms.addDateToTopic.timestamp.format": "YYYYMM", "transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.labelFooBar.renames": "delivery_address:shipping_address", } Apply these transforms Transforms config Config per Transform
- 36. 36 Extensible Source Kafka® Connect Kafka® ConnectorConverterTransform(s)
- 37. 37 Confluent Hub Online libraryof pre-packaged and ready-to-install extensions or add-ons for Confluent Platform and Apache Kafka®: ● Connectors ● Transforms ● Converters Easily install the components that suit your needs into your local environment with the Confluent Hub client command line tool . https://hub.confuent.io
- 38. 3838 Deploying Kafka® Connect Connectors -Tasks - Workers
- 39.
- 40. 40 Connectors & Tasks S3SinkJDBC Source S3 Task #1 JDBC Task #1
- 41. 41 Connectors & Tasks S3SinkJDBC Source S3 Task #1 JDBC Task #1 JDBC Task #2
- 42. 42 Tasks & Workers S3SinkJDBC Source S3 Task #1 JDBC Task #1 JDBC Task #2 Worker
- 43. 43 Kafka® Connect Standalone Worker S3Task #1 JDBC Task #1 JDBC Task #2 Offsets Worker
- 44. 44 “Scaling” the StandaloneWorker JDBC Task #1 Offsets Offsets S3 Task #1JDBC Task #2 WorkerWorker Fault-Tolerant? Nope!
- 45. 45 Kafka® Connect Distributed Worker JDBCTask #2 ● Offsets ● Config ● Status S3 Task #1 JDBC Task #1 Worker Kafka® Connect Cluster
- 46. 46 Scaling the DistributedWorker ● Offsets ● Config ● Status S3 Task #1 JDBC Task #1 WorkerWorker Fault-Tolerant? Yeah! JDBC Task #2 Kafka® Connect Cluster
- 47. 47 Distributed Worker -Fault Tolerance ● Offsets ● Config ● Status S3 Task #1 JDBC Task #1 WorkerWorker Fault-Tolerant? Yeah! JDBC Task #2 Kafka® Connect Cluster
- 48. 48 Distributed Worker -Fault Tolerance JDBC Task #2 ● Offsets ● Config ● Status S3 Task #1 JDBC Task #1 Worker Kafka® Connect Cluster
- 49. 49 Multiple Distributed Clusters ●Offsets ● Config ● Status S3 Task #1 JDBC Task #1 Kafka® Connect Cluster #1 ● Offsets ● Config ● Status Kafka® Connect Cluster #2 JDBC Task #2
- 50.
- 51. 51 Troubleshooting Kafka® Connect Task Connector RUNNING FAILED $ curl-s "http://localhost:8083/connectors/source-debezium-orders/status" | jq '.connector.state' "RUNNING" $ curl -s "http://localhost:8083/connectors/source-debezium-orders/status" | jq '.tasks[0].state' "FAILED"
- 52. 52 Troubleshooting Kafka® Connect $curl -s"http://localhost:8083/connectors/source-debezium-orders-00/status" | jq '.tasks[0].trace' "org.apache.kafka.connect.errors.ConnectExceptionntat io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)ntat io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:197)ntat io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogReader.java: 1018)ntat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:950)ntat com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)ntat com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)ntat java.lang.Thread.run(Thread.java:748)nCaused by: java.io.EOFExceptionntat com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:190)ntat com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:46)ntat com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserial izer.java:35)ntat com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserial izer.java:27)ntat com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:212)nt at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:224)ntat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:922)nt... 3 moren"
- 53. 53 The Log isthe Source of Truth $ confluent log connect $ docker-compose logs kafka-connect $ cat /var/log/kafka/connect.log
- 54.
- 55. 55 Troubleshooting Kafka® Connect “Task isbeing killed and will not recover until manually restarted” Symptom not Cause
- 56.
- 57. 5757 Common Errors withKafka® Connect
- 58.
- 59. 59 Mismatched Converters org.apache.kafka.common.errors.SerializationException: Unknown magicbyte! JSON JSON JSON JSON JSON Messages are not AVRO “Value.converter” : “AvroConverter” Use correct Converter for the Source Data
- 60. 60 Mixed Serialisation Methods org.apache.kafka.common.errors.SerializationException: Unknownmagic byte! JSON JSON AVRO AVRO JSON Some messages are not AVRO “Value.converter” : “AvroConverter” Use Error Handling to Deal with Bad Messages
- 61. 61 Error Handling &Dead Letter Queues (DLQ) https://cnfl.io/connect-dlq Handled ● Convert (read/write from Kafka®, [de]-serialisation) ● Transform Not Handled ● Start (connections to a Data Store) ● Poll / Put (read/write from/to Data Store)* * Can be retried by Kafka® Connect
- 62. 62 Fail Fast Source TopicMessages Sink Messages https://cnfl.io/connect-dlq
- 63. 63 You Only LiveOnce... Source Topic Messages Sink Messages https://cnfl.io/connect-dlq errors.tolerance=all
- 64. 64 Dead Letter Queue SourceTopic Messages Sink Messages https://cnfl.io/connect-dlq errors.tolerance=all errors.deadletterqueue.topic.name=my_dlq Dead Letter Queue
- 65. 65 Re-processing the DeadLetter Queue Source Topic Messages Sink Messages https://cnfl.io/connect-dlq Dead Letter Queue Sink Messages AVRO Sink JSON Sink
- 66. 66 No fields foundusing key and value schemas for table: foo-bar
- 67. 67 No fields foundusing key and value schemas for table: foo-bar JsonDeserializer with schemas.enable requires "schema" and "payload" fields and may not contain additional fields
- 68. 68 Schema where areYou... No fields found using key and value schemas for table: foo-bar JsonDeserializer with schemas.enable requires "schema" and "payload" fields and may not contain additional fields
- 69. 69 Schema where areYou... $ http localhost:8081/subjects[...] jq '.schema|fromjson' { "type": "record", "name": "Value", "namespace": "asgard.demo.ORDERS", "fields": [ { "name": "id", "type": "int" }, { "name": "order_id", "type": [ "null", "int" ], "default": null }, { "name": "customer_id", "type": [ "null", "int" ], "default": null }, $ kafkacat -b localhost:9092 -C -t mysql-debezium-asgard.demo.ORDERS QF@Land RoverDefender 90SheffieldSwift LLC,54258 Michigan Parkway(2019-05-09T13:42:28Z(2019-05-09T13:42:28Z Schema (from Schema Registry) Messages (AVRO)
- 70. 70 Schema where areYou... { "id": 7, "order_id": 7, "customer_id": 849, "order_total_usd": 98062.21, "make": "Pontiac", "model": "Aztek", "delivery_city": "Leeds", "delivery_company": "Price-Zieme", "delivery_address": "754 Becker Way", "CREATE_TS": "2019-05-09T13:42:28Z", "UPDATE_TS": "2019-05-09T13:42:28Z" } No Schema! Messages (JSON) No fields found using key and value schemas for table: foo-bar
- 71. 71 Schema where areYou... You need a schema! Use Avro, or use JSON with schemas.enable=true Either way, you need to re-configure the producer of the data Or, use KSQL to impose a schema on the JSON/CSV data and re-serialise it to Avro 👍🏻 No fields found using key and value schemas for table: foo-bar
- 72. 72 Schema where areYou... Schema (Embedded per JSON message) { "schema": { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, [...] ], "optional": true, "name": "asgard.demo.ORDERS.Value" }, "payload": { "id": 611, "order_id": 111, "customer_id": 851, "order_total_usd": 182190.93, "make": "Kia", "model": "Sorento", "delivery_city": "Swindon", "delivery_company": "Lehner, Kuvalis and Schaefer", "delivery_address": "31 Cardinal Junction", "CREATE_TS": "2019-05-09T13:54:59Z", "UPDATE_TS": "2019-05-09T13:54:59Z" } Messages (JSON) JsonDeserializer with schemas.enable requires "schema" and "payload" fields and may not contain additional fields
- 73. 25%OFF Standard Priced Conference Pass ConfluentCommunity Discount Code KS19Meetup
- 74. 7474 11. November 2019 SteigenbergerFrankfurter Hof 13. November 2019 NOVOTEL Zürich City West Ben Stopford Office of the CTO Confluent Axel Löhn Senior Project Manager Deutsche Bahn Kai Waehner, Technologist Confluent Ralph Debusmann IoT Solution Architect Bosch Power Tools cnfl.io/cse19frankfurt cnfl.io/cse19zurich
- 75.