Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)


Published on

High level introduction to Confluent REST Proxy and Schema Registry (leveraging Apache Avro under the hood), two components of the Apache Kafka open source ecosystem. See the concepts, architecture and features.

Published in: Software
  • Be the first to comment

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)

  1. 1. Confluent REST PROXY and SCHEMA REGISTRY The Rise of Apache Kafka as Streaming Platform Kai Waehner Technology Evangelist LinkedIn @KaiWaehner
  2. 2. 2 Apache Kafka A Distributed, Scalable Commit Log
  3. 3. 3 Apache Kafka A Distributed, Scalable Commit Log
  4. 4. 4 Apache Kafka The Rise of a Streaming Platform
  5. 5. 5 Orders Customers Payments Stock Apache Kafka Single Shared Source of Truth for (Micro)Services
  6. 6. 6 Confluent Ecosystem
  7. 7. 7 Confluent Platform - Reference Architecture
  8. 8. REST Proxy Produce and Consume Messages via HTTP(S)
  9. 9. 99 Apache Kafka Native Clients Confluent Native Clients Community Supported Clients Kafka Clients
  10. 10. 1010 REST Proxy Non-Java Applications Native Kafka Java Applications Schema Registry REST / HTTP(S) Simplifies administrative actions Simplifies message creation and consumption Provides a RESTful interface to a Kafka cluster Talking to Non-native Kafka Apps and Outside the Firewall Confluent REST Proxy
  11. 11. 1111 • Producers • Accepts produce requests targeted at specific topics or partitions and routes them all through a small pool of producers. • Consumers • Stateful consumption of messages. Simple Consumer or low-level read operations, for example to retrieve messages at specific offsets • Metadata • Most metadata about the cluster – brokers, topics, partitions, and configs • Data Formats • JSON, raw bytes encoded with base64 or using JSON-encoded Avro. With Avro, schemas are registered and validated against the Schema Registry. REST Proxy Features
  12. 12. 1212 Produce and Consume Messages
  13. 13. 13 REST Proxy Clusters and Load Balancing • Designed to support multiple instances running together to spread load and can safely be run behind various load balancing mechanisms (e.g. round robin DNS, discovery services, load balancers) • Sticky Load Balancer needed for Consumers because consumers are stateful. REST Proxy will store the offsets.
  14. 14. 14 Getting Started in Minutes…
  15. 15. 15 REST Proxy Security Plugin (Enterprise) REST Proxy Non-Java Applications Native Kafka Java Applications Schema Registry REST / HTTPS Simplifies administrative security Simplifies Security and ACLs Provides a Secure RESTful interface to a Kafka cluster Security Plugins for REST Proxy to pass through client credentials to Kafka Broker (rather than making all clients look like one single Kafka client)
  16. 16. Schema Registry Schema Management for Kafka Topics (including Schema Evolution)
  17. 17. 17 APIs are contracts between services
  18. 18. 18 But not all services talk to each other directly
  19. 19. 19 And naturally…
  20. 20. 20 Schema are APIs We need specifications We need to make changes to them We need to detect breaking changes We need versions We need tools
  21. 21. 21 Enter Schema Registry
  22. 22. 22 What do Schema Registries do? 1. Store schemas – put / get 2. Link one or more schema to each event 3. Clients (Java, .NET, REST, …) that fetch & cache schemas • Producers sending messages • Consumers receiving messages 4. Enforcement of compatibility rules 5. Graphical browser
  23. 23. 23 Version compatibility?
  24. 24. 24 Forward compatibility Forward compatibility means data written with newer schema is readable with old schemas.
  25. 25. 25 Forward & Backward compatibility Backward compatibility means data written with older schema is readable with a newer schema.
  26. 26. 26 Confluent Schema Registry • Define the expected fields for each Kafka topic • Automatically handle schema changes (e.g. new fields) • Prevent backwards incompatible changes • Multiple schemas per topic • Supports multi-datacenter environments • Leverages Apache Kafka for Fail-Over and Schema Management • Allows CI/CD with Maven Integration
  27. 27. 27 Multiple Schemas per Topic
  28. 28. 28 REST API
  29. 29. 29 Apache Avro
  30. 30. 30 Confluent Schema Registry Elastic HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic ! Schema Registry Deserializer Deserializer
  31. 31. 31 Producers contain Serializers Define the serializers: props.put("key.serializer", ”org.apache.kafka.serializers.StringSerializer"); props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); props.put("schema.registry.url", schemaUrl); … producer<String, LogLine> producer = new KafkaProducer<String, LogLine>(props); Create a record: ProducerRecord<String, LogLine> record = new ProducerRecord<String, LogLine>(topic, event.getIp().toString(), event); Send the record: producer.send(record); Consumers deserialize the same way…
  32. 32. 32 Serializers cache schemas, register new schema … and serialize serialize(topic, isKey, object): subject = getSubjectName(topic, isKey) schema = getSchema(record) schemaIdMap = schemaCache.get(subject) if (schemaIdMap.containsKey(schema): id = schemaIdMap.get(schema) else id = registerAndGetId(subject, schema) schemaIdMap.put(schema, id) output = MAGIC_BYTE + id + avroWriter(schema, object)
  33. 33. 33 Schema Registry caches schemas and validates compatibility register(schema, subject): if (schemaIsNewToSubject): prevSchema = getPrevSchema(subject) level = getCompatibilityLevel(subject) if (level == FULL): validator = new SchemaValidatorBuilder().mutualReadStrategy().validateLatest() if (validator.isCompatible(schema, prevSchema)) register else throw …
  34. 34. 34 Getting Started in Minutes…
  35. 35. 35 Schema Registry UI in Confluent Control Center • See the schema per topic for each key and value • Version history • Compare previous and current schema • Schema Management – create, edit, delete (later in 2018)
  36. 36. 36 Schema Registry Security Plugin (Enterprise)
  37. 37. 37
  38. 38. 38 Confluent Cloud • Apache Kafka™ as a Service Freedom to use Kafka in any public cloud Access the rich Confluent ecosystem Develop faster using the leading streaming platform Liberate yourself from the operations burden Including REST Proxy and Schema Registry
  39. 39. Questions? Kai Waehner Technology Evangelist LinkedIn @KaiWaehner