Download as PDF, PPTX
Uploaded byDatabricks
Stream All Things—Patterns of Modern Data Integration with Gwen Shapira
This document discusses patterns for modern data integration using streaming data. It outlines an evolution from data warehouses to data lakes to streaming data. It then describes four key patterns: 1) Stream all things (data) in one place, 2) Keep schemas compatible and process data on, 3) Enable ridiculously parallel single message transformations, and 4) Perform streaming data enrichment to add additional context to events. Examples are provided of using Apache Kafka and Kafka Connect to implement these patterns for a large hotel chain integrating various data sources and performing real-time analytics on customer events.
More Related Content
Similar to Stream All Things—Patterns of Modern Data Integration with Gwen Shapira
Stream All Things—Patterns of Modern Data Integration with Gwen Shapira
- 1. 1 Stream All Things Patternsof Modern Data Integration
- 2. 2 About Me • Working@confluent • Wrote a book or two • Product manager • Apache Kafka PMC member • Used to be an engineer, consultant, DBA • @gwenshap • Github.com/gwenshap
- 3. 3 Evolution of DataIntegration Data Warehouse Data Lake Data Stream • Difficult modeling • Difficult ingest • Missing data • Limited throughput • Batch only • Still batch only • Data quality is suspect • Data model left as exercise for each developer • Where is my data? • Kept the scale • Reduced latency • Totally new model • Data producers are 1st class participants • Microservices? Apps? • Still figuring the whole thing out
- 4. 4 Evolution of DataIntegration DWH Data Lake Stream Data ETL Engineers Software Engineers
- 5. 5 Evolution of IT 2000 Softwareengineers Data modelers DBAs Sysadmins Storage admins Network admins ETL engineers QA …. 2017 Software engineers
- 6. 6 The line between applicationdevelopment and ETL is blurring.
- 7.
- 8. 8 Few Patterns 1. Streamall things (in one place) 2. Keep Compatible and Process On 3. Ridiculously Parallel Data Integration 4. Streaming Data Enrichment
- 9.
- 10.
- 12. 12 PageViewEvent { sessionId: 676fc8983gu563, timestamp: 1413215458, viewType:"propertyView", propertyId: 7879, loyaltyId: 6764532 origin: "promotion", ...... lots of metadata.... }
- 14.
- 15. 15 Pattern #1 –Put all streams in One Kafka
- 22.
- 23. 23 APIs between servicesare Contracts In Stream Processing World – Event Schemas ARE the API …except they stick around for a lot longer
- 25. 25 Schema Registry Elastic Cassandra Streams App Example Consumers Serializer App1 Serializer App 2 ! Kafka Topic! Schema Registry Define the expected fields for each Kafka topic Automatically handle schema changes (e.g. new fields) Verify Schema Compatibility at every stage of dev cycle Supports multi-datacenter environments
- 26. 26 Pattern #3 –Ridiculously parallel data integration { sessionId: 676fc8983gu563, timestamp: 1413215458, viewType: "propertyView", propertyId: 7879, loyaltyId: 6764532, origin: "promotion", cc: “4444—3333-2222-1111” } { sessionId: 676fc8983gu563, timestamp: 1413215458, viewType: "propertyView", propertyId: 7879, loyaltyId: 6764532, origin: "promotion", cc: “xxxx—xxxx-xxxx-1111” }
- 27.
- 28. 28 Is there adata store involved? KafkaConnect can help
- 29. 29 Ecosystem of Connectors DatabasesDatastore/File Store Analytics Applications / Other
- 30. 30 How Connect Works? Log Connector MQTT Connector RESTAPI Logs MQTT Log Task Log Task MQTT Task MQTT Task
- 31. 31 Connect + SingleMessage Transformations Mix: Ridiculously parallel database integration With Ridiculously parallel simple transformations
- 32. 32 Promotion! We want tosend Platinum members Who are looking at beach properties An email about Discount package in new Hawaii hotel
- 33. 33 Pattern #4 –Enriching events From Event sessionID timestamp loyaltyID propertyID To Enriched Event sessionID timestamp loyaltyID Level isPlatium propertyID Location isBeach
- 40. 40 We use Kafka,Connect and CDC Connectors To turn state (in DB) Into stream of events (in Kafka) And back into state (in application cache)
- 42. 42 In reality… • Maintainingthe distributed cache isn’t trivial • How do we persist state? Failover? Recover? • How do we co-partition for joins? KafkaStreams makes it easy: Ktable loyaltyTbl = builder.table(…, “loyalty-cdc-topic”) PageViewStream.leftJoin(loyaltyTbl, <operation>) Example: https://www.confluent.io/blog/distributed-real-time-joins-and-aggregations-on-user-activity-events- using-kafka-streams/
- 43. Few Patterns 1. Streamall things (in one place) 2. Keep Compatible and Process On 3. Ridiculously Parallel Single Message Transformations 4. Streaming Data Enrichment
- 44.