Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
DNS for Data
The need for a Stream Registry
August 28, 2017
Get HomeAway from it all
Praveen HirsaveRené X. Parra
“Revolutionizing travel through the power of technology.”
History
2015 Original Twin Drain Problem
2015
Fill the Lake
What we did
2015
Success
Success?
What we wanted
2015
Title Text
2015
Producer Config
Topic Config
Kafka Monitoring
Zookeeper Monitoring
So we asked for some help
2015
Then we got acquired!
2016
Migrate
to AWS!
2016 - Microservices, Microservices, Microservices
2016
µ2016
2016
µ
2016
µ • Service Registration
• Service Discovery
• Health Checks
• Circuit Breakers
• Resilient
- control plane
“DNS for ...
Essence of micro services and streams
Commands
Queries
Events
Domain2016
SYNC
ASYNC
ρ2016
2017
ρ
ρ- control plane • Stream Registration
• Stream Discovery
• Stream Health
• Throttling
• Multi-Region / 

Multi-Cloud /

M...
Topic Registration (old, manual way)
• A Wiki page
• Team
• Owner
• Consumers (known)
• Topic
• Schema
• Partition Key
• N...
Stream Registration
• Portfolio
• Product
• Service
• Stream
• Schema
• Partition Key
• Num of Partitions
• Config
Confluent...
Stream Registration: Modelling
Stream Registry
2017
Code First Design First
Stream Registration: Model Library
Stream Registry
2017
Stream Discovery
Stream Registry
2017
Elastic Search
Stream Discovery Portal
Stream Health
Stream Registry
2017
Stream Throttling
2017
Multi - Region
2017
Region 1 Region 2
Producer
{topic}.local
Producer
{topic}.local
Produce “locally”
Multi - Region
2017
Region 1 Region 2
Producer
{topic}.local
Producer
{topic}.local
Produce “locally”
Consumer
{topic}.glo...
Multi - Region
2017
Region 1 Region 2
Producer
{topic}.local
Producer
{topic}.local
Produce “locally”
Consumer
{topic}.glo...
Multi - Region
2017
Region 1 Region 2
Producer
{topic}.local
Producer
{topic}.local
Produce “locally”
Consumer
{topic}.glo...
Stream Registry, Actors and Streams
2017
Multi-Region
Stream Registry
Stream Registry
Stream Registry
DATA-CENTER
US-WEST-1 US-EAST-1
2017
Multi-Cloud
Stream Registry
Stream Registry
Stream Registry
2017
Multi-Company
Stream Registry
Stream Registry
Stream Registry
2017
What we wanted
Self Service Streams
• Log Events
• Domain Events
• Message Bus
• Third Party
Clean Data Lake
• HDFS
• S3
•...
Recap
µ- control plane
“DNS for µ-services”
ρ- control plane
“DNS for ρ-streams”
Recap
µ ρ+ = $
Key Takeaway
Democratizing Streams requires a Stream Registry
Stream Registry
Title Text
Thank you
Upcoming SlideShare
Loading in …5
×

Kafka Summit SF 2017 - DNS for Data: The Need for a Stream Registry

643 views

Published on

Kafka Summit SF 2017 by Praveen Hirsave and Rene Parra

Published in: Engineering
  • Be the first to comment

Kafka Summit SF 2017 - DNS for Data: The Need for a Stream Registry

  1. 1. DNS for Data The need for a Stream Registry August 28, 2017 Get HomeAway from it all
  2. 2. Praveen HirsaveRené X. Parra
  3. 3. “Revolutionizing travel through the power of technology.”
  4. 4. History
  5. 5. 2015 Original Twin Drain Problem
  6. 6. 2015 Fill the Lake
  7. 7. What we did 2015 Success Success?
  8. 8. What we wanted 2015
  9. 9. Title Text 2015 Producer Config Topic Config Kafka Monitoring Zookeeper Monitoring
  10. 10. So we asked for some help 2015
  11. 11. Then we got acquired! 2016 Migrate to AWS!
  12. 12. 2016 - Microservices, Microservices, Microservices 2016
  13. 13. µ2016
  14. 14. 2016 µ
  15. 15. 2016 µ • Service Registration • Service Discovery • Health Checks • Circuit Breakers • Resilient - control plane “DNS for µ-services”
  16. 16. Essence of micro services and streams Commands Queries Events Domain2016 SYNC ASYNC
  17. 17. ρ2016
  18. 18. 2017 ρ
  19. 19. ρ- control plane • Stream Registration • Stream Discovery • Stream Health • Throttling • Multi-Region / 
 Multi-Cloud /
 Multi-Company “DNS for streams” 2017
  20. 20. Topic Registration (old, manual way) • A Wiki page • Team • Owner • Consumers (known) • Topic • Schema • Partition Key • Num of Partitions
  21. 21. Stream Registration • Portfolio • Product • Service • Stream • Schema • Partition Key • Num of Partitions • Config Confluent Schema Registry Stream Registry Topic Manager 2017
  22. 22. Stream Registration: Modelling Stream Registry 2017 Code First Design First
  23. 23. Stream Registration: Model Library Stream Registry 2017
  24. 24. Stream Discovery Stream Registry 2017 Elastic Search Stream Discovery Portal
  25. 25. Stream Health Stream Registry 2017
  26. 26. Stream Throttling 2017
  27. 27. Multi - Region 2017 Region 1 Region 2 Producer {topic}.local Producer {topic}.local Produce “locally”
  28. 28. Multi - Region 2017 Region 1 Region 2 Producer {topic}.local Producer {topic}.local Produce “locally” Consumer {topic}.global Consumer {topic}.global Consume “globally”
  29. 29. Multi - Region 2017 Region 1 Region 2 Producer {topic}.local Producer {topic}.local Produce “locally” Consumer {topic}.global Consumer {topic}.global Consume “globally” Replicate “locally” Mirror-Maker Mirror-Maker
  30. 30. Multi - Region 2017 Region 1 Region 2 Producer {topic}.local Producer {topic}.local Produce “locally” Consumer {topic}.global Consumer {topic}.global Mirror-Maker {topic}.global Mirror-Maker Consume “globally” Replicate “globally” Replicate “locally” Mirror-Maker Mirror-Maker
  31. 31. Stream Registry, Actors and Streams 2017
  32. 32. Multi-Region Stream Registry Stream Registry Stream Registry DATA-CENTER US-WEST-1 US-EAST-1 2017
  33. 33. Multi-Cloud Stream Registry Stream Registry Stream Registry 2017
  34. 34. Multi-Company Stream Registry Stream Registry Stream Registry 2017
  35. 35. What we wanted Self Service Streams • Log Events • Domain Events • Message Bus • Third Party Clean Data Lake • HDFS • S3 • Google Cloud Storage Streaming Data Kafka 2017
  36. 36. Recap µ- control plane “DNS for µ-services” ρ- control plane “DNS for ρ-streams”
  37. 37. Recap µ ρ+ = $
  38. 38. Key Takeaway Democratizing Streams requires a Stream Registry Stream Registry
  39. 39. Title Text Thank you

×