Building Kafka-powered Activity Stream

61 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
61
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Building Kafka-powered Activity Stream

  1. 1. BUILDING KAFKA-POWERED MESSAGE BUS Activity Stream and beyond
  2. 2. CHALLENGES ➤ Needed a way to aggregate project-related activity information generated by various services in one place ➤ Needed a messaging backbone to carry messages between services
  3. 3. SOLUTIONS ➤ Activity Stream on the product side ➤ Message Bus on the tech side
  4. 4. “A Message Bus is a combination of a common data model, a common command set, and a messaging infrastructure to allow different systems to communicate through a shared set of interfaces. - "Enterprise Integration Patterns"
  5. 5. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. by
  6. 6. WHY KAFKA?
  7. 7. KAFKA IS FAST A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.
  8. 8. KAFKA IS SCALABLE ➤ Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. ➤ It can be elastically and transparently expanded without downtime. ➤ Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers.
  9. 9. KAFKA IS DURABLE ➤ Messages are persisted on disk and replicated within the cluster to prevent data loss. ➤ Each broker can handle terabytes of messages without performance impact.
  10. 10. KAFKA IS DISTRIBUTED BY DESIGN Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
  11. 11. GENERAL SETUP
  12. 12. READING / WRITING TO A TWO-PARTITION TOPIC
  13. 13. CONSUMER GROUPS
  14. 14. MESSAGE BUS GEM A simple Ruby interface to Apache Kafka. https://github.com/crealytics/message_bus
  15. 15. Camato Shopping … service Mesosphere GewgawConsumer Bishop OUR SETUP Message Bus Producer Message Bus Producer … service Message Bus Producer Kafka cluster Message Bus Consumer TrafficDirector run DBActivity Stream … service Message Bus Consumer event message event message persistence
  16. 16. ACTIVITY STREAM
  17. 17. NEXT STEPS ➤ Keep adding messages (and message types) to Activity Stream ➤ Keep improving MessageBus gem ➤ Ensure proper production-ready Kafka setup ➤ Use Message Bus for decoupling service APIs and inter- service communication ➤ Asynchronous only, needs more research ➤ Input from Martin Mauch: use CrealyticsLogger to push JSON messages to Kafka (?) — under consideration
  18. 18. DEMO
  19. 19. DANKE!

×