Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Kafka-powered Activity Stream

285 views

Published on

  • Be the first to comment

Building Kafka-powered Activity Stream

  1. 1. BUILDING KAFKA-POWERED MESSAGE BUS Activity Stream and beyond
  2. 2. CHALLENGES ➤ Needed a way to aggregate project-related activity information generated by various services in one place ➤ Needed a messaging backbone to carry messages between services
  3. 3. SOLUTIONS ➤ Activity Stream on the product side ➤ Message Bus on the tech side
  4. 4. “A Message Bus is a combination of a common data model, a common command set, and a messaging infrastructure to allow different systems to communicate through a shared set of interfaces. - "Enterprise Integration Patterns"
  5. 5. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. by
  6. 6. WHY KAFKA?
  7. 7. KAFKA IS FAST A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.
  8. 8. KAFKA IS SCALABLE ➤ Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. ➤ It can be elastically and transparently expanded without downtime. ➤ Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers.
  9. 9. KAFKA IS DURABLE ➤ Messages are persisted on disk and replicated within the cluster to prevent data loss. ➤ Each broker can handle terabytes of messages without performance impact.
  10. 10. KAFKA IS DISTRIBUTED BY DESIGN Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
  11. 11. GENERAL SETUP
  12. 12. READING / WRITING TO A TWO-PARTITION TOPIC
  13. 13. CONSUMER GROUPS
  14. 14. MESSAGE BUS GEM A simple Ruby interface to Apache Kafka. https://github.com/crealytics/message_bus
  15. 15. Camato Shopping … service Mesosphere GewgawConsumer Bishop OUR SETUP Message Bus Producer Message Bus Producer … service Message Bus Producer Kafka cluster Message Bus Consumer TrafficDirector run DBActivity Stream … service Message Bus Consumer event message event message persistence
  16. 16. ACTIVITY STREAM
  17. 17. NEXT STEPS ➤ Keep adding messages (and message types) to Activity Stream ➤ Keep improving MessageBus gem ➤ Ensure proper production-ready Kafka setup ➤ Use Message Bus for decoupling service APIs and inter- service communication ➤ Asynchronous only, needs more research ➤ Input from Martin Mauch: use CrealyticsLogger to push JSON messages to Kafka (?) — under consideration
  18. 18. DEMO
  19. 19. DANKE!

×