Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real time analytics with Netty, Storm, Kafka


Published on

Published in: Technology
  • Login to see the comments

Real time analytics with Netty, Storm, Kafka

  1. 1. Real-time Analytics with Netty, Apache Kafka and Storm Case study with “lambda architecture” Update: 07/06/2013
  2. 2. Agenda 1. Overview Architecture 2. Log HTTP-Handler and producer: Netty 4 3. Kafka 0.8 (Stream Data Log Storage) 4. Storm Analytics Cluster
  3. 3. Overview System Architecture
  4. 4. Concept Flow
  5. 5. Concept Flow JavaScript Tracking Mobile SDK Http Log Server Kafka
  6. 6. S2 HTTP Log Server netty framework 4
  7. 7. Netty is a non-blocking I/O (NIO) client-server framework for the development of Java network applications such as protocol servers and clients. The asynchronous event-driven network application framework and tools are used to simplify network programming such as TCP and UDP socket servers.[2] Netty includes an implementation of the reactor pattern of programming.
  8. 8. Apache Kafka (version 0.8)
  9. 9. In Production ● Clustering ( 4 nodes) ● Partitions ○ user-activity: 24 partitions ● Producer and consumer are replication aware
  10. 10. Core Producer Class (S2 HTTP Log Server)
  11. 11. Storm Analytics Cluster
  12. 12. The Storm Topology Kafka Cluster topic: user-activity Tokenizer Bolt Parser Bolt Aggregate Bolt Redis Statistics Bolt Raw Data Save DWH Bolt Kafka Consumer Spout
  13. 13. Useful Links ● ● v=DRgs8L4AOgc ● https://cwiki.apache. org/confluence/display/KAFKA/Index ● with-netty-io ●