Real time analytics with Netty, Storm, Kafka

10,526 views

Published on

Published in: Technology
1 Comment
34 Likes
Statistics
Notes
No Downloads
Views
Total views
10,526
On SlideShare
0
From Embeds
0
Number of Embeds
93
Actions
Shares
0
Downloads
165
Comments
1
Likes
34
Embeds 0
No embeds

No notes for slide

Real time analytics with Netty, Storm, Kafka

  1. 1. Real-time Analytics with Netty, Apache Kafka and Storm Case study with “lambda architecture” http://nguyentantrieu.info Update: 07/06/2013
  2. 2. Agenda 1. Overview Architecture 2. Log HTTP-Handler and producer: Netty 4 3. Kafka 0.8 (Stream Data Log Storage) 4. Storm Analytics Cluster
  3. 3. Overview System Architecture
  4. 4. Concept Flow
  5. 5. Concept Flow JavaScript Tracking Mobile SDK Http Log Server Kafka
  6. 6. S2 HTTP Log Server netty framework 4
  7. 7. Netty.io Netty is a non-blocking I/O (NIO) client-server framework for the development of Java network applications such as protocol servers and clients. The asynchronous event-driven network application framework and tools are used to simplify network programming such as TCP and UDP socket servers.[2] Netty includes an implementation of the reactor pattern of programming. http://en.wikipedia.org/wiki/Netty_(software) http://nguyentantrieu.info/blog/backend-system-with-netty-io
  8. 8. Apache Kafka (version 0.8) https://cwiki.apache.org/confluence/display/KAFKA/Index
  9. 9. In Production ● Clustering ( 4 nodes) ● Partitions ○ user-activity: 24 partitions ● Producer and consumer are replication aware
  10. 10. Core Producer Class (S2 HTTP Log Server)
  11. 11. Storm Analytics Cluster
  12. 12. The Storm Topology Kafka Cluster topic: user-activity Tokenizer Bolt Parser Bolt Aggregate Bolt Redis Statistics Bolt Raw Data Save DWH Bolt Kafka Consumer Spout
  13. 13. Useful Links ● https://storm.incubator.apache.org ● https://www.youtube.com/watch? v=DRgs8L4AOgc ● https://cwiki.apache. org/confluence/display/KAFKA/Index ● http://nguyentantrieu.info/blog/backend-system- with-netty-io ●

×