This document discusses a real-time analytics architecture using Netty, Apache Kafka, and Storm. It includes an overview of the system architecture with Netty handling HTTP logging and producing to Apache Kafka for stream data storage. Storm analytics clusters then consume from Kafka topics to perform analytics functions like tokenization, parsing, aggregation, and saving results to Redis and a data warehouse.

1. Real-time Analytics
with Netty, Apache
Kafka and Storm
Case study with “lambda architecture”
http://nguyentantrieu.info
Update: 07/06/2013

2. Agenda
1. Overview Architecture
2. Log HTTP-Handler and producer: Netty 4
3. Kafka 0.8 (Stream Data Log Storage)
4. Storm Analytics Cluster

3. Overview System Architecture

4. Concept Flow

5. Concept Flow
JavaScript Tracking
Mobile SDK
Http Log
Server
Kafka

6. S2 HTTP Log Server
netty framework 4

7. Netty.io
Netty is a non-blocking I/O (NIO) client-server framework
for the development of Java network applications such as
protocol servers and clients.
The asynchronous event-driven network application
framework and tools are used to simplify network
programming such as TCP and UDP socket servers.[2]
Netty includes an implementation of the reactor pattern of
programming.
http://en.wikipedia.org/wiki/Netty_(software)
http://nguyentantrieu.info/blog/backend-system-with-netty-io

8. Apache Kafka (version 0.8)
https://cwiki.apache.org/confluence/display/KAFKA/Index

9. In Production
● Clustering ( 4 nodes)
● Partitions
○ user-activity: 24 partitions
● Producer and consumer are replication
aware

10. Core Producer Class
(S2 HTTP Log Server)

11. Storm Analytics Cluster

12. The Storm Topology
Kafka Cluster
topic: user-activity
Tokenizer
Bolt
Parser
Bolt
Aggregate
Bolt
Redis
Statistics Bolt
Raw Data
Save DWH
Bolt
Kafka
Consumer
Spout

13. Useful Links
● https://storm.incubator.apache.org
● https://www.youtube.com/watch?
v=DRgs8L4AOgc
● https://cwiki.apache.
org/confluence/display/KAFKA/Index
● http://nguyentantrieu.info/blog/backend-system-
with-netty-io
●