Streaming data mining is needed to analyze real-time streaming data and extract insights. Traditional batch data mining techniques are not suitable for streaming data due to limitations in processing infinite, continuous data streams with bounded memory and processing data in single passes. Streaming data mining uses techniques like Storm and Hoeffding trees that can process data streams in real-time to detect patterns, perform machine learning, and update models continuously to handle concept drift. This allows analyzing high-volume streaming data and gaining insights from it.

1. Streaming Data Mining
4/18/2014 Streaming Data Mining 1

2. Once upon a time.
• Life was easy
– Eg. Org. has only transaction data, analyst were happy analyzing them.
– Competition was less.
– Customer had lesser options to review product.
• Wait! Web- 2.(oh)0
– Customer who consumed data started generating data - tweets, blogs,
facebook comments, reviews………..
– Another burst came when the Mobile era came in.
• Apps recording customers location
• Actions on apps.
• Pattern of app use.
4/18/2014 Streaming Data Mining 2

3. Server
DB
DB
DB
DB
DB
DB

4. 4/18/2014 Streaming Data Mining
Its All About the Numbers!
4
58M/Day
500Tb/Day 2.1M GB/Hr 4B view/day

5. So, Its GOOD to have data,
Right?

6. Digging Into the Data
• Analyze to understand customer.
• Identify Patterns
• Machine Learning
• Statistical Model Building
• Natural Language Processing
• …….
4/18/2014 Streaming Data Mining 7

7. Usual Pipeline in Data Mining.
4/18/2014 Streaming Data Mining 8
Data of
Entire
Population
Sample
Population
Cleaning and
Preprocessing
Training and testing
Models
Production Server

9. Why?

11. Huge Training Data Set - Volume
• Organizations these days have huge datasets that can be used
to train their models.
• But Main Memory Restrictions.
– Machine Learning Algorithm.
– Batch Processing.
• Y no Sampling??
4/18/2014 Streaming Data Mining 12

12. Streams - Velocity
• Ubiquitous Computing, Mobile Devices, Social Media.
• Potentially of Infinite length
• Usual Strategy – Batch Mode.
4/18/2014 Streaming Data Mining 13

13. Contextual Trends.
• Trending topic on social media.
• Weather
• Location
• Demographics
• Market Dynamics
• Jargon Alert : Concept Drift
4/18/2014 Streaming Data Mining 14

14. What we want today?
Consume Real time data
and extract insights.
Wait.! Can I say Analyze
Streams?

15. Streaming Data Mining!

16. Philosophy
• Continuous Data Record aka Data Streams
• Bounded Storage
• Single Pass
• Real Time
• Concept Drift
4/18/2014 Streaming Data Mining 17

17. So What… We have Hadoop…
• The big Elephant doesn’t fit in here.
• Hadoop – Batch Processing
• We need Storm
– Storm is fast: a benchmark clocked it at
over a million tuples processed per
second per node.
4/18/2014 Streaming Data Mining 18

18. Algorithms.
• The conventional Machine learning algorithm were designed
for batch processing.
– The Algorithm needs to load entire dataset into the memory.
– Computes the necessary statistics, example entropyinformation gain
in decision trees.
• With Streams?
– Streams are of infinite length
– Storing everything, if you can, will be an issue on the memory of the
system $$$$
4/18/2014 Streaming Data Mining 19

19. Streaming Machine Learning
• When?
– High Data volume
– Rate at which data comes is high.
– Unbound, will always arrive in the system and we wont be able to fit it
in our memory
• Requirements to be adhered.
– Each input element to be processed atmost once.
– Space
– Time
– Start predicting from t0
4/18/2014 Streaming Data Mining 20

20. General Flow of Streaming Algorithms
4/18/2014 Streaming Data Mining 21

21. Spam Detection
• Models trained in the past by traditional data mining strategy
will become obsolete as spammers will find a way out.
• Solution : VFDT - Hoeffding Tree Steam Classification
• Train the model in streaming setup.
• When new spam pattern detected, people mark them as
spams.
• Use them to retrain the model in real time.
Concept Drift! Win!
4/18/2014 Streaming Data Mining 22

22. Answering Todays BigData Needs
• Streaming Data Mining
– Storm
– MOA
– SAMOA
– KAFKA
– ……
4/18/2014 Streaming Data Mining 23

23. Thank You!
Ankit Solanki
Neil Shah