Big data projects often fail because they lack clear key performance indicators (KPIs) to define success. Without KPIs, projects rely on batch processing which leads to complex "big data" solutions. However, with real-time analysis of events and signals against dynamic KPI thresholds, big data problems can be avoided. The optimal design is a real-time data mining approach that transforms events into signals using rules, selects the most important signals based on top-level KPIs, and identifies incidents when signals cross static or dynamic thresholds.

1. Big Data trap
francis@qmining.com
@fraka6

2. Data/Big Data Knowledge Action
People care about Knowledge/actions not data

3. Agenda
● Big data dilemma
● When are we doing Big Data?
● Maturity/Evolution steps
● The big data trap
● Optimal design = real time data-mining
● Increase your chances of success

4. The Big Data Dilemma

5. Big Data =
Data + IO bounded (disk)
CPU
<100%Data
IO bounded

6. QA
BI
Maturity
Barriers of entry Levels
Just another barrier of entry

7. Trap = no KPI
● No KPI -> batch processing -> big data
● KPI -> real time -> no big data complexity

8. Optimal design = real-time data-mining
● Events -> everything is an event
● + Rule -> create signal from events
● + KPIs -> selection of signals (top level)
● + Incident = signal static/dynamic thresholds
● + Root causes analysis
○ Bayesian inference (ratio signal)
○ Signal correlation (std signal)
○ Rule filtering (domain specific)

9. Increase chances of success
● Data driven culture
● Data quality culture (Avoid logs)
● Reach Analytics/BI level
● KISS

10. Recap
● Big Data = Small Data + IO bound
● Big data->Data->Analytics->Mining->Predictive
○ Data Quality = BIGGEST PROBLEM
○ Big Data = another barrier of entry
● Big data trap = no KPI
● KISS = real time data mining

11. hum...
Questions?
francis@qmining.com