The document discusses analyzing streaming data using streaming algorithms. It explains how to calculate the mean of a dataset incrementally as new data points are added by keeping a running sum and incrementing the count of data points. It also questions how to calculate the median incrementally as new data points are added. The document then discusses real-time tradeoffs around location, availability, throughput, and latency for high-volume data. Finally, it concludes that big data is also about analyzing small things quickly and making the data accessible.

1. V
#bigdataMY

2. V
olume
elocity
ariety
#bigdataMY

3. #bigdataMY

4. #bigdataMY

5. Feeds and notifications
Insights
Recommendation & Matching
Security
Monitoring & Reporting
Event logging
#bigdataMY

6. Feeds and notifications
Insights Change detection
Recommendation & Matching Change reaction
Security Audit
Monitoring & Reporting
Event logging
#bigdataMY

7. Get ahead of the curve
Noise
Ø
“Normal”
Ø
Ø Ø
Ø
Ø Ø
Ø
Ø Ø
ØØ Ø
Ø Ø Ø
Ø
Ø
Ø Ø
Ø
ØØ
Ø Ø Ø Ø
Ø Ø
Ø Ø Ø
Ø
“Normal” Ø Ø
Ø Ø
“Normal”
Ø
Ø
Ø
Noise
[J Gama, University of Porto] #bigdataMY

8. Get ahead of the curve
Noise
“New concept”
Ø
Ø Ø Ø
“Normal” Ø Ø
Ø
Ø Ø
Ø Ø
Ø
Ø Ø
Ø
Ø Ø
ØØ Ø
Ø Ø
Ø
Ø
“Concept
Ø
Ø
Ø
Ø drift”
ØØ
Ø Ø Ø
Ø
Ø Ø
Ø Ø Ø
Ø Ø Ø “Normal”
“Normal” Ø Ø Ø Ø
Ø
“Big Data is much more likely to catch the
black swan as it swoops in”
Noise - Norman Nie, Revolution Analytics
[J Gama, University of Porto] #bigdataMY

9. Acunu Analytics
#bigdataMY

10. #bigdataMY

11. UserID
EMEA
UK
London
N1
Female
16-21 year old
16-21 year old
Female
16-21 year old
Female
London
#bigdataMY

12. V Under the hood
21:00 all = 1345 :00 = 45 :01 = 62 ...
22:00 all = 3221 :00 = 22 :01 = 19 ...
... ...
UK all = 228 user01 = 1 user14 = 12 ...
US all = 354 user01 = 15 user14 = 0 ...
MY all = 28 user01 = 0 user02 = 0 ...
...
#bigdataMY

13. V Under the hood
21:00 all = 1345 :00 = 45 :01 = 62 ...
22:00 all = 3221 +1 :00 = 22 :01 = 19 +1 ...
{
cust_id: user01, ... ...
session_id: 102,
geography: UK,
UK all = 228 +1 user01 = 1 +1 user14 = 12 ...
browser: IE,
time: 22:01,
} US all = 354 user01 = 15 user14 = 0 ...
MY all = 28 user01 = 0 user02 = 0 ...
...
#bigdataMY

14. V
where time 21:00 - 22:00
count(*)
Under the hood
21:00 all = 1345 :00 = 45 :01 = 62 ...
22:00 all = 3221 :00 = 22 :01 = 19 ...
... ...
UK all = 228 user01 = 1 user14 = 12 ...
US all = 354 user01 = 15 user14 = 0 ...
MY all = 28 user01 = 0 user02 = 0 ...
...
#bigdataMY

15. V
where time 21:00 - 23:00
count(*)
Under the hood
21:00 all = 1345 :00 = 45 :01 = 62 ...
22:00 all = 3221 :00 = 22 :01 = 19 ...
... ...
UK all = 228 user01 = 1 user14 = 12 ...
US all = 354 user01 = 15 user14 = 0 ...
MY all = 28 user01 = 0 user02 = 0 ...
...
#bigdataMY

16. Little Trouble with Big Disks
#bigdataMY

17. COTS Journal, 2008
#bigdataMY

18. V
where time 21:00 - 23:00
count(*)
Under the hood
21:00 all = 1345 :00 = 45 :01 = 62 ...
22:00 all = 3221 :00 = 22 :01 = 19 ...
... ...
UK all = 228 user01 = 1 user14 = 12 ...
US all = 354 user01 = 15 user14 = 0 ...
MY all = 28 user01 = 0 user02 = 0 ...
...
#bigdataMY

19. Streaming algorithms
A = [a1, a2, a3, a4, a5]
mean(A) = sum it up / number of things
#bigdataMY

20. Streaming algorithms
A = [a1, a2, a3, a4, a5]
mean(A) = sum it up / number of things
now add another item a6...???
#bigdataMY

21. Streaming algorithms
A = [a1, a2, a3, a4, a5]
mean(A) = sum it up / number of things
now add another item a6...???
sum = sum + a6
inc(number of things)
#bigdataMY

22. Streaming algorithms
A = [a1, a2, a3, a4, a5]
mean(A) = sum it up / number of things
now add another item a6...???
sum = sum + a6
inc(number of things)
try this with median?
#bigdataMY

23. V Realtime tradeoffs
ity
loc
Ad
-ve
-ho
gh
c
Hi
High-volume
#bigdataMY

24. V Conclusion
Big Data also about the Little Things, done fast.
The devil is in the details.
Make it accessible.
#bigdataMY

25. V
Q?
#bigdataMY