Acunu Analytics @ Cassandra London

803 views

Published on

My talk about Acunu Analytics - for video see http://skillsmatter.com/podcast/nosql/acunu-analytics

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
803
On SlideShare
0
From Embeds
0
Number of Embeds
22
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Acunu Analytics @ Cassandra London

  1. 1. Acunu Analytics Realtime Big Data Analytics Tom Wilkie, Acunu 16th July 2012
  2. 2. • Motivation / alternatives • What is it? • How does it work? • Whats it good for?2 Analytics
  3. 3. • Motivation / alternatives • What is it? • How does it work? • Whats it good for?3 Analytics
  4. 4. time page session id duration time page session id duration time ...page session id duration ... time ... time page ... page session id ...... ... ... duration ... page session id duration ... time 14:58:03.234 time ... /index.html page session id 175 ...... ... ... duration ... ... 248.180.3.40 session id 175 duration 14:58:03.234 time... 14:58:03.234 time /index.html page ... /index.html page 248.180.3.40 session id 175 ...... ... ... duration ... 14:58:03.234 /csi/csi/council/freedom.html 14:58:03.409 ... time ... 248.180.3.40 /index.html page 248.180.3.40 session id session id 175 duration ... 248.180.3.40 1234 ... 14:58:03.409 ... time /index.html page 248.180.3.40 session id duration /csi/csi/council/freedom.html ... 248.180.3.40 1234 175 ... ... /index.html page 248.180.3.40 session id duration 14:58:03.234 /docs/access/chapter8.txt ...... page 248.180.3.40 ...session id ...... 14:58:03.234 /csi/csi/council/freedom.html 14:58:03.409 ... time 248.180.3.40 1234 175 duration /csi/csi/council/freedom.html 99.1.10.178 52 /docs/access/chapter8.txt ... page 248.180.3.40 ...session id duration 14:58:03.409 ... time 14:58:03.877 14:58:03.234 /index.html 248.180.3.40 1234 175 14:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 14:58:03.409 ... time /index.html 99.1.10.178 52 248.180.3.40 1234 175 ... ... 52 1234 175 duration 14:58:03.409 ... /csi/csi/council/freedom.html 99.1.10.178 /docs/access/chapter8.txt ... page 248.180.3.40 session id 14:58:03.87714:58:03.234 time 248.180.3.40 ...session id ...... 248.180.3.40 1234 175 duration 14:58:03.877 /index.html 248.180.3.40 14:58:03.877 /docs/access/chapter8.txt 14:58:03.409 ... time/docs/access/chapter8.txt ...99.1.10.178 /csi/csi/council/freedom.html 99.1.10.178 /index.html page 52 ...52 248.180.3.40 session id duration 14:58:03.234 14:58:03.877 /docs/access/chapter8.txt 14:58:03.877 14:58:03.234 time /docs/access/chapter8.txt ...99.1.10.178 /csi/csi/council/freedom.html 99.1.10.178 /index.html page 52 ... 1234 175 ... 52 52 ... 1234 175 duration 248.180.3.40 14:58:03.409 ... /docs/access/chapter8.txt /docs/access/chapter8.txt ...99.1.10.178 99.1.10.178 52 14:58:03.87714:58:03.409 ...... /csi/csi/council/freedom.html 99.1.10.17852 52session id 175 ...... /docs/access/chapter8.txt /index.html page 248.180.3.40 14:58:03.877 14:58:03.234 time /csi/csi/council/freedom.html 14:58:03.409 /docs/access/chapter8.txt 99.1.10.178 248.180.3.40 session id duration 99.1.10.178 248.180.3.40... 1234 14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 52 /docs/access/chapter8.txt ......99.1.10.178 14:58:03.877 14:58:03.234 time 14:58:03.877 14:58:03.409 ... /csi/csi/council/freedom.html 99.1.10.17852 52 ... 1234 175 duration /docs/access/chapter8.txt /index.html page 248.180.3.40 14:58:03.877 14:58:03.877 /docs/access/chapter8.txt 52 14:58:03.877 14:58:03.409 ... /csi/csi/council/freedom.html 99.1.10.17852 52 ... 1234 175 ...... 14:58:03.877 14:58:03.234 248.180.3.40 /docs/access/chapter8.txt /index.html 99.1.10.178 248.180.3.40 52 99.1.10.178 14:58:03.877 /docs/access/chapter8.txt248.180.3.40 14:58:03.877 14:58:03.234 /docs/access/chapter8.txt ... 248.180.3.40 /docs/access/chapter8.txt /index.html 99.1.10.178 248.180.3.40 52 14:58:03.877 /csi/csi/council/freedom.html 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 14:58:03.409 ... /docs/access/chapter8.txt ... 1234 14:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 99.1.10.17852 52 ... 1234 175 ... 99.1.10.178 248.180.3.40 /index.html 99.1.10.178 248.180.3.40 /csi/csi/council/freedom.html /docs/access/chapter8.txt /index.html 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 14:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 99.1.10.17852 52 52 1234 175 99.1.10.178 1234 248.180.3.40 248.180.3.40 14:58:03.877 /csi/csi/council/freedom.html 248.180.3.40 /docs/access/chapter8.txt /index.html 99.1.10.178 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 1234 52 /docs/access/chapter8.txt/csi/csi/council/freedom.html 99.1.10.17852 52 52 1234 175 14:58:03.87714:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 99.1.10.178 248.180.3.40 1234 175 14:58:03.409 /docs/access/chapter8.txt 99.1.10.17852248.180.3.40 /docs/access/chapter8.txt /index.html 99.1.10.178 248.180.3.40 52 /csi/csi/council/freedom.html 99.1.10.178 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 1234 52 /docs/access/chapter8.txt /csi/csi/council/freedom.html 99.1.10.178 248.180.3.40 14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 99.1.10.178 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 99.1.10.17852248.180.3.40 14:58:03.87714:58:03.877 14:58:03.234 /csi/csi/council/freedom.html 99.1.10.17852 52 52 1234 1234 14:58:03.877 14:58:03.877 /docs/access/chapter8.txt 248.180.3.40 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852248.180.3.40 /docs/access/chapter8.txt/csi/csi/council/freedom.html 99.1.10.17852 52 52 1234 1234 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 99.1.10.178 248.180.3.40 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 14:58:03.409 /docs/access/chapter8.txt 99.1.10.17852248.180.3.40 /csi/csi/council/freedom.html/docs/access/chapter8.txt 1234 99.1.10.17852 52 52 1234 248.180.3.40 /docs/access/chapter8.txt/csi/csi/council/freedom.html 14:58:03.877 14:58:03.877 /docs/access/chapter8.txt 248.180.3.40 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 1234 99.1.10.178 14:58:03.409 14:58:03.87714:58:03.409 14:58:03.877 /csi/csi/council/freedom.html/docs/access/chapter8.txt 1234 99.1.10.17852 52 52 /docs/access/chapter8.txt 248.180.3.40 /csi/csi/council/freedom.html 99.1.10.178 /docs/access/chapter8.txt 248.180.3.40123452 1234 99.1.10.178 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 99.1.10.178 1234 52 52 52 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt 14:58:03.87714:58:03.877 /csi/csi/council/freedom.html/docs/access/chapter8.txt /docs/access/chapter8.txt 248.180.3.40 52 99.1.10.178 248.180.3.40 99.1.10.178 /docs/access/chapter8.txt /docs/access/chapter8.txt 99.1.10.178 99.1.10.17852 52 52 14:58:03.877 /docs/access/chapter8.txt 99.1.10.17852 99.1.10.178 14:58:03.409 14:58:03.877 /docs/access/chapter8.txt /docs/access/chapter8.txt 248.180.3.40 14:58:03.877 14:58:03.409 14:58:03.877 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 248.180.3.40 99.1.10.178 14:58:03.877 /csi/csi/council/freedom.html/docs/access/chapter8.txt 1234 52 1234 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 52 52 14:58:03.87714:58:03.877 /docs/access/chapter8.txt 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 1234 /docs/access/chapter8.txt /docs/access/chapter8.txt /csi/csi/council/freedom.html 99.1.10.178 /docs/access/chapter8.txt 248.180.3.401234 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 52 52 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 99.1.10.178 14:58:03.87714:58:03.877 /docs/access/chapter8.txt /docs/access/chapter8.txt /csi/csi/council/freedom.html 99.1.10.178 /docs/access/chapter8.txt 248.180.3.401234 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 52 52 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 99.1.10.178 14:58:03.87714:58:03.877 /docs/access/chapter8.txt /docs/access/chapter8.txt /csi/csi/council/freedom.html 99.1.10.178 /docs/access/chapter8.txt 248.180.3.401234 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 52 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 99.1.10.178 14:58:03.87714:58:03.877 /docs/access/chapter8.txt /csi/csi/council/freedom.html 99.1.10.178 /docs/access/chapter8.txt 248.180.3.40 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 52 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 1234 /docs/access/chapter8.txt /csi/csi/council/freedom.html 99.1.10.178 /docs/access/chapter8.txt 248.180.3.40 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 52 14:58:03.877 14:58:03.409 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 1234 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 /docs/access/chapter8.txt /docs/access/chapter8.txt 248.180.3.40 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 99.1.10.178 52 1234 14:58:03.409 /docs/access/chapter8.txt 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 /csi/csi/council/freedom.html 99.1.10.17852 1234 248.180.3.40 14:58:03.409 /docs/access/chapter8.txt 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 /csi/csi/council/freedom.html 99.1.10.17852 1234 52 1234 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 248.180.3.40 14:58:03.409 /docs/access/chapter8.txt /csi/csi/council/freedom.html 99.1.10.178 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt 248.180.3.40 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.17852 1234 52 1234 248.180.3.40 14:58:03.409 /docs/access/chapter8.txt 14:58:03.877 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 99.1.10.17852 1234 52 /docs/access/chapter8.txt 248.180.3.40 14:58:03.409 /docs/access/chapter8.txt 14:58:03.877 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 99.1.10.17852 1234 52 /docs/access/chapter8.txt 248.180.3.40 14:58:03.877 /docs/access/chapter8.txt 14:58:03.877 /csi/csi/council/freedom.html 14:58:03.409 99.1.10.17899.1.10.17852 1234 52 /docs/access/chapter8.txt 248.180.3.40 14:58:03.409 /docs/access/chapter8.txt 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 248.180.3.40 52 1234 14:58:03.409 /docs/access/chapter8.txt 14:58:03.877 /csi/csi/council/freedom.html 99.1.10.178 248.180.3.40 52 1234 14:58:03.877 14:58:03.409 /docs/access/chapter8.txt /csi/csi/council/freedom.html 99.1.10.178 248.180.3.40 52 1234 14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 52 14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 52 14:58:03.877 /docs/access/chapter8.txt 99.1.10.178 524 Analytics
  5. 5. Combining “big” and “real-time” is hard Live & historical Drill downs Trends... aggregates... and roll ups5 Analytics
  6. 6. Solution Con Scalability $$$ Not realtime Inefficient Recomputation Spartan query semantics => complex, DIY solutions6 Analytics
  7. 7. • Motivation / alternatives • What is it? • How does it work? • Whats it good for?7 Analytics
  8. 8. Analytics counter updatesClick stream events AcunuSensor data Analytics etc • Simple, real-time, incremental analytics • Push processing into ingest phase
  9. 9. { time : TIME(HOUR; MIN; SEC), page : PATH(/), category : STRING, loadTime : LONG } { select : ["COUNT", "AVG(loadTime)"], where : “time, ?path”, group : “time, ?category” }9 Analytics
  10. 10. • Motivation / alternatives • What is it? • How does it work? • Whats it good for?10 Analytics
  11. 11. Introduction11 Analytics
  12. 12. count grouped by ... day count distinct(session) count ... geographyavg(duration) ... browser12 Analytics
  13. 13. time : TIME(HOUR; MIN; SEC), cust_id : LONG, Data session_id : LONG, Definition geography : STRING, browser : STRING, load_time : LONG { select: “COUNT” patterns: [ { where : “?time”, group : “?time” }, Query { where : “”, group : “geography” }, { where : “”, group : “browser” } Patterns ] }, { select: [“COUNT_DISTINCT(session_id)”, “AVG(load_time)”], where: “time”, group: “” }13 Analytics
  14. 14. 21:00 all→1345 :00→45 :01→62 :02→87 ... 22:00 all→3221 :00→22 :00→19 :02→104 ...{ cust_id: user01, ... ... session_id: 102, UK all→228 user01→1 user14→12 user99→7 ... geography: UK, US all→354 user01→4 user04→8 user56→17 ... browser: IE, time: 22:02, ...} UK, 22:00 all→1904 ... ∅ all→87314 UK→238 US→354 ...14 Analytics
  15. 15. 21:00 all→1345 :00→45 :01→62 :02→87 ... 22:00 all→3222 :00→22 :00→19 :02→105 ...{ cust_id: user01, ... ... session_id: 102, UK all→229 user01→2 user14→12 user99→7 ... geography: UK, US all→354 user01→4 user04→8 user56→17 ... browser: IE, time: 22:02, ...} UK, 22:00 all→1905 ... ∅ all→87315 UK→239 US→354 ...15 Analytics
  16. 16. where time 21:00-22:00 count(*) 21:00 all→1345 :00→45 :01→62 :02→87 ...where time 22:00-23:00, 22:00 all→3222 :00→22 :01→19 :02→105 ... group by minute ... ... UK all→229 user01→2 user14→12 user99→7 ...where geography=UK US all→354 user01→4 user04→8 user56→17 ... group all by user, ... UK, 22:00 all→1905 ...count all ∅ all→87315 UK→239 US→354 ...group all by geo16 Analytics
  17. 17. • SUM, COUNT, MIN, MAX, STDDEV, AVG, TOP k, COUNT DISTINCT • Also: approx top k, approx count distinct • Also: idempotent update • RESTful JSON interface, CLI17 Analytics
  18. 18. • Motivation / alternatives • What is it? • How does it work? • Whats it good for?18 Analytics
  19. 19. Manufacturing Social Media Ad Analytics Systems Financial Oil + Gas Monitoring Services Analytics
  20. 20. “We keep discovering use cases we hadn’t thought of ”“We found out a competitor was scraping our data” “Up and running in about 4 hours” Analytics
  21. 21. Thanks! Questions? analytics@acunu.com21 Analytics

×