Restats

836 views

Published on

Restats is a highly customized real-time framework that processes data and creates data and geo cubes that later can be queried. It makes use of the data types redis supports, like the sorted-set.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
836
On SlideShare
0
From Embeds
0
Number of Embeds
106
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Restats

  1. 1. r restats Processing real-time data tocubes on data structure servers
  2. 2. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo incement lessons leascaling scales
  3. 3. theProblem r Rails mySQL RedisApplication Warehouse Statistics Trackings Calculate every X hours Doesn’t Scale! 800 M+ 500 GB+
  4. 4. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo incement lessons leascaling scales
  5. 5. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo incement lessons leascaling scales
  6. 6. r theSolution Rails Redis rApplication Statistics Trackings Process restats Real-Time
  7. 7. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo incement lessons leascaling scales
  8. 8. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo incement lessons leascaling scales
  9. 9. what is restats? r Redis Statisticsadvanced key-value store data cubes data structure server geo-location cubes
  10. 10. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo incement lessons leascaling scales
  11. 11. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  12. 12. r how does it work: tracking { Rails "created_at": "2012-10-18T21:34:56+02:00", "type": "adrequest", Produces "location_id": 1, "campaign_id": 1, "publisher_id": 1, JSON "latitude": "53.36669921875", "longitude": "-1.5",Application "bannersize": "320x50" } At 2012-10-18, there was an adrequest for campaign 1 at 53.366,-1.5 for the bannersize 320x50
  13. 13. rhow does it work: dimensions Attribute Geo Time type location_id day latitude campaign_id month longitude publisher_id year bannersize
  14. 14. how does it work: cubing r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize Maps 1 Attribute 1 Redis Key Combination Redis INCR Time Attribute day-month-year type Maps INCR1st Depth day-month-year-type: 18-10-2012 adrequest Redis INCR 18-10-2012-adrequest
  15. 15. how does it work: 1st depth r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize Time Attribute day-month-year type Maps INCR day-month-year-type: 18-10-2012 adrequest Redis INCR 18-10-2012-adrequest Time Attribute month-year type Maps INCR month-year-type: 10-2012 adrequest Redis INCR 10-2012-adrequest Time Attribute year type Maps INCR year-type: 2012 adrequest Redis INCR 2012-adrequest
  16. 16. how does it work: 2nd depth r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize Time Attributeday-month-year type location_id Maps INCR day-month-year-type-location_id: 18-10-2012 adrequest 1 18-10-2012-adrequest-1 Redis INCR Time Attributeday-month-year type campaign_id Maps INCR day-month-year-type-campaign_id: 18-10-2012-adrequest-1 ... 18-10-2012 adrequest 1 Redis INCR Combinations
  17. 17. how does it work: 3rd depth r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize Time Attributeday-month-year type location_id campaign_id Maps INCR day-month-year-type-location_id -campaign_id: 18-10-2012 adrequest 1 1 18-10-2012-adrequest-1-1 Redis INCR Time Attributeday-month-year type location_id publisher_id Maps INCR day-month-year-type-location_id -publisher_id: 18-10-2012-adrequest-1-1 ... 18-10-2012 adrequest 1 1 Redis INCR Combinations
  18. 18. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  19. 19. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  20. 20. how does it work: keys r Attribute Geo Time Depth type location_id day latitude campaign_id month 3 longitude publisher_id year bannersize n = attribute dimensions (5) d = max depth (3) t = time dimensions (3)Total Amount of =Keys per Tracking (5 + 10 + 10) • 3 = 75 Keys
  21. 21. how does it work: increment r Computed Keys day-month-year-type:18-10-2012-adrequest month-year-type:10-2012-adrequest year-type:2012-adrequestday-month-year-type-location_id-publisher_id:18-10-2012-adrequest-1-1 ... Digest SHA1 Redis Batch INCR INCR 76f374viq4vi7fti374oo76 INCR o8qf374fvlqizfvil7v2o8v INCR fqwevlrfli74l3fzlizflzf INCR fbliaguzvf736i4cjsztudz ...
  22. 22. how does it work: geo cubing r Redis Sorted Set set with a score for each member ZADD key score member # adds a member to key ZINCRBY key 1 member # increases member by 1 Mapping Rounded position specifies precision member = “latitude:longitude” score = # of trackings at that location ZINCRBY sdvouzg 1 53.3666:-1.5 ZRANGE sdvouzg 0 -1 withscores 1) 54.3215:0.3456 2) 5 3) 53.3666:-1.5 4) 2763
  23. 23. how does it work: geo increment r Computed Keys INCR 76f374viq4vi7fti374oo76 INCR o8qf374fvlqizfvil7v2o8v INCR fqwevlrfli74l3fzlizflzf INCR fbliaguzvf736i4cjsztudz ... Round Position ZINCRBY 1 Redis Batch ZINCRBY INCR 76f374viq4vi7fti374oo76 ZINCRBY 76f374viq4vi7fti374oo76g 1 53.3666:-1.5 INCR o8qf374fvlqizfvil7v2o8v ZINCRBY o8qf374fvlqizfvil7v2o8vg 1 53.3666:-1.5 INCR fqwevlrfli74l3fzlizflzf ZINCRBY fqwevlrfli74l3fzlizflzfg 1 53.3666:-1.5 INCR fbliaguzvf736i4cjsztudz ZINCRBY fbliaguzvf736i4cjsztudzg 1 53.3666:-1.5 ...
  24. 24. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  25. 25. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  26. 26. rthe queryinterface Query converts the attributes to keys multi-fetch to save round-trip time
  27. 27. the queryresponse r Query-able Ruby Hash allows to iterate over the responses merged aggregations
  28. 28. the geoqueryresponse r Query-able Ruby Hash list of positions scores are trackings at that position
  29. 29. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  30. 30. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  31. 31. the scalingstrategy r Trackings In add worker PR > 0 remove worker PR < 0 monitor value over a time period, e.g. 10 minsProcessing Ratio = Trackings Out
  32. 32. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  33. 33. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  34. 34. the speedtest r2.4 GHz Intel Core i5 with 8 GB 1067 MHz DDR3 5 attribute dimensions 5 max depth geo cube enabled 85 processings / second (5 + 10 + 10 + 5) • 3 • 2 = 180 keys / tracking = 15,300 keys / second bottleneck = cpu; calculation of combinations
  35. 35. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  36. 36. Proble Solutio problem solution What is re Trackin tracking dimensi cubin 1st dep lessons learned dimensions 2nd dep 3rd dep keys restats cubing increme geo cub speed geo increm test 1st, 2nd, 3rd depth query inte query resp query interface keys geo query in scaling str response speed t geo increment lessons leascaling scales
  37. 37. the lessonslearned r not NoSQL, newSQL! we’re still querying, right?dare to customize know your data you’ll be rewarded dynamics kills performance process on write if you need fast read-access denormalize no CRONin this field, it’s your friend recurring tasks don’t scale use mini-benchmarks when you’re unsure during implementation
  38. 38. rnow let’sscale! try scales! itscales.org sthomas fankhauser, m.sc. tommy@southdesign.de southdesign.de

×