Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Harmony in TuneHow we Refactored Cube   to Terabyte Scale   Philip (flip) Kromer    Huston Hoburg     infochimps.com       ...
Big Data for All
Big Data for All
why dashboards?
Lightweight Dashboards•   Understand what’s happening•   Understand data in context•   NOT exploratory analytics•   real-t...
The “Church of Graphs”
Predictive Kvetching
Lightweight Dashboards
Approach to Tuning• Measure: “Why can’t it be faster?”• Harmonize: “Use it right”• Tune: “Align it to production resources”
cube is awesome
What’s so great?• Streaming, real-time• Ad-hoc data: write whatever you want• Ad-hoc queries: make up new queries whenever...
Event Stream•{   time: "2013-02-15T01:02:03Z",     type: "webreq", data: {       path: "/order", method: "POST",       dur...
Events vs MetricsEvent: •{    time: "2013-02-15T01:02:03Z",       type: "tweet", id: 8675309, data: {         text: "Mongo...
Events vs MetricsEvent: •{   time: "2013-02-15T01:02:03Z",      type: "webreq", data: {        path: "/order", method: "PO...
Events vs Metrics•   Events:              { time: "2013-02-15T01:02:03Z",                           type: "webreq",    • b...
Events vs Metrics• Events:             { time: "2013-02-15T01:02:03Z",                        type: "webreq", • baskets of...
billions and billions
3000 events/second
tuning methodology
Monkey See Monkey Do            Google for            the #s the            cool kids use
Spinal Tap        Turn        everything        to 11!!!!
Hillbilly Mechanic            Rewrite for            memcached            HBase on            Cassandra!!!
Moneybags       SSD plz       Moar CPU       Moar RAM       Moar Replica
Tuning How to do it• Measure: “Why can’t it be faster?”• Harmonize: “Use it right”• Tune: “Align it to production resources”
see through the magic
• Why can’t it be faster than it is now?
• dstat (http://j.mp/dstatftw): dstat   -drnycmf -t 5• htop• mongostat
Grok: client-side• Made a sprayer to inject data • invalidate a time range at max speed • writes variously-shaped data: no...
Fault injection• raise when packet comes in with certain flag •{   time: "2013...", data: {...},      _raise:"db_write" }• ...
app-side tracing  metalog.event(connect,    { method: ws,      ip: connection.remoteAddress,      path: request.url }, min...
app-side tracing
fits on machine
3000 events/second• Rate: • 3000 ev/sec ≈ 250 M ev/day ≈ 2 BILLION/wk• Expensive. Difficult. • 250 GB accumulated per day ...
Metrics• Rate: • 3M tensec/year (π· 10 sec/year)                         7 • < 100 bytes/metric ...• Manageable! • a 30 me...
20% gains are boringAt scale, your first barriers are either:• Easy• ImpossibleMetrics: 10 GB/yearEvents: 10 TB/month
Scalability síPerformance no
Still CPU and Memory Use • Problem  • Mongo seems to be working  • but high resident memory and fault rate  • Memory-mappe...
Capped Collections• Fixed size circular queue• records are in order of insertion         A        B   C   D A   E         ...
Capped Collections• Extremely efficient on write   A          B    C     D A     E         F• Extremely efficient for insert...
don’t like the answer?change the question.
mainlineuncapped eventscapped metrics:metrics are a view on data
hi-scale branchcapped eventsuncapped metrics:events are ephemeral
Harmony• Make your pattern of access  match your system’s strengths and rhythm
Validate Mental Model
Easy fixes• Duplicate requests = duplicate calculations • Cube patch for request queues exists • Easy fix!• Non-pyramidal ar...
cube 101
Cube Systems
Collector• Receives events• writes to MongoDB• marks metrics for re-calculation (“invalidates”)
Evaluator• receives, parses requests for metrics• calculates metrics “pyramidally”• then stores them, cached
Pyramidal Aggregation                                  90                                                                 ...
Pyramidal Aggregation                                      5min                                      1min1 5 2 0 2 0 6 4 7...
Uses Cached Results                                                              5min    10          20          15       ...
Pyramidal Aggregation• calculates metrics... • from metrics and constants ... from metrics ...    • from events• (then sto...
fast writes
how fast can we write?
how fast can we write?       FASTstreaming writes: way efficient
locked out
Writes and Invalidations
Inserts Stop Every 5s •   working •   working •   ANGRY •   ANGRY •   working •   working
Thanks, mongostat!•   working•   working•   ANGRY                     ...•   ANGRY•   working•   working                  ...
Inserts Stop Every 5s Events Collection   ...G               H                       C                   D A              ...
Inserts Stop Every 5s Events Collection   ...G               H                       C                   D A              ...
Inserts Stop Every 5s• What’s really going on? • Database write locks • Events and metrics have conflicting locks • Solutio...
fast reads
Pre-cache Metrics• Keep metrics fresh (Warmer)• Only calculate recent updates (Horizons)
fancy metrics
Non-pyramidal Aggregates • Can’t calculate from warmed metrics • Store values with counts in metrics  • Counts can be vivi...
finally, scaling
Multicore• MongoDB • Writes limited to single core • Requires sharding for multicore
Multicore• Cube (node.js) • Concurrent, but not multi-threaded• Easy solution • Multiple collectors on different ports • P...
Multicore
Hardware• High Memory • Capped events size scale with memory• CPU • Mongo / cube not optimized for multicore • Faster core...
Cloud helps• Tune machines to application• Dedicating databases for each application makes life  a lot easier
Cloud helps• Tune machines to application•
jobs@infochimps.comgithub.com/   infochimps-labs
good ideas that  didn’t help
Queues• Different queueing methods• Should optimize metric calculations • No significant improvement
Locks: update VS remove   • Uncapped metrics allow ‘remove’ as     invalidation option   • Remove doesn’t help with databa...
Mongo Aggregations• Mongo has aggregations!• Node ends up working better • Mongo aggregations aren’t faster • Less flexible...
Why not Graphite?• Data model • Metrics-centric vs Events-centric    (metrics code not intertwingled with app code)• Envir...
Upcoming SlideShare
Loading in …5
×

Playing in Tune: How We Refactored Cube to Terabyte Scale

4,522 views

Published on

  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT, ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Playing in Tune: How We Refactored Cube to Terabyte Scale

  1. 1. Harmony in TuneHow we Refactored Cube to Terabyte Scale Philip (flip) Kromer Huston Hoburg infochimps.com Feb 15 2013
  2. 2. Big Data for All
  3. 3. Big Data for All
  4. 4. why dashboards?
  5. 5. Lightweight Dashboards• Understand what’s happening• Understand data in context• NOT exploratory analytics• real-time insight... but not just about real-timemainline: j.mp/sqcubehi-scale branch: j.mp/icscube
  6. 6. The “Church of Graphs”
  7. 7. Predictive Kvetching
  8. 8. Lightweight Dashboards
  9. 9. Approach to Tuning• Measure: “Why can’t it be faster?”• Harmonize: “Use it right”• Tune: “Align it to production resources”
  10. 10. cube is awesome
  11. 11. What’s so great?• Streaming, real-time• Ad-hoc data: write whatever you want• Ad-hoc queries: make up new queries whenever• Efficient (“pyramidal”) calculations
  12. 12. Event Stream•{ time: "2013-02-15T01:02:03Z", type: "webreq", data: { path: "/order", method: "POST", duration: 50.7, status: 400, ua:"...MSIE 6.0..." } }•{ time: "2013-02-15T01:02:03Z", type: "tweet", id: 8675309, data: { text: "MongoDB talk yay", retweet_count: 121, user: { screen_name: "infochimps", followers_count: 7851, lang: "en", ...} } }
  13. 13. Events vs MetricsEvent: •{ time: "2013-02-15T01:02:03Z", type: "tweet", id: 8675309, data: { text: "MongoDB talk yay", retweet_count: 121, user: { screen_name: "infochimps", followers_count: 7851, lang: "en", ...} } }Metrics:• “# of tweets in 10s bucket at 1:02:10 on 2013-02-15”• “# of non-english-language tweets in 1hr bucket at ...”
  14. 14. Events vs MetricsEvent: •{ time: "2013-02-15T01:02:03Z", type: "webreq", data: { path: "/order", method: "POST", duration: 50.7, status: 400, ua:"...MSIE 6.0..." } }Metrics:• “# of requests in 10s bucket at 3:05:10 on 2013-02-15”• “Average duration of requests with 4xx status in the 5 minute bucket at 3:05:00 on 2013-02-15”
  15. 15. Events vs Metrics• Events: { time: "2013-02-15T01:02:03Z", type: "webreq", • baskets of facts data: { path: "/order", method: "POST", • narcissistic duration: 50.7, status: 400, ua:"...MSIE 6.0..." } } • LOTS AND LOTS
  16. 16. Events vs Metrics• Events: { time: "2013-02-15T01:02:03Z", type: "webreq", • baskets of facts data: { path: "/order", method: "POST", • narcissistic duration: 50.7, status: 400, ua:"...MSIE 6.0..." } } • LOTS AND LOTS• Metrics: • a timestamped number • look like the graph{ time: "2013-02-15T01:02:03Z", value: 90 } • one per time bucket
  17. 17. billions and billions
  18. 18. 3000 events/second
  19. 19. tuning methodology
  20. 20. Monkey See Monkey Do Google for the #s the cool kids use
  21. 21. Spinal Tap Turn everything to 11!!!!
  22. 22. Hillbilly Mechanic Rewrite for memcached HBase on Cassandra!!!
  23. 23. Moneybags SSD plz Moar CPU Moar RAM Moar Replica
  24. 24. Tuning How to do it• Measure: “Why can’t it be faster?”• Harmonize: “Use it right”• Tune: “Align it to production resources”
  25. 25. see through the magic
  26. 26. • Why can’t it be faster than it is now?
  27. 27. • dstat (http://j.mp/dstatftw): dstat -drnycmf -t 5• htop• mongostat
  28. 28. Grok: client-side• Made a sprayer to inject data • invalidate a time range at max speed • writes variously-shaped data: noise, ramp, sine, etc• Or just reach into the DB and poke • delete range of metrics, leave events • delete range of events, leave metrics
  29. 29. Fault injection• raise when packet comes in with certain flag •{ time: "2013...", data: {...}, _raise:"db_write" }• (only in development mode, obvs.)
  30. 30. app-side tracing metalog.event(connect, { method: ws, ip: connection.remoteAddress, path: request.url }, minor);• “Metalog” announces lifecycle progress: • writes to log... • ... or as cube metrics!
  31. 31. app-side tracing
  32. 32. fits on machine
  33. 33. 3000 events/second• Rate: • 3000 ev/sec ≈ 250 M ev/day ≈ 2 BILLION/wk• Expensive. Difficult. • 250 GB accumulated per day (@1000 bytes/ev) • 95 TB accumulated per year (@1000 bytes/ev)
  34. 34. Metrics• Rate: • 3M tensec/year (π· 10 sec/year) 7 • < 100 bytes/metric ...• Manageable! • a 30 metric dashboard is ~ 10 GB/year @10sec • a 30 metric dashboard is ~ 170 MB/year @ 5min
  35. 35. 20% gains are boringAt scale, your first barriers are either:• Easy• ImpossibleMetrics: 10 GB/yearEvents: 10 TB/month
  36. 36. Scalability síPerformance no
  37. 37. Still CPU and Memory Use • Problem • Mongo seems to be working • but high resident memory and fault rate • Memory-mapped Files • 1Tb data served by 4Gb ram is no good
  38. 38. Capped Collections• Fixed size circular queue• records are in order of insertion A B C D A E F• oldest records are discarded when full ...G H C D A E F G ...
  39. 39. Capped Collections• Extremely efficient on write A B C D A E F• Extremely efficient for insertion-order reads• Very efficient if queries are ‘local’ • events in same timebucket typically arrived at nearby times and so are nearby on disk
  40. 40. don’t like the answer?change the question.
  41. 41. mainlineuncapped eventscapped metrics:metrics are a view on data
  42. 42. hi-scale branchcapped eventsuncapped metrics:events are ephemeral
  43. 43. Harmony• Make your pattern of access match your system’s strengths and rhythm
  44. 44. Validate Mental Model
  45. 45. Easy fixes• Duplicate requests = duplicate calculations • Cube patch for request queues exists • Easy fix!• Non-pyramidal are inefficient • Remove until things are under control • ( solve paralyzing problems first )
  46. 46. cube 101
  47. 47. Cube Systems
  48. 48. Collector• Receives events• writes to MongoDB• marks metrics for re-calculation (“invalidates”)
  49. 49. Evaluator• receives, parses requests for metrics• calculates metrics “pyramidally”• then stores them, cached
  50. 50. Pyramidal Aggregation 90 5min 10 20 15 25 10 10 1min1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2 5 5 4 6 4 1 2 7 0 0 0 1 6 0 0 1 0 3 10s ev ev ev ev ev ev ...
  51. 51. Pyramidal Aggregation 5min 1min1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2 10s ev ev ev ev ev ev ...
  52. 52. Uses Cached Results 5min 10 20 15 25 10 1min1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2 5 5 4 6 4 1 2 7 0 0 0 1 10s ev ev ev ev ev ev ...
  53. 53. Pyramidal Aggregation• calculates metrics... • from metrics and constants ... from metrics ... • from events• (then stores them, cached) 5 min 1 min 10 secev ev ev ev ev....
  54. 54. fast writes
  55. 55. how fast can we write?
  56. 56. how fast can we write? FASTstreaming writes: way efficient
  57. 57. locked out
  58. 58. Writes and Invalidations
  59. 59. Inserts Stop Every 5s • working • working • ANGRY • ANGRY • working • working
  60. 60. Thanks, mongostat!• working• working• ANGRY ...• ANGRY• working• working (simulated)
  61. 61. Inserts Stop Every 5s Events Collection ...G H C D A E F G ... hi-speed writes localized reads Metrics Collection . . . . . . . . . . . . . . . . . . . . . x x xx x x . . . . . . . . . . . . x . randomish hi-speed reads deletes updates
  62. 62. Inserts Stop Every 5s Events Collection ...G H C D A E F G ... hi-speed writes localized reads Metrics Collection . . . . . . . . . . . . . . . . . . . . . x x xx x x . . . . . . . . . . . . x . randomish hi-speed reads deletes updates
  63. 63. Inserts Stop Every 5s• What’s really going on? • Database write locks • Events and metrics have conflicting locks • Solution: split the databases Events Collection ...G H C D A E F G ... hi-speed writes localized reads Metrics Collection . . . . . . . . . . . . . . . . . . . . . x x xx x x . . . . . . . . . . . . x . randomish hi-speed reads deletes
  64. 64. fast reads
  65. 65. Pre-cache Metrics• Keep metrics fresh (Warmer)• Only calculate recent updates (Horizons)
  66. 66. fancy metrics
  67. 67. Non-pyramidal Aggregates • Can’t calculate from warmed metrics • Store values with counts in metrics • Counts can be vivified for aggregations • Smaller footprint than full events • Works best for dense, finite values
  68. 68. finally, scaling
  69. 69. Multicore• MongoDB • Writes limited to single core • Requires sharding for multicore
  70. 70. Multicore• Cube (node.js) • Concurrent, but not multi-threaded• Easy solution • Multiple collectors on different ports • Produces redundant invalidations • Requires external load balancing
  71. 71. Multicore
  72. 72. Hardware• High Memory • Capped events size scale with memory• CPU • Mongo / cube not optimized for multicore • Faster cores• EC2 Best value: m2.2xlarge • < $700/mo, 34.2GB RAM, 13 bogo-hertz
  73. 73. Cloud helps• Tune machines to application• Dedicating databases for each application makes life a lot easier
  74. 74. Cloud helps• Tune machines to application•
  75. 75. jobs@infochimps.comgithub.com/ infochimps-labs
  76. 76. good ideas that didn’t help
  77. 77. Queues• Different queueing methods• Should optimize metric calculations • No significant improvement
  78. 78. Locks: update VS remove • Uncapped metrics allow ‘remove’ as invalidation option • Remove doesn’t help with database locks • It was a stupid idea anyway: that’s OK • “Hey, poke it and see what happens!”
  79. 79. Mongo Aggregations• Mongo has aggregations!• Node ends up working better • Mongo aggregations aren’t faster • Less flexible • Would require query language rewrite
  80. 80. Why not Graphite?• Data model • Metrics-centric vs Events-centric (metrics code not intertwingled with app code)• Environment familiarity • Cube: d3, node.js, mongo • Graphite: Django, Whisper, C

×