Harmony intune final

599 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
599
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Harmony intune final

  1. 1. Harmony in TuneHow we Refactored Cube to Terabyte Scale Philip (flip) Kromer Huston Hoburg infochimps.com Feb 15 2013
  2. 2. Big Data for All
  3. 3. Big Data for All
  4. 4. why dashboards?
  5. 5. Lightweight Dashboards• Understand what’s happening• Understand data in context• NOT exploratory analytics• real-time insight... but not just about real-timemainline: j.mp/sqcubehi-scale branch: j.mp/icscube
  6. 6. The “Church of Graphs”
  7. 7. Predictive Kvetching
  8. 8. Lightweight Dashboards
  9. 9. Approach to Tuning• Measure: “Why can’t it be faster?”• Harmonize: “Use it right”• Tune: “Align it to production resources”
  10. 10. cube is awesome
  11. 11. What’s so great?• Streaming, real-time• Ad-hoc data: write whatever you want• Ad-hoc queries: make up new queries whenever• Efficient (“pyramidal”) calculations
  12. 12. Event Stream•{ time: "2013-02-15T01:02:03Z", type: "webreq", data: { path: "/order", method: "POST", duration: 50.7, status: 400, ua:"...MSIE 6.0..." } }•{ time: "2013-02-15T01:02:03Z", type: "tweet", id: 8675309, data: { text: "MongoDB talk yay", retweet_count: 121, user: { screen_name: "infochimps", followers_count: 7851, lang: "en", ...} } }
  13. 13. Events vs MetricsEvent: •{ time: "2013-02-15T01:02:03Z", type: "tweet", id: 8675309, data: { text: "MongoDB talk yay", retweet_count: 121, user: { screen_name: "infochimps", followers_count: 7851, lang: "en", ...} } }Metrics:• “# of tweets in 10s bucket at 1:02:10 on 2013-02-15”• “# of non-english-language tweets in 1hr bucket at ...”
  14. 14. Events vs MetricsEvent: •{ time: "2013-02-15T01:02:03Z", type: "webreq", data: { path: "/order", method: "POST", duration: 50.7, status: 400, ua:"...MSIE 6.0..." } }Metrics:• “# of requests in 10s bucket at 3:05:10 on 2013-02-15”• “Average duration of requests with 4xx status in the 5 minute bucket at 3:05:00 on 2013-02-15”
  15. 15. Events vs Metrics• Events: { time: "2013-02-15T01:02:03Z", type: "webreq", • baskets of facts data: { path: "/order", method: "POST", • narcissistic duration: 50.7, status: 400, ua:"...MSIE 6.0..." } } • LOTS AND LOTS
  16. 16. Events vs Metrics• Events: { time: "2013-02-15T01:02:03Z", type: "webreq", • baskets of facts data: { path: "/order", method: "POST", • narcissistic duration: 50.7, status: 400, ua:"...MSIE 6.0..." } } • LOTS AND LOTS• Metrics: • a timestamped number • look like the graph{ time: "2013-02-15T01:02:03Z", value: 90 } • one per time bucket
  17. 17. billions and billions
  18. 18. 3000 events/second
  19. 19. tuning methodology
  20. 20. Monkey See Monkey Do Google for the #s the cool kids use
  21. 21. Spinal Tap Turn everything to 11!!!!
  22. 22. Hillbilly Mechanic Rewrite for memcached HBase on Cassandra!!!
  23. 23. Moneybags SSD plz Moar CPU Moar RAM Moar Replica
  24. 24. Tuning How to do it• Measure: “Why can’t it be faster?”• Harmonize: “Use it right”• Tune: “Align it to production resources”
  25. 25. see through the magic
  26. 26. • Why can’t it be faster than it is now?
  27. 27. • dstat (http://j.mp/dstatftw): dstat -drnycmf -t 5• htop• mongostat
  28. 28. Grok: client-side• Made a sprayer to inject data • invalidate a time range at max speed • writes variously-shaped data: noise, ramp, sine, etc• Or just reach into the DB and poke • delete range of metrics, leave events • delete range of events, leave metrics
  29. 29. Fault injection• raise when packet comes in with certain flag •{ time: "2013...", data: {...}, _raise:"db_write" }• (only in development mode, obvs.)
  30. 30. app-side tracing metalog.event(connect, { method: ws, ip: connection.remoteAddress, path: request.url }, minor);• “Metalog” announces lifecycle progress: • writes to log... • ... or as cube metrics!
  31. 31. app-side tracing
  32. 32. fits on machine
  33. 33. 3000 events/second• Rate: • 3000 ev/sec ≈ 250 M ev/day ≈ 2 BILLION/wk• Expensive. Difficult. • 250 GB accumulated per day (@1000 bytes/ev) • 95 TB accumulated per year (@1000 bytes/ev)
  34. 34. Metrics• Rate: • 3M tensec/year (π· 10 sec/year) 7 • < 100 bytes/metric ...• Manageable! • a 30 metric dashboard is ~ 10 GB/year @10sec • a 30 metric dashboard is ~ 170 MB/year @ 5min
  35. 35. 20% gains are boringAt scale, your first barriers are either:• Easy• ImpossibleMetrics: 10 GB/yearEvents: 10 TB/month
  36. 36. Scalability síPerformance no
  37. 37. Still CPU and Memory Use • Problem • Mongo seems to be working • but high resident memory and fault rate • Memory-mapped Files • 1Tb data served by 4Gb ram is no good
  38. 38. Capped Collections• Fixed size circular queue• records are in order of insertion A B C D A E F• oldest records are discarded when full ...G H C D A E F G ...
  39. 39. Capped Collections• Extremely efficient on write A B C D A E F• Extremely efficient for insertion-order reads• Very efficient if queries are ‘local’ • events in same timebucket typically arrived at nearby times and so are nearby on disk
  40. 40. don’t like the answer?change the question.
  41. 41. mainlineuncapped eventscapped metrics:metrics are a view on data
  42. 42. hi-scale branchcapped eventsuncapped metrics:events are ephemeral
  43. 43. Harmony• Make your pattern of access match your system’s strengths and rhythm
  44. 44. Validate Mental Model
  45. 45. Easy fixes• Duplicate requests = duplicate calculations • Cube patch for request queues exists • Easy fix!• Non-pyramidal are inefficient • Remove until things are under control • ( solve paralyzing problems first )
  46. 46. cube 101
  47. 47. Cube Systems
  48. 48. Collector• Receives events• writes to MongoDB• marks metrics for re-calculation (“invalidates”)
  49. 49. Evaluator• receives, parses requests for metrics• calculates metrics “pyramidally”• then stores them, cached
  50. 50. Pyramidal Aggregation 90 5min 10 20 15 25 10 10 1min1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2 5 5 4 6 4 1 2 7 0 0 0 1 6 0 0 1 0 3 10s ev ev ev ev ev ev ...
  51. 51. Pyramidal Aggregation 5min 1min1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2 10s ev ev ev ev ev ev ...
  52. 52. Uses Cached Results 5min 10 20 15 25 10 1min1 5 2 0 2 0 6 4 7 1 0 2 2 3 2 4 2 2 5 5 4 6 4 1 2 7 0 0 0 1 10s ev ev ev ev ev ev ...
  53. 53. Pyramidal Aggregation• calculates metrics... • from metrics and constants ... from metrics ... • from events• (then stores them, cached) 5 min 1 min 10 secev ev ev ev ev....
  54. 54. fast writes
  55. 55. how fast can we write?
  56. 56. how fast can we write? FASTstreaming writes: way efficient
  57. 57. locked out
  58. 58. Writes and Invalidations
  59. 59. Inserts Stop Every 5s • working • working • ANGRY • ANGRY • working • working
  60. 60. Thanks, mongostat!• working• working• ANGRY ...• ANGRY• working• working (simulated)
  61. 61. Inserts Stop Every 5s Events Collection ...G H C D A E F G ... hi-speed writes localized reads Metrics Collection . . . . . . . . . . . . . . . . . . . . . x x xx x x . . . . . . . . . . . . x . randomish hi-speed reads deletes updates
  62. 62. Inserts Stop Every 5s Events Collection ...G H C D A E F G ... hi-speed writes localized reads Metrics Collection . . . . . . . . . . . . . . . . . . . . . x x xx x x . . . . . . . . . . . . x . randomish hi-speed reads deletes updates
  63. 63. Inserts Stop Every 5s• What’s really going on? • Database write locks • Events and metrics have conflicting locks • Solution: split the databases Events Collection ...G H C D A E F G ... hi-speed writes localized reads Metrics Collection . . . . . . . . . . . . . . . . . . . . . x x xx x x . . . . . . . . . . . . x . randomish hi-speed reads deletes
  64. 64. fast reads
  65. 65. Pre-cache Metrics• Keep metrics fresh (Warmer)• Only calculate recent updates (Horizons)
  66. 66. fancy metrics
  67. 67. Non-pyramidal Aggregates • Can’t calculate from warmed metrics • Store values with counts in metrics • Counts can be vivified for aggregations • Smaller footprint than full events • Works best for dense, finite values
  68. 68. finally, scaling
  69. 69. Multicore• MongoDB • Writes limited to single core • Requires sharding for multicore
  70. 70. Multicore• Cube (node.js) • Concurrent, but not multi-threaded• Easy solution • Multiple collectors on different ports • Produces redundant invalidations • Requires external load balancing
  71. 71. Multicore
  72. 72. Hardware• High Memory • Capped events size scale with memory• CPU • Mongo / cube not optimized for multicore • Faster cores• EC2 Best value: m2.2xlarge • < $700/mo, 34.2GB RAM, 13 bogo-hertz
  73. 73. Cloud helps• Tune machines to application• Dedicating databases for each application makes life a lot easier
  74. 74. Cloud helps• Tune machines to application•
  75. 75. jobs@infochimps.comgithub.com/ infochimps-labs
  76. 76. good ideas that didn’t help
  77. 77. Queues• Different queueing methods• Should optimize metric calculations • No significant improvement
  78. 78. Locks: update VS remove • Uncapped metrics allow ‘remove’ as invalidation option • Remove doesn’t help with database locks • It was a stupid idea anyway: that’s OK • “Hey, poke it and see what happens!”
  79. 79. Mongo Aggregations• Mongo has aggregations!• Node ends up working better • Mongo aggregations aren’t faster • Less flexible • Would require query language rewrite
  80. 80. Why not Graphite?• Data model • Metrics-centric vs Events-centric (metrics code not intertwingled with app code)• Environment familiarity • Cube: d3, node.js, mongo • Graphite: Django, Whisper, C

×