Latency: The Silent Monitoring System Killer

2,019 views
1,944 views

Published on

Slides from a lighting talk presented at the January Sydney DevOps Meetup.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,019
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
16
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Latency: The Silent Monitoring System Killer

  1. 1. LATENCY THE SILENT MONITORING SYSTEM KILLERSaturday, 21 January 2012
  2. 2. #MONITORINGSUCKSSaturday, 21 January 2012
  3. 3. SCALABILITYSaturday, 21 January 2012
  4. 4. execute large volumes of monitoring checks under a variety of conditions (good + bad) with a consistent throughputSaturday, 21 January 2012
  5. 5. CONSISTENT THROUGHPUTSaturday, 21 January 2012
  6. 6. SHORT CHECK EXECUTION TIMESaturday, 21 January 2012
  7. 7. WHAT INTRODUCES VARIABILITY?Saturday, 21 January 2012
  8. 8. LATENCY INTRODUCED c SYNCHRONOUS CALLSSaturday, 21 January 2012
  9. 9. Saturday, 21 January 2012
  10. 10. = 1 monitoring checkSaturday, 21 January 2012
  11. 11. Saturday, 21 January 2012
  12. 12. 150 monitoring checks each executed every 300 seconds each takes 1 second checks are executed seriallySaturday, 21 January 2012
  13. 13. all checks executed in 150 seconds monitoring system at 50% capacitySaturday, 21 January 2012
  14. 14. DOUBLE THE CHECKSSaturday, 21 January 2012
  15. 15. Saturday, 21 January 2012
  16. 16. all checks executed in 300 seconds monitoring system at 100% capacitySaturday, 21 January 2012
  17. 17. DOUBLE THE EXECUTION TIMESaturday, 21 January 2012
  18. 18. Saturday, 21 January 2012
  19. 19. all checks executed in 600 seconds monitoring system at 200% capacity only 50% of the checks are “on time”Saturday, 21 January 2012
  20. 20. CHECK LATENCYSaturday, 21 January 2012
  21. 21. Saturday, 21 January 2012
  22. 22. HOW DO WE FIX THIS!?Saturday, 21 January 2012
  23. 23. PERFORMANCE ANALYSIS!Saturday, 21 January 2012
  24. 24. AN ANALOGYSaturday, 21 January 2012
  25. 25. MONITORING CHECK == “ACTION” ON MVC WEB APPSaturday, 21 January 2012
  26. 26. Saturday, 21 January 2012
  27. 27. Saturday, 21 January 2012
  28. 28. SEPARATE DATA COLLECTION FROM THRESHOLDING f NOTIFICATIONSSaturday, 21 January 2012
  29. 29. Saturday, 21 January 2012
  30. 30. THIS SHIFTS LATENCYSaturday, 21 January 2012
  31. 31. IT DOES NOT ELIMINATE IT!Saturday, 21 January 2012
  32. 32. Saturday, 21 January 2012
  33. 33. RRDTOOL IS EVILSaturday, 21 January 2012
  34. 34. USE SOMETHING BETTER!Saturday, 21 January 2012
  35. 35. USE OPENTSDB OR GANGLIA WITH CHECK_TSDB OR CHECK_GMONDSaturday, 21 January 2012
  36. 36. Saturday, 21 January 2012
  37. 37. A DIFFERENT SET a PROBLEMSSaturday, 21 January 2012
  38. 38. STORAGE WILL GO AWAYSaturday, 21 January 2012
  39. 39. CHAOS WILL ENSUESaturday, 21 January 2012
  40. 40. PAGERS WILL MELTSaturday, 21 January 2012
  41. 41. SET UP “META- PARENTING”Saturday, 21 January 2012
  42. 42. BUILD A KILL SWITCHSaturday, 21 January 2012
  43. 43. Saturday, 21 January 2012
  44. 44. READ MORE ABOUT THIS:Saturday, 21 January 2012
  45. 45. bit.ly/yN4mdySaturday, 21 January 2012

×