Latency: The Silent Monitoring System Killer

  • 1,714 views
Uploaded on

Slides from a lighting talk presented at the January Sydney DevOps Meetup.

Slides from a lighting talk presented at the January Sydney DevOps Meetup.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,714
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
14
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. LATENCY THE SILENT MONITORING SYSTEM KILLERSaturday, 21 January 2012
  • 2. #MONITORINGSUCKSSaturday, 21 January 2012
  • 3. SCALABILITYSaturday, 21 January 2012
  • 4. execute large volumes of monitoring checks under a variety of conditions (good + bad) with a consistent throughputSaturday, 21 January 2012
  • 5. CONSISTENT THROUGHPUTSaturday, 21 January 2012
  • 6. SHORT CHECK EXECUTION TIMESaturday, 21 January 2012
  • 7. WHAT INTRODUCES VARIABILITY?Saturday, 21 January 2012
  • 8. LATENCY INTRODUCED c SYNCHRONOUS CALLSSaturday, 21 January 2012
  • 9. Saturday, 21 January 2012
  • 10. = 1 monitoring checkSaturday, 21 January 2012
  • 11. Saturday, 21 January 2012
  • 12. 150 monitoring checks each executed every 300 seconds each takes 1 second checks are executed seriallySaturday, 21 January 2012
  • 13. all checks executed in 150 seconds monitoring system at 50% capacitySaturday, 21 January 2012
  • 14. DOUBLE THE CHECKSSaturday, 21 January 2012
  • 15. Saturday, 21 January 2012
  • 16. all checks executed in 300 seconds monitoring system at 100% capacitySaturday, 21 January 2012
  • 17. DOUBLE THE EXECUTION TIMESaturday, 21 January 2012
  • 18. Saturday, 21 January 2012
  • 19. all checks executed in 600 seconds monitoring system at 200% capacity only 50% of the checks are “on time”Saturday, 21 January 2012
  • 20. CHECK LATENCYSaturday, 21 January 2012
  • 21. Saturday, 21 January 2012
  • 22. HOW DO WE FIX THIS!?Saturday, 21 January 2012
  • 23. PERFORMANCE ANALYSIS!Saturday, 21 January 2012
  • 24. AN ANALOGYSaturday, 21 January 2012
  • 25. MONITORING CHECK == “ACTION” ON MVC WEB APPSaturday, 21 January 2012
  • 26. Saturday, 21 January 2012
  • 27. Saturday, 21 January 2012
  • 28. SEPARATE DATA COLLECTION FROM THRESHOLDING f NOTIFICATIONSSaturday, 21 January 2012
  • 29. Saturday, 21 January 2012
  • 30. THIS SHIFTS LATENCYSaturday, 21 January 2012
  • 31. IT DOES NOT ELIMINATE IT!Saturday, 21 January 2012
  • 32. Saturday, 21 January 2012
  • 33. RRDTOOL IS EVILSaturday, 21 January 2012
  • 34. USE SOMETHING BETTER!Saturday, 21 January 2012
  • 35. USE OPENTSDB OR GANGLIA WITH CHECK_TSDB OR CHECK_GMONDSaturday, 21 January 2012
  • 36. Saturday, 21 January 2012
  • 37. A DIFFERENT SET a PROBLEMSSaturday, 21 January 2012
  • 38. STORAGE WILL GO AWAYSaturday, 21 January 2012
  • 39. CHAOS WILL ENSUESaturday, 21 January 2012
  • 40. PAGERS WILL MELTSaturday, 21 January 2012
  • 41. SET UP “META- PARENTING”Saturday, 21 January 2012
  • 42. BUILD A KILL SWITCHSaturday, 21 January 2012
  • 43. Saturday, 21 January 2012
  • 44. READ MORE ABOUT THIS:Saturday, 21 January 2012
  • 45. bit.ly/yN4mdySaturday, 21 January 2012