0
LATENCY                                THE SILENT                            MONITORING SYSTEM                            ...
#MONITORINGSUCKSSaturday, 21 January 2012
SCALABILITYSaturday, 21 January 2012
execute large volumes of monitoring                   checks under a variety of conditions                (good + bad) wit...
CONSISTENT                            THROUGHPUTSaturday, 21 January 2012
SHORT CHECK                            EXECUTION TIMESaturday, 21 January 2012
WHAT                            INTRODUCES                            VARIABILITY?Saturday, 21 January 2012
LATENCY INTRODUCED            c SYNCHRONOUS CALLSSaturday, 21 January 2012
Saturday, 21 January 2012
= 1 monitoring checkSaturday, 21 January 2012
Saturday, 21 January 2012
150 monitoring checks                            each executed every 300 seconds                            each takes 1 s...
all checks executed in 150 seconds                            monitoring system at 50% capacitySaturday, 21 January 2012
DOUBLE THE CHECKSSaturday, 21 January 2012
Saturday, 21 January 2012
all checks executed in 300 seconds                            monitoring system at 100% capacitySaturday, 21 January 2012
DOUBLE THE                            EXECUTION TIMESaturday, 21 January 2012
Saturday, 21 January 2012
all checks executed in 600 seconds                            monitoring system at 200% capacity                          ...
CHECK LATENCYSaturday, 21 January 2012
Saturday, 21 January 2012
HOW DO                            WE FIX THIS!?Saturday, 21 January 2012
PERFORMANCE                              ANALYSIS!Saturday, 21 January 2012
AN ANALOGYSaturday, 21 January 2012
MONITORING CHECK                    ==            “ACTION”        ON   MVC WEB APPSaturday, 21 January 2012
Saturday, 21 January 2012
Saturday, 21 January 2012
SEPARATE DATA                              COLLECTION                                 FROM                       THRESHOLD...
Saturday, 21 January 2012
THIS SHIFTS                             LATENCYSaturday, 21 January 2012
IT DOES NOT                            ELIMINATE IT!Saturday, 21 January 2012
Saturday, 21 January 2012
RRDTOOL IS EVILSaturday, 21 January 2012
USE SOMETHING                               BETTER!Saturday, 21 January 2012
USE                             OPENTSDB                                 OR                              GANGLIA          ...
Saturday, 21 January 2012
A       DIFFERENT SET a                              PROBLEMSSaturday, 21 January 2012
STORAGE WILL                              GO AWAYSaturday, 21 January 2012
CHAOS WILL ENSUESaturday, 21 January 2012
PAGERS WILL MELTSaturday, 21 January 2012
SET UP “META-                             PARENTING”Saturday, 21 January 2012
BUILD A KILL                              SWITCHSaturday, 21 January 2012
Saturday, 21 January 2012
READ MORE                            ABOUT THIS:Saturday, 21 January 2012
bit.ly/yN4mdySaturday, 21 January 2012
Upcoming SlideShare
Loading in...5
×

Latency: The Silent Monitoring System Killer

1,808

Published on

Slides from a lighting talk presented at the January Sydney DevOps Meetup.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,808
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
16
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Latency: The Silent Monitoring System Killer"

  1. 1. LATENCY THE SILENT MONITORING SYSTEM KILLERSaturday, 21 January 2012
  2. 2. #MONITORINGSUCKSSaturday, 21 January 2012
  3. 3. SCALABILITYSaturday, 21 January 2012
  4. 4. execute large volumes of monitoring checks under a variety of conditions (good + bad) with a consistent throughputSaturday, 21 January 2012
  5. 5. CONSISTENT THROUGHPUTSaturday, 21 January 2012
  6. 6. SHORT CHECK EXECUTION TIMESaturday, 21 January 2012
  7. 7. WHAT INTRODUCES VARIABILITY?Saturday, 21 January 2012
  8. 8. LATENCY INTRODUCED c SYNCHRONOUS CALLSSaturday, 21 January 2012
  9. 9. Saturday, 21 January 2012
  10. 10. = 1 monitoring checkSaturday, 21 January 2012
  11. 11. Saturday, 21 January 2012
  12. 12. 150 monitoring checks each executed every 300 seconds each takes 1 second checks are executed seriallySaturday, 21 January 2012
  13. 13. all checks executed in 150 seconds monitoring system at 50% capacitySaturday, 21 January 2012
  14. 14. DOUBLE THE CHECKSSaturday, 21 January 2012
  15. 15. Saturday, 21 January 2012
  16. 16. all checks executed in 300 seconds monitoring system at 100% capacitySaturday, 21 January 2012
  17. 17. DOUBLE THE EXECUTION TIMESaturday, 21 January 2012
  18. 18. Saturday, 21 January 2012
  19. 19. all checks executed in 600 seconds monitoring system at 200% capacity only 50% of the checks are “on time”Saturday, 21 January 2012
  20. 20. CHECK LATENCYSaturday, 21 January 2012
  21. 21. Saturday, 21 January 2012
  22. 22. HOW DO WE FIX THIS!?Saturday, 21 January 2012
  23. 23. PERFORMANCE ANALYSIS!Saturday, 21 January 2012
  24. 24. AN ANALOGYSaturday, 21 January 2012
  25. 25. MONITORING CHECK == “ACTION” ON MVC WEB APPSaturday, 21 January 2012
  26. 26. Saturday, 21 January 2012
  27. 27. Saturday, 21 January 2012
  28. 28. SEPARATE DATA COLLECTION FROM THRESHOLDING f NOTIFICATIONSSaturday, 21 January 2012
  29. 29. Saturday, 21 January 2012
  30. 30. THIS SHIFTS LATENCYSaturday, 21 January 2012
  31. 31. IT DOES NOT ELIMINATE IT!Saturday, 21 January 2012
  32. 32. Saturday, 21 January 2012
  33. 33. RRDTOOL IS EVILSaturday, 21 January 2012
  34. 34. USE SOMETHING BETTER!Saturday, 21 January 2012
  35. 35. USE OPENTSDB OR GANGLIA WITH CHECK_TSDB OR CHECK_GMONDSaturday, 21 January 2012
  36. 36. Saturday, 21 January 2012
  37. 37. A DIFFERENT SET a PROBLEMSSaturday, 21 January 2012
  38. 38. STORAGE WILL GO AWAYSaturday, 21 January 2012
  39. 39. CHAOS WILL ENSUESaturday, 21 January 2012
  40. 40. PAGERS WILL MELTSaturday, 21 January 2012
  41. 41. SET UP “META- PARENTING”Saturday, 21 January 2012
  42. 42. BUILD A KILL SWITCHSaturday, 21 January 2012
  43. 43. Saturday, 21 January 2012
  44. 44. READ MORE ABOUT THIS:Saturday, 21 January 2012
  45. 45. bit.ly/yN4mdySaturday, 21 January 2012
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×