Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Everything obfuscurity taught me about monitoring

882 views

Published on

Monitorama 2016

Published in: Technology
  • Be the first to comment

Everything obfuscurity taught me about monitoring

  1. 1. @petecheslock Everything @obfuscurity Taught Me About Monitoring Monitorama 2016
  2. 2. @petecheslock “Just use Graphite” - Jason Dixon
  3. 3. @petecheslock Thank you
  4. 4. @petecheslock Pete Cheslock @petecheslock Tech Ops & Support at Threat Stack
  5. 5. @petecheslock get.threatstack.com/monitorama16
  6. 6. @petecheslock
  7. 7. constantly
  8. 8. @petecheslock It is an AMAZING time to be working in technology
  9. 9. @petecheslock The “Good ‘ole Days” kinda sucked
  10. 10. @petecheslock
  11. 11. @petecheslock
  12. 12. @petecheslock
  13. 13. @petecheslock
  14. 14. @petecheslock
  15. 15. @petecheslock
  16. 16. @petecheslock I knew how to cd and ls
  17. 17. @petecheslock # chown -Rf bob:bob files / Whoops
  18. 18. @petecheslock Should've used bcfg2? <Path name="/home/file" mode="0644" group="bob" owner="bob"> <ACL type="access" scope="user" user="bob" perms="rwx"/> </Path> Won’t exist for 3+ more years…
  19. 19. @petecheslock
  20. 20. @petecheslock ColdFusion
  21. 21. @petecheslock * * * * * /scripts/mrtg-netmon
  22. 22. @petecheslock
  23. 23. @petecheslock
  24. 24. @petecheslock
  25. 25. @petecheslock **PROBLEM Host Alert: Everything Is FUCKED**
  26. 26. @petecheslock
  27. 27. @petecheslock
  28. 28. @petecheslock Jen: Yo, are you ready to support ~10 beta customers on a completely unautomated, unmonitored env? Pete: New Phone, who dis? Jen: 5 9s of availability.
  29. 29. @petecheslock The Right Way™
  30. 30. @petecheslock
  31. 31. @petecheslock
  32. 32. @petecheslock
  33. 33. @petecheslock
  34. 34. @petecheslock http://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/
  35. 35. @petecheslock
  36. 36. @petecheslock Premature optimization is the root of all evil
  37. 37. @petecheslock
  38. 38. @petecheslock
  39. 39. @petecheslock
  40. 40. @petecheslock collectd -> write_http -> (60 Second)
  41. 41. @petecheslock
  42. 42. @petecheslock
  43. 43. @petecheslock
  44. 44. @petecheslock
  45. 45. @petecheslock collectd -> write_http -> (60 Second) statsd plugin app1 app2 appN
  46. 46. @petecheslock
  47. 47. @petecheslock • Hosted TSDB is super useful and “just works” • The opportunity cost of fully replacing it was too high.
  48. 48. @petecheslock • Use the same tools already deployed • Devs wrote libraries to send metrics to statsd • Don’t want to “redeploy the world” • Limited time to invest
  49. 49. @petecheslock • Use the same tools already deployed • Devs wrote libraries to send metrics to statsd • Don’t want to “redeploy the world” • Limited time to invest
  50. 50. @petecheslock collectd -> write_http -> (60 Second) collectd -> write_graphite -> (10 Second) statsd plugin statsd plugin
  51. 51. @petecheslock
  52. 52. @petecheslock https://github.com/obfuscurity/ synthesize https://github.com/hw-cookbooks/ graphite
  53. 53. @petecheslock https://github.com/obfuscurity/ synthesize https://github.com/hw-cookbooks/ graphite
  54. 54. @petecheslock
  55. 55. @petecheslock Want your Devs to Ops? Build Consumable Services
  56. 56. @petecheslock
  57. 57. @petecheslock
  58. 58. @petecheslock
  59. 59. @petecheslock collectd -> write_graphite -> (10 Second) statsd_plugin app1 app2 appN
  60. 60. @petecheslock
  61. 61. A lot of people must have suddenly just stopped caring
  62. 62. @petecheslock Community Matters Relationships Matter
  63. 63. @petecheslock You are (probably) not Google
  64. 64. @petecheslock and that is OK (Solve YOUR Problems)
  65. 65. get.threatstack.com/monitorama16 Thank You!

×