Instrumenting the Twelve-Factor App                                  TM
@josephruscioCo-Founder/CTO LibratoI <3 graphs
Twelve Factor App• Decouple app from environment• Service Oriented Architecture• Continuous Deployment• Horizontal Scaling
?   How do we validate the effects of a deployment?
?   How do we detect aberrations?
?   How do we determine when to scale?
?   How do accomplish this with ephemeral processes?
Instrument during development
Instrumentation is   Unit Testing  for Operations
Counter#enqueue a jobFoo.counter(mygem.jobs).increment#complete a jobFoo.counter(mygem.jobs).decrement
Timer# block formFoo.timer(req.foo).time do  process_requestend# explicit formt = Foo.timer(req.bar).timeprocess_requestt....
Decouple collection from reporting
Observer effect
Math is involved
Logs as streams
Logs• logplex• l2met• logstash, greylog2• Splunk, Papertrail, et al
Process Reporter   Request   Metrics   Thread    Registry             Reporter              Thread Process
Process Reporter• CodaHale Metrics (JVM)• metriks (Ruby)• folsom (Erlang)• librato-rails, librato-rack
UDP    Nginx         Unicorn                  StatsDFront-End
Store metrics in a common repository
Anti-Pattern         Storage               Storage                    Storage                                           .....
Arbitrary correlations
TM
Significant events
TM
TM
Time Series Datastores• RRDTool• Graphite (Whisper)• OpenTSDB (HBase)• Librato et. al (SaaS)
Define per-service operating ranges
Trigger alerts
Apply backpressure
TM
Collaborate through Dashboards
“No aberration detectionmechanism is more sophisticatedthan your VP Marketing staring ata dashboard whilst drinkingcoffee.”
TM
Iterate and Refine
Signal------------   Noise
• Instrument during development• Decouple collection from reporting• Store metrics in a common repository• Define per-servi...
fin
Librato's Joseph Ruscio at Heroku's 2013: Instrumenting 12-Factor Apps
Librato's Joseph Ruscio at Heroku's 2013: Instrumenting 12-Factor Apps
Upcoming SlideShare
Loading in …5
×

Librato's Joseph Ruscio at Heroku's 2013: Instrumenting 12-Factor Apps

2,669 views

Published on

Librato's CTO Joseph Ruscio took to the Waza 2013 stage to present "Instrumenting Twelve-Factor Apps". For more from Ruscio ping him at @josephruscio. For more on Waza visit http://waza.heroku.com/2013.

For Waza videos stay tuned at http://blog.heroku.com or visit http://vimeo.com/herokuwaza

Published in: Technology
  • Be the first to comment

Librato's Joseph Ruscio at Heroku's 2013: Instrumenting 12-Factor Apps

  1. 1. Instrumenting the Twelve-Factor App TM
  2. 2. @josephruscioCo-Founder/CTO LibratoI <3 graphs
  3. 3. Twelve Factor App• Decouple app from environment• Service Oriented Architecture• Continuous Deployment• Horizontal Scaling
  4. 4. ? How do we validate the effects of a deployment?
  5. 5. ? How do we detect aberrations?
  6. 6. ? How do we determine when to scale?
  7. 7. ? How do accomplish this with ephemeral processes?
  8. 8. Instrument during development
  9. 9. Instrumentation is Unit Testing for Operations
  10. 10. Counter#enqueue a jobFoo.counter(mygem.jobs).increment#complete a jobFoo.counter(mygem.jobs).decrement
  11. 11. Timer# block formFoo.timer(req.foo).time do process_requestend# explicit formt = Foo.timer(req.bar).timeprocess_requestt.stop
  12. 12. Decouple collection from reporting
  13. 13. Observer effect
  14. 14. Math is involved
  15. 15. Logs as streams
  16. 16. Logs• logplex• l2met• logstash, greylog2• Splunk, Papertrail, et al
  17. 17. Process Reporter Request Metrics Thread Registry Reporter Thread Process
  18. 18. Process Reporter• CodaHale Metrics (JVM)• metriks (Ruby)• folsom (Erlang)• librato-rails, librato-rack
  19. 19. UDP Nginx Unicorn StatsDFront-End
  20. 20. Store metrics in a common repository
  21. 21. Anti-Pattern Storage Storage Storage ...• Ping • CPU• CPU • Interface • Custom Stats• Memory • Memory • MySQL threads• Disks • Ping • VMstat• SNMP Service • Battery charge • ....• .... • .... Project X Project Y Project Z
  22. 22. Arbitrary correlations
  23. 23. TM
  24. 24. Significant events
  25. 25. TM
  26. 26. TM
  27. 27. Time Series Datastores• RRDTool• Graphite (Whisper)• OpenTSDB (HBase)• Librato et. al (SaaS)
  28. 28. Define per-service operating ranges
  29. 29. Trigger alerts
  30. 30. Apply backpressure
  31. 31. TM
  32. 32. Collaborate through Dashboards
  33. 33. “No aberration detectionmechanism is more sophisticatedthan your VP Marketing staring ata dashboard whilst drinkingcoffee.”
  34. 34. TM
  35. 35. Iterate and Refine
  36. 36. Signal------------ Noise
  37. 37. • Instrument during development• Decouple collection from reporting• Store metrics in a common repository• Define per-service thresholds• Collaborate through dashboards• Iterate and refine
  38. 38. fin

×