I am going M.A.D. M onitoring A ided D evelopment From system administration to development and back
We all code
Monitoring and metrics: a lifetime love Love for monitoring career
One day in data warehousing... “ There is no such thing as too much data, only data you don't know how to make sense of”
do mind information overflow
You, yes you, do you test your code?
Tests and success rate TDD
What about coveragE ?
Tests and COverage
Lines of code? Thanks, but no thanks
LOC, Tests and COverage refactoring
Complexity is your enemy <ul><li>Callgraph and NESTING
Number and size of functions
Code CLOSURe
Complexity of the build system </li></ul>
Complexity, Tests, COverage
Do it with  style <ul><li>Lintian
Pep8
Reward beautiful code  </li></ul>
Take a step back, a large breath and  reflect
hello HUDSON  Jenkins Do not do what a machine can do.
Being on-call sucks big TIME
HOW > IF “ It is never too early to start monitoring your application's behaviour”
Write code that is monitoring friendly
OPS is changing <ul><li>Configuration management
Upcoming SlideShare
Loading in …5
×

I am going M.A.D - Monitoring Aided Development

1,172 views

Published on

I've gone M.A.D - monitoring aided development: the backstory of a
sysadmin going all the way to development and back. Monitoring and
metrics are the bread and butter of any admin and a must have for any
production infrastructure. I will present how certain development
techniques are already mirrored in operations without neither groups
being aware of it and how developers can take advantage of monitoring
systems to feedback into the development process. Furthermore I will
highlight how a common language developed around testing can help the
two groups communicating and increase the level of trust, which is
fundamental for any functional organization and the true nature of
devops.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,172
On SlideShare
0
From Embeds
0
Number of Embeds
289
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Wanna share my experience What I learned going dev and what I brought back SHOW OF HANDS, how many admins, how many devs? How many that do both? Do we have QA?
  • * Start by saying strongly believe into: everybody codes * As a sysadmin start lots oneliners * Not enough bash scripts * More complex, perl python ruby scripts * Still doesn&apos;t feel like coding * You&apos;re looked upon as a non dev * But was “hello world” coding? Why? * Yes maybe not a great dev, but still * This is bad, creates wrong culture * You can test bash scripts * Also another important stepping stone: infrastructure as code. * Puppet manifests * what about QA? Test scripts and automation, isn&apos;t that code? * So you see, we all code
  • * Loved monitoring since day one * Simple idea of knowing nagios was checking stuff for me made me sleep better * Then discovered metrics, world of wonders. * A complete change of perspective, not only you know if something is broken, but you know how it&apos;s doing * How am I supposed to answer any question on the state of my systems? * think of it as if we you were a doctor. * western doctors Vs chinese doctors * Anecdotal chinese doctor , you didn&apos;t pay when you were sick . No longer true unfortunately
  • * I was on loan to dw for a few months * that&apos;s where my passion for data and viz was born * met a fellow data monger that was amazing, he could dig out of piles of data the most amazing things and he was usually right. * the all process was like magic, like looking at the matrix encrypted and see&apos;ing people * the guy could look at a pile of data and see patterns, questions and answers – next * before sounding completely nuts about metrics
  • * can be mislead * miss things in the noise * requires more effort to make sense of things * there are no free lunches. But the benefit can be huge
  • * didnt use test, too hard, no understand, felt no dev * worked in the right place, was exposed to tests, it INSPIRED ME * still no tests cuz team was SE, me complacent * When startup making a product, I&apos;ve got no excuse, especially being alone with no QA. * Bugs will make you lose customers * started with unittesting, then tdd * realized was doing TDD with nagios, check deployed before srv and then pass service was live * Testing upfront was awesome, confidence went up * Use TDD to do security * New metrics! Add tests, success rate, monitor my progress
  • * says something at a glance * was sloppy testing * until I wanted to release * and I diverged again afterwards * so I started to do tdd I didn&apos;t diverge anymore * would have I been able to notice without this graph? * the graph empowered me to spot a behaviour * what&apos;s the benefit? You know, you can prove the effectiveness, – next * but then you wonder, how much code am I testing?
  • * Adding tests alone is not meaningful * You wonder about them * In Nagios you did the same, you test a port, but what does that cover? * More and more people do end-to-end checks
  • * again, you can notice something at a glance * adding tests does not mean adding coverage * also see for 0.2 I hit 100% but I&apos;ve only added a few tests? * let&apos;s see next slide
  • * you gotta start somewhere * that somewhere is usually wrong :) * nonetheless, I found &apos;em useful for something == Next slide * remember from the previous slide that I hit 100% without adding many tests?
  • * once again metrics tell us something we&apos;d otherwise missed * tell us why we achieved 100% == Next * but loc is a poor metric after all * more or less lines of code aren&apos;t that important * what matters is complexity!
  • * complexity is your enemy * KISS * calculating complexity is a huge thing, you could make a talk on it * no secret formula, I just got the idea from spamming systems to use scoring * so what are interesting metrics about complexity? * ask audience, can anybody think of another? http://michaelfeathers.typepad.com/michael_feathers_blog/2011/01/measuring-the-closure-of-code.html
  • * once again metrics tell us a story * stories are far more powerful than numbers
  • * style matters * good code is more readable * easier to understand and refactor * less prone to contain bugs * it makes you feel better
  • * discussed lots of metrics * wanted to make a point they should help you, not hinder you * people say “create a metric and people will game it” * lots of books on psychology make a scary point about rewards * by any mean toss them if they hurt you * remind yourself about the peril * with great powers, come great responsibility. So it does with metrics.
  • * quite simple really * moved it into a CI * buildbot would work too – Next slide * all excited by these metrics and developing that I forgot where I came from
  • * that&apos;s where I come from * how many people here have actually been on-call before? * Hard to appreciate if you&apos;ve never been on-call in an ops team. * Had been a sysadmin for a while, but not oncall and it was a shock * After I left every time I was in a public places and someone had the same ringtone I twitched * SO WHAT CAN YOU DO?
  • * how long do your unit tests take? * has this changed? * how much cpu or mem during the last global pass? * do you build your sw? How long does that take? Has that changed? * Do you run integration tests? That&apos;s like live, monitor those systems! * cheap way to profile your app * Very useful and possible to spot mem leak or cpu increase
  • If it&apos;s never too early to start monitoring your app, never too early yo make your app easy to monitor We&apos;re making good progress, think memcached mgmt interface, mysql session variables MAKE COLLECTION EASY SOME EXAMPLES: HTTP REQUEST TIMING SIZE OF REQUEST DB QUERY TIME NUMBER OF CONNECTED CLIENTS TOP REQUESTS
  • * CM been a long time resident (cfengine) * but now many more people do it (still lots to do tho) * infra-as-code is a result of CM. Name says it all * Consulting for a company now that uses jenkins and we&apos;re building one in ops to do CI for scripts and puppet manifests * as we move toward this direction need more access to devs to be successful
  • * use a vcs * branch * test
  • * anybody wants to guess? * trust in your code ** that&apos;s why you test, test your code works, trust you can change it * trust in the people ** stealing from patric debois ** trust tax * at the end of the day you don&apos;t want to test for fun * test for profit, to make a good job * you can&apos;t make a good job without trusting yourself and the people that work with you * especially true across departments (Dev/ops) * I BELIEVE THAT TESTING CAN BE THE PLATFORM TO BUILD TRUST UPON AND FOSTER THE CULTURAL CHANGE THAT DEVOPS IS ALL ABOUT
  • I am going M.A.D - Monitoring Aided Development

    1. 1. I am going M.A.D. M onitoring A ided D evelopment From system administration to development and back
    2. 2. We all code
    3. 3. Monitoring and metrics: a lifetime love Love for monitoring career
    4. 4. One day in data warehousing... “ There is no such thing as too much data, only data you don't know how to make sense of”
    5. 5. do mind information overflow
    6. 6. You, yes you, do you test your code?
    7. 7. Tests and success rate TDD
    8. 8. What about coveragE ?
    9. 9. Tests and COverage
    10. 10. Lines of code? Thanks, but no thanks
    11. 11. LOC, Tests and COverage refactoring
    12. 12. Complexity is your enemy <ul><li>Callgraph and NESTING
    13. 13. Number and size of functions
    14. 14. Code CLOSURe
    15. 15. Complexity of the build system </li></ul>
    16. 16. Complexity, Tests, COverage
    17. 17. Do it with style <ul><li>Lintian
    18. 18. Pep8
    19. 19. Reward beautiful code </li></ul>
    20. 20. Take a step back, a large breath and reflect
    21. 21. hello HUDSON Jenkins Do not do what a machine can do.
    22. 22. Being on-call sucks big TIME
    23. 23. HOW > IF “ It is never too early to start monitoring your application's behaviour”
    24. 24. Write code that is monitoring friendly
    25. 25. OPS is changing <ul><li>Configuration management
    26. 26. Infrastructure-as-c0de
    27. 27. Behaviour driven development </li><ul><li>Cucumber
    28. 28. Robotframework </li></ul><li>Continuous integration for OPS </li></ul>
    29. 29. ...and you can help <ul><li>If you are an op </li><ul><li>Realize and accept that you code
    30. 30. There are lots of good reusable development patterns
    31. 31. Advertise your achievements
    32. 32. Engage your developers </li></ul></ul>
    33. 33. ...and you can help <ul><li>If you are a dev </li><ul><li>Treat ops as developer, don't exclude them
    34. 34. Share the knowledge
    35. 35. Code applications that are easier to monitor
    36. 36. Learn from your ops how to make your app production ready </li></ul></ul>
    37. 37. The most important metric TRUST
    38. 38. Don't let uncertainty drive you insane , go M.A.D.
    39. 39. Thanks everybody <ul><li>Http://www.spikelab.org
    40. 40. [email_address]
    41. 41. @spikelab </li></ul>

    ×