Your SlideShare is downloading. ×
Feature: Ruby on Rails Application Monitoring with Cucumber

 In order to ensure continuous application availability
 A de...
VP of Research & Development
                            railsmachine.com

                           jesse@railsmachine.c...
Before we get into monitoring or cucumber, let’s talk about testing.

In my career as a dev, my testing habits have evolve...
No
more
clicking
around
Save in your editor / refresh in your browser / lather / rinse repeat.

Occasional human preformed...
I then made the jump to unit testing using Ruby’s Test::Unit - specifically the generated
Model and Controller tests Rails ...
R

Enter Rspec and the BDD movement.

Rspec helped me, and I’m sure a lot of others, associate the business value with wri...
C
  U
  C
  U
  M
  B
  E
  R
Basically, BDD nirvana. Stakeholder-*writable* if you’re crazy.
Cucumber lets software
development teams describe
how software should behave in
plain text. The text is written in
a busin...
TATFT!
The most important part of the evolution of these tools is that they make it easy and -
legitimately - fun to test ...
Production
 Monitoring
But what about production? We’re testing all the time in development, while we’re developing
the th...
Revenue
                                          Preservation




...there’s actually revenue being earned. Why not test ...
Current
                                                  Monitoring
                                                  Lan...
What’s being monitored - what URLs, metrics, system statistics, etc are being watched
How
closely
are
you
looking
at
it?
We’ll call this one the crazy monkey test - How frequently these URLs / metrics are bei...
It seems that in many situations, the home page of an application is the only thing checked
closely
The crazy monkey has laser focus
but if the crazy monkey is that focused
Bad things can happen when he’s not looking.

For example, in Rails apps, I see this happen all the time with...
Search is a part of many applications that I’ve seen go unmonitored. I’m not singling out
sphinx here - this is just a swe...
Search can fail when the rest of a site works fine due to many reasons:

* search daemon may go down
* the indicies may be ...
0 results for “beer”




Wherein no results are returned when they obviously should be.
TATFT?
So are we really testing all the time?
TATFT*
                     *except in production
It doesn’t seem so
_why?




But why? Why are we testing so....ferociously in development, but so weakly in production?
Old
                                                                  Broken
                                             ...
How many of your recognize this? Oh, nagios.
It’s the industry standard tool for infra monitoring. I haven’t met a single person that’s used
nagios that’s been an hone...
is the noise. Unless masterly configured, Nagios is a noisy beast. This leads to “boy cries
wolf” type scenarios, wherein a...
EVIL



Because of the noise, and the piece of crap interface, esoteric configuration language, and for
years and years of ...
Pingdom’s a relatively new tool that’s gained a good bit of traction. It’s a hosted monitoring
service, that can test HTTP...
✔
Nagios and pingdom pass the crazy monkey intense focus test
✘

but in their default configuration generally only monitor a snapshot of what’s neccessary.
A recent entry into the space that’s doesn’t get a quick EVIL stamp from me is watchmouse
Twitter uses watchmouse to provide a public API status page, hitting many different API
endpoints and watching for outages...
✔
Twitter’s use of Watchmouse passes the “what are you looking at test”
Business
Value
Disconnect




However, one thing that all of these tools are missing is a clear link between the business
...
Hey, I know something that does that well!
Cucumber lets software
development teams describe
how software should behave in
plain text. The text is written in
a busin...
Cucumber lets software
development teams describe
how software should behave in
plain text. The text is written in
a busin...
Cucumber also lets operations
teams describe how
infrastructure should behave in
plain text. The text is written in a
busi...
#devops




Some of you following the twitterz may have noticed some people in the ops and
development space talking about...
devs

                    ops



                        working together
While calling this a movement is pretty wild - a...
Cucumber also lets #devops
teams describe how
applications should behave in
plain text. The text is written in a
business-...
kumbaya




And thus result in a big happy #devops family
Example
Production
Cucumber
 Features
Benchmarking
Feature: slashdot.com
  To keep the geek masses satisfied
  Slashdot must be responsive

  Scenario: Cached pages are supe...
Email Deliverability
Feature: Signup Emails
  In order to prevent bots from taking over the site
  A new user should receive a verification ema...
Existing
Metrics
Feature: Response Time
  As a impatient user
  Our web server should be in tip-top shape
  So our app can be super fast

 ...
Feature: Response Time
  As a impatient user
  Our app should be super fast

  Background:
    Given my NewRelic license k...
SEO
Feature: Cucumber wiki discoverability
  In order to learn more about Cucumber
  As an uninformed developer
  I should be ...
Security
Feature: example.org ssh logins
  As a user of example.org
  I need to login remotely

  Scenario: Login with a key
    Gi...
Infrastructure
Feature: RAID
  To ensure optimal server operation
  And guarantee data is stored redundantly
  The RAID array should be i...
DNS
Feature: rubygems.org
  As a member of the Ruby community
  I should be able to easily install Ruby gems

 Scenario: DNS
 ...
Possibilities
Credit
Card
Transactions
SSL
Exception
    Rate
Running
     features
in production
Sorta Quick Setup                                     Ge
                                                               ne...
Cucumber Scout Plugin
Cucumber Scout Plugin




http://github.com/jnewland/scout-plugins/raw/cucumber_ci/
                cucumber_ci/cucumber_c...
Cucumber Scout Plugin
Cucumber Scout Plugin
                ZOMG!
Cucumber Scout Plugin




Sent via email
Power
to
monitor
anything
and
everything
Clearly
Defined
Business
Value
I
N

R
U
B
Y
24/7/365
Know before your customers do
Thanks!
Any questions?
Get in touch:

     Jesse Newland
jesse@railsmachine.com
       @jnewland
  github.com/jnewland
Flickr Creative Commons Photos
http://flic.kr/p/4EjsDJ   http://flic.kr/p/nxAqt
http://flic.kr/p/c1UTf    http://flic.kr/p/4VW...
Continuous (Production) Integration: Ruby on Rails Application Monitoring with Cucumber
Continuous (Production) Integration: Ruby on Rails Application Monitoring with Cucumber
Upcoming SlideShare
Loading in...5
×

Continuous (Production) Integration: Ruby on Rails Application Monitoring with Cucumber

10,969

Published on

Feature: Ruby on Rails Application Monitoring with Cucumber

 In order to ensure continuous application availability

A developer should be able to assert the behavior of production apps

From the outside in

Without using antiquated monitoring tools

 To protect revenue

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,969
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
156
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Transcript of "Continuous (Production) Integration: Ruby on Rails Application Monitoring with Cucumber"

  1. 1. Feature: Ruby on Rails Application Monitoring with Cucumber In order to ensure continuous application availability A developer should be able to assert the behavior of production apps From the outside in Without using antiquated monitoring tools To protect revenue
  2. 2. VP of Research & Development railsmachine.com jesse@railsmachine.com @jnewland github.com/jnewland About me: I get to hack on Ruby tools to manage large Rails deployments all day long. Not a bad job, eh?
  3. 3. Before we get into monitoring or cucumber, let’s talk about testing. In my career as a dev, my testing habits have evolved over time, largely inspired by available tools. I’m sure some of you have shared a similar journey - let’s take a quick look back.
  4. 4. No more clicking around Save in your editor / refresh in your browser / lather / rinse repeat. Occasional human preformed quality assurance Broken by design
  5. 5. I then made the jump to unit testing using Ruby’s Test::Unit - specifically the generated Model and Controller tests Rails generated. This was nice, but it was often devalued by stakeholders due to poor communication of the business value of this work on my part.
  6. 6. R Enter Rspec and the BDD movement. Rspec helped me, and I’m sure a lot of others, associate the business value with writing tests / specs. Stakeholder-digestable code if you’re really good, stakeholder-digestable output if you’re doing things right.
  7. 7. C U C U M B E R Basically, BDD nirvana. Stakeholder-*writable* if you’re crazy.
  8. 8. Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain- specific language and serves as documentation, automated tests and development-aid - all rolled into one format. For those of you that aren’t familiar with Cuke
  9. 9. TATFT! The most important part of the evolution of these tools is that they make it easy and - legitimately - fun to test first and test all of the time as you’re developing your application.
  10. 10. Production Monitoring But what about production? We’re testing all the time in development, while we’re developing the that’s going to create revenue. But in production...
  11. 11. Revenue Preservation ...there’s actually revenue being earned. Why not test with the same veracity in production?
  12. 12. Current Monitoring Landscape Quiz: * Raise your hand if you are at least partially responsible for the continuous operation of a business critical production rails app * If you have ZERO monitoring of the site’s uptime - meaning your customers or boss would be the one to tell you that the homepage was down - put your hand down * If your monitoring solution runs on your server itself - monit or god, for example - put your hand down * If your external monitoring solution only hits one URL on the site, put your hand down Some sites are monitored very closely, but I’ve found that in most cases, the monitoring of many production apps is rather slim. I generally evaluate monitoring solutions on two axes:
  13. 13. What’s being monitored - what URLs, metrics, system statistics, etc are being watched
  14. 14. How closely are you looking at it? We’ll call this one the crazy monkey test - How frequently these URLs / metrics are being queried, what values are acceptable, etc.
  15. 15. It seems that in many situations, the home page of an application is the only thing checked closely
  16. 16. The crazy monkey has laser focus
  17. 17. but if the crazy monkey is that focused
  18. 18. Bad things can happen when he’s not looking. For example, in Rails apps, I see this happen all the time with...
  19. 19. Search is a part of many applications that I’ve seen go unmonitored. I’m not singling out sphinx here - this is just a sweet picture - the same thing happens to Solr, etc
  20. 20. Search can fail when the rest of a site works fine due to many reasons: * search daemon may go down * the indicies may be corrupt * or things may fail in a more interesting kind of way...
  21. 21. 0 results for “beer” Wherein no results are returned when they obviously should be.
  22. 22. TATFT? So are we really testing all the time?
  23. 23. TATFT* *except in production It doesn’t seem so
  24. 24. _why? But why? Why are we testing so....ferociously in development, but so weakly in production?
  25. 25. Old Broken Tools I’m largely convinced it’s because the tools that are presented to us for use in the monitoring space are largely old and broken.
  26. 26. How many of your recognize this? Oh, nagios.
  27. 27. It’s the industry standard tool for infra monitoring. I haven’t met a single person that’s used nagios that’s been an honest fan. The most widely despised part of nagios
  28. 28. is the noise. Unless masterly configured, Nagios is a noisy beast. This leads to “boy cries wolf” type scenarios, wherein alerts are improperly categorized as noise and discarded.
  29. 29. EVIL Because of the noise, and the piece of crap interface, esoteric configuration language, and for years and years of waking me up for false positives, I’m going to paint this all in black and white and just call nagios evil.
  30. 30. Pingdom’s a relatively new tool that’s gained a good bit of traction. It’s a hosted monitoring service, that can test HTTP and many other types of services from a network of computers around the world.
  31. 31. ✔ Nagios and pingdom pass the crazy monkey intense focus test
  32. 32. ✘ but in their default configuration generally only monitor a snapshot of what’s neccessary.
  33. 33. A recent entry into the space that’s doesn’t get a quick EVIL stamp from me is watchmouse
  34. 34. Twitter uses watchmouse to provide a public API status page, hitting many different API endpoints and watching for outages and service problems
  35. 35. ✔ Twitter’s use of Watchmouse passes the “what are you looking at test”
  36. 36. Business Value Disconnect However, one thing that all of these tools are missing is a clear link between the business value of the things they’re checking and the alerts they’re sending out
  37. 37. Hey, I know something that does that well!
  38. 38. Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain- specific language and serves as documentation, automated tests and development-aid - all rolled into one format. Cucumber’s served well for me in my experience in bringing stakeholders and developers together.
  39. 39. Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain- specific language and serves as documentation, automated tests and development-aid - all rolled into one format. But with a couple quick edits
  40. 40. Cucumber also lets operations teams describe how infrastructure should behave in plain text. The text is written in a business-readable domain- specific language and serves as documentation, monitoring and deployment-aid - all rolled into one format. We have a tool that can help us bring together developers, operations, *and* stakeholders
  41. 41. #devops Some of you following the twitterz may have noticed some people in the ops and development space talking about the ‘devops movement’
  42. 42. devs ops working together While calling this a movement is pretty wild - a hashtag does not a movement make - the ideas surrounding this ‘movement’ are things that I believe in personally, and things we’re working on everyday at Rails Machine - blurring the line between development and ops, and the line between the infrastructure and the application.
  43. 43. Cucumber also lets #devops teams describe how applications should behave in plain text. The text is written in a business-readable domain- specific language and serves as documentation, monitoring and deployment-aid - all rolled into one format. Using cucumber in production embodies everything that is devops, and can blur those lines even more
  44. 44. kumbaya And thus result in a big happy #devops family
  45. 45. Example Production Cucumber Features
  46. 46. Benchmarking
  47. 47. Feature: slashdot.com To keep the geek masses satisfied Slashdot must be responsive Scenario: Cached pages are super quick Given I am benchmarking When I go to http://slashdot.org/ Then the elapsed time should be less than 500 milliseconds When I follow "Login" Then the elapsed time should be less than 500 milliseconds When I follow "Contact" Then the elapsed time should be less than 500 milliseconds
  48. 48. Email Deliverability
  49. 49. Feature: Signup Emails In order to prevent bots from taking over the site A new user should receive a verification email upon signup Scenario: New User signup Given I visit "http://example.com" And I follow "Signup!" When I signup with a random email address and password And I press "Go" And I wait 10 seconds # an unfortunate reality Then I should have one email in my inbox And the email subject should match "^Welcome" And the email body should match "http://example.com/v/w+" https://github.com/technicalpickles/mailinator-spec
  50. 50. Existing Metrics
  51. 51. Feature: Response Time As a impatient user Our web server should be in tip-top shape So our app can be super fast Background: Given my Scout account name is 'railsmachine' And my Scout email and password are 'jesse@railsmachine.com' and 'sekret' Scenario: Passenger Queue When I get the metrics from the 'Passenger' plugin on 'example.com' Then the 'passenger_queue_depth' should be 0 Scenatiro: CPU usage is low When I get the metrics from the 'Server Overview' plugin on 'example.com' Then 'cpu_last_minute' should be less than 1 http://github.com/jnewland/cucumber-scout/
  52. 52. Feature: Response Time As a impatient user Our app should be super fast Background: Given my NewRelic license key is 'omgwtfbbq' Scenario: Average Response time Given that my application is being monitored by New Relic Then my application's 'response time' should be less than 500 milliseconds Scenario: Apdex Given that my application is being monitored by New Relic Then my application's 'apdex' should be 1 http://github.com/jnewland/cucumber-newrelic
  53. 53. SEO
  54. 54. Feature: Cucumber wiki discoverability In order to learn more about Cucumber As an uninformed developer I should be able easily find the GitHub wiki Scenario: Searching for Cucumber on Google When I go to http://www.google.com/ And I fill in "q" with "cucumber" And I press "Google Search" Then I should see "BDD that talks to domain experts first and code second"
  55. 55. Security
  56. 56. Feature: example.org ssh logins As a user of example.org I need to login remotely Scenario: Login with a key Given I have the following public keys: | keyfile | | /home/jnewland/.ssh/id_dsa | Then I can ssh to the following hosts with these credentials: | hostname | username | | example.org | jnewland | | mail.example.org | jnewland | Scenario: Checking /etc/passwd When I ssh to "example.org" with the following credentials: | username | password | keyfile | | jnewland | | /home/jnewland/.ssh/id_dsa | And I run "cat /etc/passwd" Then I should see "jnewland" in the output And I should not see "that_dude_we_just_fired" in the output http://github.com/auxesis/cucumber-nagios
  57. 57. Infrastructure
  58. 58. Feature: RAID To ensure optimal server operation And guarantee data is stored redundantly The RAID array should be in a good state Scenario: RAID Array status When I check the raid array status Then controller "1" should have a status of "optimal" And controller "2" should have a status of "optimal" And controller "1" should have "1" logical device with a status of "optimal" And controller "1" should have "4" drives in "online" state And controller "2" should have "1" logical device with a status of "optimal" And controller "2" should have "4" drives in "online" state http://github.com/auxesis/cucumber-nagios
  59. 59. DNS
  60. 60. Feature: rubygems.org As a member of the Ruby community I should be able to easily install Ruby gems Scenario: DNS When I lookup "rubygems.org" Then the name should resolve an IP http://github.com/auxesis/cucumber-nagios
  61. 61. Possibilities
  62. 62. Credit Card Transactions
  63. 63. SSL
  64. 64. Exception Rate
  65. 65. Running features in production
  66. 66. Sorta Quick Setup Ge ne com rato $ gem install cucumber-json cucumber-newrelic soo ing r cucumber-scout cucumber-nagios n! $ cd RAILS_ROOT $ mkdir -p production_features/step_definitions $ mkdir -p production_features/support $ vi config/cucumber.yml production: production_features -f Cucumber::Formatter::JSON --out tmp/cuke.json $ vi production_features/support/env.rb require 'cucumber/nagios/steps' require 'cucumber/newrelic' require 'cucumber/scout' # etc $ # hack on features $ cucumber -p production # doesn’t load the Rails env, just the defined steps $ # profit!
  67. 67. Cucumber Scout Plugin
  68. 68. Cucumber Scout Plugin http://github.com/jnewland/scout-plugins/raw/cucumber_ci/ cucumber_ci/cucumber_ci.rb
  69. 69. Cucumber Scout Plugin
  70. 70. Cucumber Scout Plugin ZOMG!
  71. 71. Cucumber Scout Plugin Sent via email
  72. 72. Power to monitor anything and everything
  73. 73. Clearly Defined Business Value
  74. 74. I N R U B Y
  75. 75. 24/7/365
  76. 76. Know before your customers do
  77. 77. Thanks! Any questions?
  78. 78. Get in touch: Jesse Newland jesse@railsmachine.com @jnewland github.com/jnewland
  79. 79. Flickr Creative Commons Photos http://flic.kr/p/4EjsDJ http://flic.kr/p/nxAqt http://flic.kr/p/c1UTf http://flic.kr/p/4VWY5F http://flic.kr/p/5BGJMu http://flic.kr/p/EKbuF http://flic.kr/p/5iTuua http://flic.kr/p/5xitHh http://flic.kr/p/77oAy2 http://flic.kr/p/4uE9Wz http://flic.kr/p/CdYo8 http://flic.kr/p/65KZaJ http://flic.kr/p/k2LCm http://flic.kr/p/7JKj5H http://flic.kr/p/71vxY6 http://flic.kr/p/79HLb5 http://flic.kr/p/5aZYkP http://flic.kr/p/xiYny http://flic.kr/p/79ikH2 http://flic.kr/p/68vjKV http://flic.kr/p/6rhqad http://flic.kr/p/rvc1 http://flic.kr/p/4MdrW8 http://flic.kr/p/6y7EfX http://flic.kr/p/5WuXzM http://flic.kr/p/2Mxkhp http://flic.kr/p/3jzrJ http://flic.kr/p/5t7h5 http://flic.kr/p/5B4TaF http://flic.kr/p/29qc7 http://flic.kr/p/4FAf2R http://flic.kr/p/JBaj http://flic.kr/p/39poLP http://flic.kr/p/smfab http://flic.kr/p/6nq52E http://flic.kr/p/4t5Qf9 http://flic.kr/p/yQuMG http://flic.kr/p/M8kdv http://flic.kr/p/jZ5Ae http://flic.kr/p/z3eWm http://flic.kr/p/4yhzz http://flic.kr/p/4XAQs7

×