Software Quality and Test Strategies for Ruby and Rails Applications


Published on

Presentation of the talk I gave at RubyConf India 2011 ->

Published in: Technology

Software Quality and Test Strategies for Ruby and Rails Applications

  1. 1. Software Quality and Test Strategies forRuby and Rails Applications - Bhavin Javia May 29th, 2011
  2. 2. About Me Ex ThoughtWorkerAgile/RoR Consultant - Ennova, Australia Founder - Ennova, India
  3. 3. bhavinjavia
  4. 4. Co-Author Dr Adrian SmithManaging Director - Technology Agile Coach @
  5. 5. Content Software Quality Test Strategy Ruby/Rails Testing EcosystemQuality and Testing for Rails Projects
  6. 6. Software Quality
  7. 7. Importance of Quality
  8. 8. “Quality is the most important factor for deriving long term value from a software product” - Jim Highsmith (Co-Author of Agile Manifesto)
  9. 9. ”Continuous attention to quality can helpmaintain velocity, maintain the ability to deliver, without it the team will slow down” - Robert C. Martin (Author of Clean Code)
  10. 10. Why worry ? Needs to be built in (implicitly) Needs to be defined (to prioritize vs cost/scope/time)Agile project delivery relies on continuous attention to quality Begins at the code level
  11. 11. code_quality != "App Quality"
  12. 12. Internal Quality vsExternal Quality
  13. 13. External Quality( Quality as perceived by users ) interface design performance defects user experience
  14. 14. Internal Quality( Quality of the implementation and system architecture ) duplication complexity coupling test coverage
  15. 15. “Technical Debt may be costing you more than you imagined!” - Jim Highsmith
  16. 16. The trade-off
  17. 17. Managing Quality
  18. 18. Managing Quality Set Goals "Max 3 clicks to any page" Measure "Page X takes 5 clicks" Prioritize "Make page X reachable in <= 3 clicks"
  19. 19. How to achieve Quality ?
  20. 20. TEST TEST TEST
  21. 21. What/When/How to Test ?
  22. 22. " Test Strategy "
  23. 23. Test Strategy Template
  24. 24. Test Strategy Template Not a Waterfall style documentA template to structure the discussion Prompts for things to consider Flexible format
  25. 25. PurposeCreate a shared understanding of approach tools targets timing of test activities
  26. 26. Guiding Principles Shared Responsibility Everyone is Responsible for Testing and Quality Test AutomationAll types of tests should be automated (except exploratory tests) Data Management Production data must be obfuscated before use Test Management Tests, documents and data treated as production code
  27. 27. Quality and Test Objectives Correctness Integrity Maintainability Availability Interoperability Performance
  28. 28. Quality and Test Objectives Correctness Features and functions work as intended e.g. 0 Critical Defects Integrity Prevent unauthorized access or information loss e.g. All access via HTTPS, Encrypted passwords Maintainability Easy to add new features or fix defects e.g. Code Complexity < 8, Unit Test Coverage > 80% ...
  29. 29. Quality and Test Objectives Availability Planned uptime percentage e.g. 99.99 % availability Interoperability Ease of information exchange with other systemse.g. Published & versioned API, Supports - IE 8, FF 3.5, Chrome 11 etc Performance Responsiveness & Scalability of the system e.g. Apdex Score > 0.9, Response Time < 200ms
  30. 30. Quality and Test Objectives Target Measure Prioritize
  31. 31. Test Scope Systems and features in and out of scope Test both business processes and the technical solution Specify regions and sub-regions included in testing Identify interfaces with other systemsSystem / feature / integration / region / environment /dependency / service In Scope Things to be tested with automation Things to be tested manually Out of Scope Things we cant test e.g. Third Party systems
  32. 32. Test Approach Identify the test types Identify the toolsInclude the timing of execution
  33. 33. Test Types Unit Functional Integration Acceptance PerformanceData Conversion
  34. 34. Test Approach Identify Tools e.g. rspec, cucumber etc Decide Timing of Executione.g. dev machine, CI server, after each commit etc
  35. 35. Test Automation Pyramid
  36. 36. Test Preparation Use requirements captured in user stories Include acceptance criteria in user storyDecide approach to prepare and execute tests
  37. 37. Example
  38. 38. Acceptance Test
  39. 39. Environments Development Unit, Functional, Acceptance tests IntegrationContinuous Integration - Unit, Functional, Acceptance tests Code Analysis/Metrics Staging For exploratory testing Demos to customers Production Smoke/Sanity tests & Monitoring
  40. 40. Test Executionsteps in preparation for deployment/release Build the system Populate test/reference data Execute automated tests Generate test report/code metrics
  41. 41. Test Data Management System and user acceptance tests Use subset of production data Performance/volume/stress test Use full size production files Generate large volumes of data
  42. 42. begin database.load! "Sensitive Production Data"ensure database.obfuscate! "Sensitive Data"end
  43. 43. Defect Management
  44. 44. Why defects ?
  45. 45. :missing_tests => "Defects"
  46. 46. Defect ManagementDefects only raised when not fixed immediately Capture conditions and severity Capture steps to reproduce Critical Major Minor Trivial
  47. 47. Defect LifeCycle Identify Prioritize Assign Severity Analyze Write test to reproduce Resolve Fix code to pass the test Verify Run all tests, verify manually Close
  48. 48. Test Strategy Example
  49. 49. Principles and Objectives
  50. 50. Test Scope
  51. 51. Test Approach
  52. 52. Ruby/RailsTesting Ecosystem
  53. 53. Whats different in Ruby/Rails ?
  54. 54. Advantages Testing built right-in Passionate developersVibrant open source community Excellent tools support
  55. 55. So what are our options ?
  56. 56. There are ... 23 Testing Frameworks 8 Javascript Testing Frameworks 8 Browser Testing Frameworks 8 Distributed Testing Frameworks 6 Object Mocking Frameworks 7 Web Mocking Frameworks 2 Continuous Testing Frameworks7 Continuous Integration Frameworks 12 Code Metrics Frameworks and many more ...
  57. 57. raise "OMG !!"
  58. 58. What to Test ?Anything that could possibly break Public interface of each layer Whole application stackDepends on what else youre testing
  59. 59. What NOT to test ?Cost of test > 10 * Cost of breakage (very rare) Internal implementation Framework code
  60. 60. Youre doing it wrong if Fixing tests all the time Tests that always fail together Tests that never fail
  61. 61. What to Test ? Models Controllers Views Helpers Libraries APIs
  62. 62. Testing Options Models Test::Unit (Ruby 1.8, built-in)MiniTest (Ruby 1.9, built-in, replaces Test::Unit) Unit Tests (ActiveSupport::TestCase) Model Specs (rspec) Controllers Functional Tests (built-in) Controller Specs (rspec)
  63. 63. Testing Options Views assert_select (built-in) View Specs (rspec) Helpers ActionView::TestCase Helper Specs (rspec)
  64. 64. Testing Options Routes assert_routing (built-in) Routing specs (rspec) Mailers Unit Tests (built-in) Functional Tests (built-in)
  65. 65. Testing Options Workflow Integration Tests (built-in) Request Specs (rspec) BDD Cucumber + Capybara Cucumber + Rspec
  66. 66. Testing Options Performance Performance Tests (built-in) NewRelic
  67. 67. Testing Helpers Data Setup Fixtures (built-in) factory_girl Machinist
  68. 68. Testing Helpers Mocks/Stubs Rspec mocks Mocha Flexmock API / Web Services VCR Fakeweb Webmock
  69. 69. Testing Helpers Speed NullDB Autotest/Infinity Test/Watchr Spork parallel_tests TLB CI cijoe cruisecontrol.rb/goldberg integrity
  70. 70. Quality/Metrics Test Coverage rcov SimpleCov Code Analysis metric_fu Metrical Rails Best Practices
  71. 71. Which is the Best ? Even the God(s) Cant Say !!
  72. 72. Which is the Best for Me ?
  73. 73. IMHOWHATS_BEST = { :newbie => "Start with built-ins", :intermediate => "Checkout alternatives", :expert => "Venture into the obscure" }
  74. 74. WHATS_BEST[:god] = "Write your own !"
  75. 75. References http://www.jimhighsmith.com
  76. 76. Questions ? bhavinjavia @bhavinjavia