For enterprise projects, maintaining automation tests suite is always a challenge. * Detailed walkthru of automation pyramid containing different types of specification driven tests layer. * Framework built using RSpec, Spinach, Cucumber, webkit and Webdriver with examples * Process of evolving and building specification tests * Tips to maintain test suite and minimize test run durations

1. SPECIFICATIONS – TEST
AUTOMATION PYRAMID
By
Sathyan Sethumadhavan
satnair@thoughtworks.com

2. Pre-Requisite:
http://www.slideshare.net/satnair/final
-version-for-public

3. Agenda
Why specifications
Approach to specifications
Specifications pyramid
Learning's
Q&A

4. Keywords
Specification – More than test – Feature/Requirement
Cucumber – Software behavior in plain text
Capybara – Tests application by simulating a real user
Spinach - Encapsulation and modularity of your step definitions
Headless webkit - A capybara driver - WebKit via QtWebKit.
R-Spec - Test-Driven Development

5. Let’s talk about
Why specifications
Approach to specifications
Specifications pyramid
Learning's
Q&A

6. Trying to solve these challenges?
Defects leakage - missing
Requirements
Tests and requirements
Frequent changes -
documents becomes stale
breaks existing
functionalities
Long Regression Spends more time on defect
Cycles management and triage
Less confidence -
Existing test coverage

7. Trying to solve these challenges?
Defects leakage - missing
Requirements
Less confidence -
Existing test coverage

8. Let’s talk about
Why specifications
Approach to specifications
Specifications pyramid
Learning's
Q&A

9. Specifications
Maintaining
Specifications
Executing
Specifications
Implementing
Specifications
Evolving
Specifications

10. Evolving Implementing Executing Maintaining
Story
• Intial Cukes creation Development • Manual testing for
• Inputs from Business, each Story
Technical and Testing • For TDD, writing • Evolved as part of
• Tests from Unit to RSpec exploratory testing
End-to-End • Checklist for story
completion
• Referred during dev-
Story box
Testing
Creation

11. Evolving Implementing Executing Maintaining
Story
• Intial Cukes creation Development • Manual testing for
• Inputs from Business, each Story
Technical and Testing • For TDD, writing • Evolved as part of
• Tests from Unit to RSpec exploratory testing
End-to-End • Checklist for story
completion
• Referred during dev-
Story box
Testing
Creation

12. Evolving Implementing Executing Maintaining
Story
• Intial Cukes creation Development • Manual testing for
• Inputs from each Story
Business, Technical • For TDD, writing • Evolved as part of
and Testing RSpec exploratory testing
• Tests from Unit to • Checklist for story
End-to-End completion
• Referred during dev-
Story box
Testing
Creation

13. Let’s talk about
Why specifications
Approach to specifications
Specifications pyramid
Learning’s
Q&A

14. Evolving Implementing Executing Maintaining
Cucumber +
Capybara Webdriver
Specification
Pyramid Spinach + Headless
Webkit
RSpec + API

15. Evolving Implementing Executing Maintaining
Algorithms / business logic
validation
API methods validation
Non-UI
RSpec + API Created as part of story
development

16. Evolving Implementing Executing Maintaining
Algorithms / business logic
validation
API methods validation
Non-UI
RSpec + API Created as part of story
development

17. Evolving Implementing Executing Maintaining
Presentation layer validations
Story level validations
Spinach + Jobs related functionalities
Headless Webkit
Created after story development
RSpec + API complete (QA’s / Dev’s)

18. Evolving Implementing Executing Maintaining
Presentation layer validations
Story level validations
Spinach + Jobs related functionalities
Headless Webkit
Created after story development
RSpec + API complete (QA’s / Dev’s)

19. Evolving Implementing Executing Maintaining
End user entry and exit scenarios
Cucumber +
Capybara
Webdriver Story level UI validations
Spinach + Headless End-To-End business scenarios
Webkit
Created after story development
RSpec + API complete (QA’s / Dev’s)

20. Evolving Implementing Executing Maintaining
End user entry and exit scenarios
Cucumber +
Capybara
Webdriver Story level UI validations
Spinach + Headless End-To-End business scenarios
Webkit
Created after story development
RSpec + API complete (QA’s / Dev’s)

21. Evolving Implementing Executing Maintaining
Cucumber + Developer/Tester
Laptop
CI pipeline
Capybara
• Smoke Tests • Smoke and E2E tests
Webdriver • Less mocks
Spinach + Headless
Webkit
Developer/Tester CI pipeline
Laptop
• Respective modules • After check-in
• Before each check-in • Before smoke and
RSpec + API • Mocks and Cassettes end-to-end

22. Evolving Implementing Executing Maintaining
Change requests Identify specification layer
New requirements Modify existing
specifications/if exists
Regression/User acceptance
defects Create/Update/Delete as
requirement evolves

23. Let’s talk about
Why specifications
Approach to specifications
Specifications pyramid
Learning's
Q&A

24. Key learning's
Specifications - Before development
Fail fast – 95% of functionalities gets validated before check-in
Specifications verified during dev-box (including multiple
browsers)
Categorize specifications - Appropriate layer
Parallelize test runs
Keep test code equivalent to developer’s code
Executable specifications
Defects backlog not beyond a threshold (for example 20 defects)
Defects root-cause - Feedback to the team

25. Let’s Recap
Defects leakage - missing
Requirements
Tests and requirements
Frequent changes -
documents becomes stale
breaks existing functionalities
Long Regression Spends more time on defect
Cycles management and triage
Less confidence -
Existing test coverage

26. Tools Available
• Jbehave, Cucumber, Cuke4Duke with
Java different drivers (Selenium,Sahi)
• Twist (Thoughtworks Studios)
• Specflow, Cucumber, Cuke4Nuke,
.NET Coypu with drivers like Selenium, White,
WatiN

27. Q&A

28. How can we help?
ThoughtWorks is a global custom software solutions consultancy trusted by
many of the world’s leading businesses with their most complex and critical
systems. We deliver consulting grounded in delivery expertise, build custom
applications and help organisations across all market sectors to drive IT
efficiency – working to an exceptionally high standard.
Contact us
satnair@thoughtworks.com
Private and Confidential. © 2012 ThoughtWorks. All rights reserved.