Our project was a classic example of Selenium gone wild! As our team embraced the test automation journey, we went crazy and implemented tons of Selenium tests, one for every permutation possible. Soon we realized our feedback cycles were delayed. Our builds were taking hours instead of minutes. And we had a set of complex, fragile tests, which resulted in a lot of false-negative scenarios and finger pointing. At this point, our team had realized that this is not the path forward. We decided to seriously look at our Selenium tests. We pretty much moved 80% of our Se tests to lower-layers (non-GUI based) tests. And now we have the right testing pyramid on our project. In this presentation, Naresh Jain explains IDeaS' journey (strategy, techniques, tools, mindset-change and approaches we took) through this transition.

1. ManualChecking
End-to End Selenium Tests 70%
Integration Tests
20%
Unit Tests
10%
Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
End to End Flow Tests 4%
UI 1%
Naresh Jain @nashjain
Selenium Detox for Achieving the Right Test Pyramid

2. Selenium Detox for Achieving the Right Test Pyramid
Naresh Jain
@nashjain

4. Achieving the Right Test Pyramid
Aditya Saigaonkar
Kirtesh Wani

5. Background
IDeaS is capable of applying 25 years of experience, expertise and client dedication toward an expanding suite of automated revenue management solutions.

6. Background

7. Background

8. Background -2011
5%
95%No AutomationAutomation Coverage

9. Background
1 Months
2Months
0
0.5
1
1.5
2
2.5
3
3.5
MonthsTypical Release Cycle
Development
Regression

10. Background

11. Critical Issues reported from Production
0
2
4
6
8
10
12
Critical Data Issues
Cosmetic
Localization
Issues

12. Background

13. After Automation Introduced -2013
Selenium Tests 45%
Unit & Integration Tests 20%
No Automation 35% Automation Coverage

14. After Automation Introduced
2.25 Months
3 Weeks
0
0.5
1
1.5
2
2.5
3
3.5
MonthsTypical Release Cycle
Development
Regression

15. Critical Issues reported from Production
0
2
4
6
8
10
12
Critical Data Issues
Cosmetic
Localization

16. Duration…
0
0.5
1
1.5
2
2.5
YearsDuration of Implementation
2 Years

17. Demo
•Selenium WebDriverDemo

18. State of the Product
Tightlycoupledwith UI
Browser DependentData tests
Falsefails
Difficult to pinpoint real culprit
Difficult to maintainand time consuming as test are fragile
Laggingwith current release development
Image Courtesy: Alex Yakyma

19. Analysis of Bugs Report
10%
90%Data FailuresBug Report Analysis
UI Navigation Failures

20. State of the Product
ManualChecking
End-to End Selenium Tests 70%
Integration Tests
20%
Unit Tests
10%

21. ManualChecking
End-to End Selenium Tests 70%
Integration Tests
20%
Unit Tests
10%
Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
End to End Flow Tests 4%
UI 1%
Typical test strategies lead to an inverted testing pyramid
This is the need of the hour….
Courtesy: Naresh Jain @AgileFAQs.com
The Test Pyramid

22. Our transition story…
•We decided to "Test things right , at right place on right time" which would result in "right test pyramid“
Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
End to End Flow Tests 4%
UI 1%

23. Our transition story…
Application UI
Service Layer
DAO Layer
Automated UI Tests
Tightly Coupled
Fragile
Lag with current development
Automate One Level Below UI
UI Independent
Robust & Stable
With current devcycle
Fast Execution

24. Introduced BDD with Cucumber

25. Dev & QA Pairing
Developer
QA

26. Quick Demo

27. Current Status
Selenium
Tests 10%
Workflow Tests: 40%
No Automation 20%
Unit & Integration Tests 30% Test Automation Status

28. After Right Testing Pyramid
1 Week
2.75 Months
0
0.05
0.1
0.15
0.2
0.25
0.3
0
0.5
1
1.5
2
2.5
3
3.5
MonthsTypical Release Cycle
Regression Time
Development

29. Critical Issues reported from Production
0
2
4
6
8
10
12
Critical Data Issues
Cosmetic
Localization

30. Duration…
0
0.5
1
1.5
2
2.5
3
3.5
MonthsDuration of Implementation
3 Months

31. Pain points
•Learning curve for the team
•Difficult to implement in case where no API is exposed or tight coupling with UI
•Heavy dependency on pre-populated database

32. Looking ahead
•Expose RESTful APIs for all upcoming features
•Move BDD specs to lower layers
•Moving away from baseline database
Courtesy: http://3.bp.blogspot.com/

33. © Copyright Integrated Decisions and Systems, Inc. (IDeaS–A SAS COMPANY)
Visit IDeaSonline at www.ideas.com
Questions?
Aditya.Saigaonkar@ideas.com
Kirtesh.Wani@ideas.com
Naresh Jain @nashjain