I presented this deck at a SIGiST Israel event. This presentation is of a case study that compares Coded UI, TestComplete and Ranorex in a complex project over a period of 6 months.

1. Maximizing UI Automation
– A Case Study
Noam Kfir | ALM Senior Consultant | Sela Group
Copyright © Sela Group 2012

2. The Company
 Hi-tech security industry
 Hundreds of employees
 Offices and customers all over the world
 Customers include governments and large corporations
Copyright © Sela Group 2012

3. Process Characteristics
 Waterfall methodology
 10 years in development
 30 programmers
 12 manual testers
 No automated tests
 Very few automated processes
Copyright © Sela Group 2012

4. Product Overview
 Intelligence – gathering, managing, dispersing, analyzing
 Mission-critical
 More than 800 concurrent users
 Real time client/server application
 Many complex input forms and reports
 Multi-lingual and bi-directional
Copyright © Sela Group 2012

5. The Product is Complex
 More than a million LOC
 3rd party control libraries (Telerik, Janus, Infragistics,
and more)
 Very complex application flow, many screens
 WinForms embedded in WPF
 Clients communicate directly with the DB instead of
via the server
 Integration with other company products
 Difficult to debug due to security considerations
 The company cannot examine client data
 Limited logging
Copyright © Sela Group 2012
The Large Hadron Collider (LHC) is a particle accelerator located at CERN, near
Geneva, Switzerland.
It lies in a tunnel under France and Switzerland.

6. The Big Problem
 The end users report too many bugs
 The product fails to handle the expected load
 The client is dissatisfied and threatens to end
the relationship
 The company does not trust their product
 Every QA cycle is very long (lots of “full
regressions”)
 The company reevaluates its quality control
strategy
Copyright © Sela Group 2012
Problem Solving
We can’t solve problems by using the same kind of
thinking we used when we created them…

7. Short Term Solutions
 On-site QA
 On-site development
 On-site fixes – not synchronized with
source control
 Lots of hacks and improvised solutions
 Endless “lessons learned”
 Off with their heads…
Copyright © Sela Group 2012

8. Long Term Solutions
 Full transparency
 Improve communication with the customer
 Reports – more, richer, live
 Improve and add automated builds
 Gated check-ins
 Nightly builds
 Full builds
 Automated deployment
 Much more testing
Copyright © Sela Group 2012

9. New Testing Strategy
 Unit testing
 Use TFS lab environments
 Easier to reproduce bugs
 Snapshots
 Traceability
 Manual testing
 End-to-end scenarios
 Sanity and full regressions
 Load testing – realistic conditions
 Server API testing
 Automated UI testing
Copyright © Sela Group 2012

10. Attempt 1: Coded UI (2010)
The Good
 Intention-aware recording
 Support for multiple UI technologies
 Full TFS build integration including full
support for lab environments
 Extensible object recognition
 Very low learning curve for programmers
 Same language as the product – C#
The Bad
 Incomplete tooling
 Object recognition needs improvement
(Janus and pure graphical controls)
 Very difficult for manual testers
Copyright © Sela Group 2012

11. Coded UI (2010) ROI – After 2 Man-Months
 Included with VS Ultimate
 0 end-to-end tests
 Excellent infrastructure
Copyright © Sela Group 2012

12. Attempt 2: TestComplete
 Better object recognition
 Designed for manual testers, good for
building simple keyword-driven tests
 Inconsistent object recognition
 Very difficult scripting platform, weird
language, very poor documentation
 Not the same language as the product,
C# possible with poor COM Interop
implementation
 Poor integration with TFS
 Steep learning curve
Copyright © Sela Group 2012
The Good The Bad

13. Test Complete ROI – After 2 Man-Months
 $15,000 for several IDE licenses
 $400 per build agent
 30 end-to-end tests
 No infrastructure or reusable scripts
 Agents too expensive for load and stress testing
Copyright © Sela Group 2012

14. Attempt 3: Ranorex
 Excellent object recognition
 Designed for both testers and
programmers, has IDE and .NET API
 Partial TFS lab build process integration
 Unlimited agents usable for load and
stress tests
 TFS integration needs some work
 Tests need maintenance
 Medium learning curve
Copyright © Sela Group 2012
The Good The Bad

15. Ranorex ROI – After 2 Man-Months
 $22,000 for 5 IDE licenses
 Unlimited agents
 60 end-to-end tests
 Rich reusable infrastructure
 After 3 man-months  120 end-to-end tests
Copyright © Sela Group 2012

16. Comparison Matrix
Coded UI TestComplete Ranorex
Target Programmers Testers Both
Cost Part of VS Ultimate $15,000 $22,000
Agents Unlimited $400 per agent Unlimited
Learning Curve Easy Difficult Medium
TFS Integration Yes No Partial
After 2 Man-months 0 tests (due to Janus)
Good infrastructure
30 tests
No infrastructure
60 tests
Good infrastructure
Copyright © Sela Group 2012

17. The Ranorex Solution
 Excellent object recognition for multiple UI technologies
 Load and stress tests use same UI automation platform
 Reporting under one umbrella
 Bugs are caught much sooner
 Better collaboration between testers and programmers
 Customers can see and run the UI automation tests
 Fewer defects and deployment errors in production
 Fewer on-site team members and sessions
 Adopted by other product teams
Copyright © Sela Group 2012

18. DEMO
UI Automation with Ranorex
Copyright © Sela Group 2012

19. Thank You!
Email: noamkfir@sela.co.il
Blog: http://noam.kfir.cc
Twitter: @NoamKfir
Copyright © Sela Group 2012