Stefan Mohacsi - 15 years of Evolving Model-Based Testing - EuroSTAR 2012
•
0 likes•930 views
This document summarizes Stefan Mohacsi's 15 years of experience with model-based testing. It describes the evolution of his automated test case generator from early Prolog tools to modern model-based testing tools integrated with test management systems. Key developments included adding support for graphical user interfaces, task flow modeling, requirements tracing, test data generation, and integration with test execution tools. The experience showed that model-based testing improves test efficiency and reduces maintenance costs compared to manual testing when applied correctly with a focus on reusable test models and integration within a test framework.
Recommended
Recommended
More Related Content
Similar to Stefan Mohacsi - 15 years of Evolving Model-Based Testing - EuroSTAR 2012
Similar to Stefan Mohacsi - 15 years of Evolving Model-Based Testing - EuroSTAR 2012 (20)
More from TEST Huddle
More from TEST Huddle (20)
Recently uploaded
Recently uploaded (20)
Stefan Mohacsi - 15 years of Evolving Model-Based Testing - EuroSTAR 2012
- 1. 15 Years of Evolving Model-based Testing - An Experience Report Stefan Mohacsi, Atos (Austria) www.eurostarconferences.com @esconfs #esconfs
- 2. 2 15 Years of Evolving Model-based Testing - An Experience Report Stefan Mohacsi, Atos (Austria)
- 3. 3 Traditional Test Automation can be Useful, but… ▶ Test cases still have to be designed manually ▶ Achieving systematic coverage is a challenging task ▶ Capture & replay of GUI test cases produces scripts that are hard to understand & maintain ▶ Maintaining hundreds of test scripts in each test cycle can require enormous effort Solution: Model-based Testing!
- 4. 4 What is Model-based Testing (MBT)? ▶ Tests are generated automatically from a model of the system under test (SUT) ▶ In case of changes to the SUT, only the model needs to be maintained, the tests are updated automatically ▶ Benefits: Systematic coverage, early defect detection, visualisation, significantly reduced maintenance effort Create/Update Test Model Generate Test Cases Generate Test Data Execute Test Cases
- 5. 5 Challenges of MBT (1) Which Model to Use? Reusing a model created for code generation seems like the obvious solution but ... ▶ UML models are often complex, not formal enough and not focused on testing ▶ If code and test cases are generated from the same source, defects in the source will not be detected! A separate test model is recommended, but requires additional effort
- 6. 6 Challenges of MBT (2) How to Bridge the Gap between Keyword and Script? ▶ Most test generators only produce abstract keywords instead of executable scripts ▶ For each keyword, it is necessary to manually implement and maintain a test script ▶ Additional effort for preparing test data, GUI object repositories etc. may be required A generator that can produce executable test scripts would reduce the creation and maintenance effort significantly
- 7. 7 Now Let’s Go on a Journey through Evolution...
- 8. 8 First Life Forms: Automated Test case Generator ATG ▶ Prolog tool written in 1993 by Armin Beer and Joachim Manz for testing Siemens Hicom software ▶ Test ‟model” had also to be specified in Prolog ▶ Generation algorithm tried to find test sequences that satisfied all constraints ▶ Suitable for text-based interfaces (not for GUIs) Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary
- 9. 9 Cambrian Explosion: The Research Project IDATG IDATG = Integrating Design and Automated Test Case Generation ▶ Started in 1997 as Cooperation between Siemens and University of Linz ▶ Model editor and generator written in Visual C++ ▶ Layout information for simple Win32 GUIs could be imported from MFC resource files ▶ Transition-based constraint satisfaction algorithm for finding valid test sequences ▶ Export of test scripts for Segue QA-Partner® (later known as Borland SilkTest®) Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary
- 10. 10 Early IDATG Model Editor Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary Imported GUI Objects State Transitions Lesson Learned: Model creation requires skill and additional effort, but facilitates early defect detection
- 11. Evolutionary Improvements: The GUI Spy ▶ Various types of GUI objects were not included in MFC 11 resource files and had to be specified by hand ▶ In 1999, a built-in GUI Spy solved the problem: Complete Win32 GUIs could now be recorded directly from the screen Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary
- 12. Diversification: WinRunner® Interface 12 ▶ Being restricted to only one test execution tool had several disadvantages ▶ A new interface to Mercury WinRunner® (later HP) considerably increased the number of potential users Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary
- 13. Exploring a new Realm: Task Flow Modeling 13 ▶ Working with transition-based models was arduous and required advanced skills ▶ In 2001, a more intuitive ‟task flow” concept was invented based on Activity Diagrams ▶ Building Block Concept: Each step may either represent an atomic step (blue) or an entire step sequence (yellow) Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary Lesson Learned: Re-use of parametrizable building blocks minimizes the effort for test creation and maintenance
- 14. Task Flows in IDATG 14 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary Task flow with building blocks Reusable building block ‟Login” Recorded ‟Login” dialog Lesson Learned: Start modelling at an abstract level, then add more detail step by step
- 15. Conquering new Niches: Support for Java and Web GUIs 15 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary ▶ Being restricted to Win32 GUIs was a major drawback ▶ In 2001, a special GUI Spy for importing Java GUIs was developed ▶ In 2002, an interface to Internet Explorer for recording web pages followed Lesson Learned: Ensuring GUI object recognition is THE key challenge of automated test execution. Don’t expect unskilled testers to do this but employ specialists instead!
- 16. Evolving new Capabilities: Requirements Tracing 16 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary ▶ Tracing requirements is essential for measuring coverage during the system test ▶ An editor for defining a requirement hierarchy and assigning requirements to tasks was the logical next step Lesson Learned: Involve test experts in early reviews of requirement and design specifications in order to avoid testability problems later on.
- 17. Mass Extinction & Reanimation: ICN Disaster and First ESA Project 17 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary ▶ In 2002, the drastic reduction of the Siemens mobile phone sector ICN left the IDATG project without its main sponsor ▶ Close to cancellation, the project was revived in 2004 by a new opportunity in space ▶ After adding support for non-GUI interfaces, IDATG became key part of a generic test framework for ESA’s Satellite Control and Operation System
- 18. 18 ESA Intelligent Test Framework IDATG Test Commander (Tcl/Tk) GUI Test Drivers Satellite Control and Operation System API Test Drivers Manual Test Photo: ESA
- 19. Continual Growth: Test Data Generation 19 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary ▶ From 2006, it became possible to generate test data in addition to test sequences ▶ Several systematic methods are supported (equivalence partitioning, boundary value analysis) ▶ Invention of CECIL (Cause-Effect Coverage Incorporating Linear boundaries) in collaboration with the SoftNet Austria competence network ▶ Connection between test data and task flows can be established by references
- 20. Chaotic Mutations: Randomized Testing 20 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary Random test methods can be applied to ▶ cover a much larger section of the input domain with relatively low effort ▶ discover faults that are caused by rare input combinations ▶ obtain statistic measures, e.g., for assessing the stability of a system We invented a hybrid algorithm that combines pure random testing with heuristic search strategies in interleaving phases to increase the overall coverage
- 21. Symbiosis: Test Management Integration 21 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary ▶ A close integration between all components of a test framework is crucial for an efficient test process ▶ In 2010, IDATG was integrated into the TEMPPO test framework and renamed to ‟TEMPPO Designer” ▶ Test structures are passed from TEMPPO Test Manager to TEMPPO Designer, which returns the generated test cases Lesson Learned: The efficiency of MBT depends on the process maturity and a well integrated tool framework
- 22. Dawn of a New Age: Atos and QTP 22 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary ▶ The future of IDATG was once again shrouded in uncertainty when Siemens IT Solutions and Services was taken over by Atos in 2011 ▶ Fortunately, the new company recognized the method’s potential and established a Global Competence Center for Model-based Testing ▶ Finally, an interface to HP QuickTest Professional® (QTP) was created ▶ In addition to generating QTP scripts, importing object repositories into IDATG is supported
- 23. The Summit(s) of Evolution: Model-based Testing Today 23 Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary ▶ In a major ESA project, the ROI of MBT was measured ▶ It turned out that the additional effort for introducing MBT paid off after 4 test cycles ▶ See also: Experiences of Test Automation by Dorothy Graham and Mark Fewster (Addison-Wesley 2012) 4000 3500 3000 2500 2000 1500 1000 500 0 1 4 7 10 13 16 19 Test Repetitions Effort (Hours) Manual Test MBT
- 24. Model-based Test Process with TEMPPO Designer V15.0 24 TEMPPO Designer (IDATG) Test Structure Definition Task Flow Modeling Test Data Generation Low-Level Specification Test Case Generation Test Execution Tool TEMPPO Test Manager Test scripts GUI data Control test run Test structure data Test case data
- 25. 25 Practical Hints ▶ Start MBT with test scenarios that have the highest ROI (many test repetitions, easy to automate) ▶ Pay attention to the reusability and maintainability of the model components, e.g., use parametrizable building blocks ▶ If MBT is correctly applied, test maintenance costs can be significantly reduced! ▶ Don’t forget to plan some experience-based tests. A test model is a good thing but cannot replace human experience and intuition! Precambrian Cambrian Ordovician Silurian Devonian Carboniferous Permian Triassic Jurassic Cretaceous Paleogene Neogene Quaternary
- 26. 26 Contact Information Stefan Mohacsi Atos IT Solutions and Services Head of CoC Model-based Test Phone: +43 (0) 50618 / 43417 e-Mail: stefan.mohacsi@atos.net THANK YOU !