Downloaded 274 times
![1. Create Test
Model Actions
• [Rule] tags a
method for
exploration
• Requires defines
a exploration
precondition –
must be true to
explore
Model Driven Development: Taking BDD/ATDD to the Next Level
39](https://image.slidesharecdn.com/taking-bdd-to-the-next-level-140227131354-phpapp01/85/Model-based-Testing-Taking-BDD-ATDD-to-the-Next-Level-40-320.jpg)
Uploaded byBob Binder
Model-based Testing: Taking BDD/ATDD to the Next Level
The document discusses model-based testing (MBT) and its relationship with behavior-driven development (BDD) and acceptance test-driven development (ATDD), emphasizing the need for more models and less testing to effectively manage the complexities of software development. It highlights the limitations of traditional testing methods and advocates for a systematic, model-driven approach that can generate extensive test suites more efficiently. Additionally, it outlines the processes and tools involved, including the use of SpecFlow for generating test code and exploring software behavior.
More Related Content
Similar to Model-based Testing: Taking BDD/ATDD to the Next Level
Model-based Testing: Taking BDD/ATDD to the Next Level
- 1. Model-based Testing: Taking BDD/ATDDto the Next Level Robert V. Binder System Verification Associates rvbinder@sysverif.com #MoreModelsLessTests Chicago Quality Assurance Association February 25, 2014
- 2. The Incredible Vastness of Testing ModelBased Testing EatenAlive by the Testing Backblob Tell Me A Story More Models, More Tests Less Models, Less Tests Model Driven Development: Taking BDD/ATDD to the Next Level 1
- 3. The Incredible Vastness ofTesting
- 4. Saved by RevealingSub-domains • Grains of sand to fill the Grand Canyon ~ 1.68 × 1025 ~ 10 septillion • Trivial Program – – – – Data space: 21024 Sequences: 21024 Execution space: 22048 1.34 × 10616 Large subsets of any program’s execution space are revealing: if a bug exists in any member of that space and it is executed, an observable failure results. But, we rarely know exactly which sub-domains are revealing until we test. Model Driven Development: Taking BDD/ATDD to the Next Level 3
- 5. It’s release time. Doyou know where your bugs are? Aerospace Medical IT 0.7 0.8 Other Agile 1.9 1.8 4.4 5.0 6.0 6.0 Mobile Best In Class Average bugs per thousand equivalent source code lines, post-release 4.7 Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP . Model Driven Development: Taking BDD/ATDD to the Next Level 4
- 6. It’s release time. Doyou know where your bugs are? Aerospace Medical IT Best In Class Other 325 375 Agile Predicted total latent bugs, 500 KLOC Java 925 875 2511 2202 2350 Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP Model Driven Development: Taking BDD/ATDD to the Next Level 5
- 7. Driven to Distraction •Test Driven Development – TDD • Behavior Driven Development – BDD • Acceptance Test Driven Development • What is it? • How does it work? • What can it do for me? – ATDD • Model-based Testing – MBT Model Driven Development: Taking BDD/ATDD to the Next Level 6
- 8. Eaten Alive by the Testing Backblob EATENALIVE BY THE TESTING BACKBLOB
- 9. Where does thetime go? Total Number of Test Cases Sprint 1 Sprint 2 Sprint 3 Sprint 4 } Tests/testing for new features Existing manual Test Cases not executed Available Dev/Test Time Existing automated Test Cases not maintained Model Driven Development: Taking BDD/ATDD to the Next Level 8
- 10. The Attack ofthe BackBlob http://www.youtube.com/watch?v=GODDLgM1gKo
- 11. Tell Me aStory TELL ME A STORY
- 12. The Chat Server • • • • • Logon Listactive users Post a message Broadcast message Logoff !!! Client Endpoint Client Endpoint Model Driven Development: Taking BDD/ATDD to the Next Level Chat Server 11
- 13. TDD – BDD– ATDD • Test Driven Development • Behavior Driven Development • Acceptance Test Driven Development • Programming strategies – Test suites are scaffolding – Test each method/feature at least once – Rerun after any add/change • Not testing strategies Model Driven Development: Taking BDD/ATDD to the Next Level 12
- 14. The TDD Red-GreenCycle • Imagine a class and its responsibilities • Code its API only Logon Logon GetStatus GetStatus Logoff Logoff Model Driven Development: Taking BDD/ATDD to the Next Level 13
- 15. The Red-Green Cycle •Imagine a class and its responsibilities • Code its API only TestLogon • Write and run a simple test – it should fail Logon GetStatus Logoff Model Driven Development: Taking BDD/ATDD to the Next Level 14
- 16. The Red-Green Cycle •Imagine a class and its responsibilities • Code its API only TestLogon • Write and run a simple test – it should fail Logon GetStatus • Code implementation • Rerun all tests – expect pass Model Driven Development: Taking BDD/ATDD to the Next Level Logoff 15
- 17. The Red-Green Cycle •Imagine a class and its responsibilities • Code its API only • Write and run a simple test – it should fail TestLogon Logon TestGetStatus GetStatus • Code implementation • Rerun all tests – expect pass Logoff • Repeat until feature complete Model Driven Development: Taking BDD/ATDD to the Next Level 16
- 18. The Red-Green Cycle •Imagine a class and its responsibilities • Code its API only • Write and run a simple test – it should fail TestLogon Logon TestGetStatus GetStatus • Code implementation • Rerun all tests – expect pass TestLogoff Logoff • Repeat until feature complete Model Driven Development: Taking BDD/ATDD to the Next Level 17
- 19. TDD Limitations Process, Activities Components, Test Objects •No explicit linkage for user/customer needs, requirements, or stories • Gaps – UI automation – Feature testing Model Driven Development: Taking BDD/ATDD to the Next Level 18
- 20. Behavior Driven Development Process, Activities Components, TestObjects Features, Scenarios Feature Files Scenario A *DD Tool Scenario B Scenario C Test Objects *Unit Components Model Driven Development: Taking BDD/ATDD to the Next Level 19
- 21. Acceptance Test DrivenDevelopment Process, Activities Features, Scenarios UI Test Cases Feature Files Scenario A *DD Tool Scenario B Scenario C Test Scripts Test Objects Test Tooling *Unit System Under Test Model Driven Development: Taking BDD/ATDD to the Next Level 20
- 22. Features and Scenarios Feature:story Scenario: instance of a story Steps: Given: pre-condition When: trigger Then: verification condition Feature: Logon In order to use the chat server, As a validated chat client, I want to submit my credentials and establish a session Scenario: Submit Good Given A connection to When I send the Login Then I should see the Model Driven Development: Taking BDD/ATDD to the Next Level Credentials the server is open message "Hi Chatter" message 21
- 23. Overview Feature Files ScenarioA SpecFlow Scenario B Scenario C Test Objects Nunit Visual Studio Components http://www.specflow.org • Enter features, scenarios • Generates skeleton test code for each step – Extracts step names – Can run immediately • Developer adds step code • Test run reports • CI integration, more Model Driven Development: Taking BDD/ATDD to the Next Level 22
- 24. 1. Create a FeatureFile • SpecFlow generates a stub file • Edit as necessary • One feature • 1:N scenarios Model Driven Development: Taking BDD/ATDD to the Next Level 23
- 25. 2. Enter Scenario Steps •Given – Precondition – Data State • When – Trigger – API call, GUI action • Then – Verification conditions (expected results) Model Driven Development: Taking BDD/ATDD to the Next Level 24
- 26. 3. Inspect Skeleton TestCode • SpecFlow parses feature file • Generates skeleton test code for each scenario and step Model Driven Development: Taking BDD/ATDD to the Next Level 25
- 27. 3. Add TestCase Code • User adds more code as needed to implement each test case Model Driven Development: Taking BDD/ATDD to the Next Level 26
- 28. 4. Run TestSuite • The generated test code can be run immediately • Supports the redgreen cycle Model Driven Development: Taking BDD/ATDD to the Next Level 27
- 29. SpecFlow • BDD for VisualStudio • Link Feature Files to test code • Generates partial test code Feature File contains statements in Gherkin – Keywords + natural language • • Generates test code skeleton from steps • Feature File 1:N Steps per Scenario • Feature 1:N Scenarios per Feature File Steps may be parameterized Steps Rows • Variables and variable bindings in Rows – Scenarios – Test code generation using regular expression patterns Integration with TFS, CI, … Model Driven Development: Taking BDD/ATDD to the Next Level 28
- 30. Modelbased Testing THE VERY MODELOF A MODERN …
- 31. What is aModel? • Focus • Smaller, simpler • In IT systems – Behavior – Structure – Content Model Driven Development: Taking BDD/ATDD to the Next Level 30
- 32. IT Models Process, Activities Behavior, Structure, Content Model DrivenDevelopment: Taking BDD/ATDD to the Next Level Technology 31
- 33. IT Models • Notional(design) • As built (documentation) • Test Models – Similar to notional and as-built – Unique aspects
- 34. Test Models • Focuson aspects that support testing goals • Behavior – Sequence and time • Transformation – Input, output function • Verification (oracle) • Coverage • Risk Management Model Driven Development: Taking BDD/ATDD to the Next Level 33
- 35. Chat Server • Logon ChatServer – Must be first – Can’t be repeated Logon GetStatus Logoff • GetStatus – Only after logon – Returns number of calls since logon • Logoff – Only after logon • Typical RFC status codes Model Driven Development: Taking BDD/ATDD to the Next Level 34
- 36. Simple Test Model Logoff nc= 0 GetStatus System Under Test (SUT) Logon Logon Logon GetStatus assert (200) assert (401) assert (401) assert (401) Logoff nc++ GetStatus Logoff assert (200) Logoff assert (200) Model Driven Development: Taking BDD/ATDD to the Next Level assert (200) assert (n == nc) 35
- 37. Chat Server MessageSequence Login a 1234 Hi Chatter a Login b secret Hi Chatter b List Users Users: a, b !!! Client Endpoint Client Endpoint Model Driven Development: Taking BDD/ATDD to the Next Level Chat Server 36
- 38. Chat Server MessageSequence List Users Users: a, b Post “Pumpkin Latte #mypics” Ack a1 “Pumpkin Latte #mypics” Bcast a1 “Pumpkin Latte #mypics” Logoff b Ack Logoff b !!! Client Endpoint Client Endpoint Model Driven Development: Taking BDD/ATDD to the Next Level Chat Server 37
- 39. Spec Explorer Demo •Model-based testing tool – Microsoft Research – Used to test all MSFT APIs – Robust and stable – Visual Studio “power tool” • C# code, not cartoons • Generates standalone executable test suite Model Driven Development: Taking BDD/ATDD to the Next Level 38
- 40. 1. Create Test ModelActions • [Rule] tags a method for exploration • Requires defines a exploration precondition – must be true to explore Model Driven Development: Taking BDD/ATDD to the Next Level 39
- 41. 2. Define Accepting State(s) •Tag condition(s) that terminate an exploration Model Driven Development: Taking BDD/ATDD to the Next Level 40
- 42. 3. Create CordFile • Set options • Define machines – Limits Actions used in an exploration • Similar to regular expression • Composable • Very powerful! Model Driven Development: Taking BDD/ATDD to the Next Level 41
- 43. 4. Explore theModel • Select any explorable machine – Explore – Generate test suite – Run on-the-fly test Model Driven Development: Taking BDD/ATDD to the Next Level 42
- 44. 5. Review Exploration Graph • Showsaction/event sequence • Ellipse: model state computed by model program machine • Diamond: observation “event” – get something from SUT (input to the model program) • Right click to inspect any item Model Driven Development: Taking BDD/ATDD to the Next Level 43
- 45. 6. Define machine(s) togenerate test suites • Set options to generate test code • Use any machine or composite machine Model Driven Development: Taking BDD/ATDD to the Next Level 44
- 46. 7. Explore theTest Suite machine • Each root to leaf path becomes a test suite Model Driven Development: Taking BDD/ATDD to the Next Level 45
- 47. 8. Generate TestCode • Explores the test machine • Writes out test code using VS testing framework Model Driven Development: Taking BDD/ATDD to the Next Level 46
- 48. 8. Generate TestCode • Not intended to be maintained • Re-generate any time • Add helper code in the model to extend/customize Model Driven Development: Taking BDD/ATDD to the Next Level 47
- 49. 9. Run theTest Suite • Uses built-in Visual Studio test runner • Adapters – Start/stop chat clients and chat server – Other SUT setup/cleanup • Model Driven Development: Taking BDD/ATDD to the Next Level Test code sends messages, checks replies 48
- 50. Model State (test caseinputs) 10. Review Test Results • Detailed run log – – – – Event Handler (pass/fail) Requirement Trace VS test runner States Events Requirements • Add any other programmable function Model Driven Development: Taking BDD/ATDD to the Next Level 49
- 51. Spec Explorer Exploration • Interactivemodel development • Generate test suites • Machines Cord Declarations Model Program Options Machines define composable Action sequences • Options control exploration and data generation • Actions are explored – Rules define exploration conditions Model Data Structures Rules Actions Events Adapters – Events define expected SUT response • Data structures support exploration • Adapters abstract, control, and observe any interface, any platform Model Driven Development: Taking BDD/ATDD to the Next Level 50
- 52. MBT Driven Development Requirements,Stories, Scenarios... Explore Ambiguous, missing, contradictory, incorrect, redundant, incomplete Test Model Run Generate Model error, omission Test Suite Inputs (Test Sequences) Control Missing, incorrect Coverage Requirements Model Evaluate Expected Outputs (Test Oracle) SUT Observe Model Driven Development: Taking BDD/ATDD to the Next Level Bug 51
- 53. More Models, LessTests MORE MODELS, LESS TESTS
- 54. Model-based Testing inthe Real World • Windows protocol compliance (Web services, RPC, DCOM) • Total effort: 250 person years (mostly junior SDETs) • Saved 50 person years with model-based testing Model-based Testing 1.4 Days/Requirement 42% Less Time Per Requirement Traditional Testing 2.4 Days/Requirement Requirements Study Modeling Test Coding Adapter Coding Model Driven Development: Taking BDD/ATDD to the Next Level Test Execution 53
- 55. Scope of TestAsset Maintenance Model-based Testing Behavior Driven Development Model How much MBTgenerated test code must be maintained? Test Code None! Adapters Model Driven Development: Taking BDD/ATDD to the Next Level 54
- 56. More Models, LessTests • More Better Testing • Icing the BackBlob – Rule-based – Maintain model – Systematic and deep exploration – Regenerate test suites – Generate 1000s of scenarios in seconds – Models grow much more slowly than test code Model Driven Development: Taking BDD/ATDD to the Next Level 55
- 57.
- 58. Q&A #MoreModelsLessTests rvbinder@sysverif.com Model Driven Development:Taking BDD/ATDD to the Next Level 57
- 59. Learn More -MBT • UCCAT (User Conference on Advanced Automated Testing) Presentations http://ucaat.etsi.org/2013/program_conf.html • 2012 Model-based Testing User Survey http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf • Open Source MBT tools http://www.robertvbinder.com/open-source-tools-for-model-based-testing/ • Overview of MBT, list of commercial MBT tools http://www.slideshare.net/robertvbinder/model-basedtestingignite • Spec Explorer http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745 http://msdn.microsoft.com/library/ee620411.aspx • Microsoft Open Protocol Project http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#! Model Driven Development: Taking BDD/ATDD to the Next Level 58
- 60. Learn More: BDD,etc. • SpecFlow http://www.specflow.org/ • Using SpecFlow with WatiN http://msdn.microsoft.com/en-us/magazine/gg490346.aspx • SpecFlow Quick Start http://www.youtube.com/watch?v=SVwBz-ZTliw • Writing Cucumber Feature Files https://github.com/cucumber/cucumber/wiki • Using Lava Lamps to indicate Build Status https://wiki.jenkins-ci.org/display/JENKINS/Hudson+Build+Status+Lava+Lamps Model Driven Development: Taking BDD/ATDD to the Next Level 59
- 61. Content and ImageSources Grand Canyon Time Lapse. Ezra Kamer http://www.youtube.com/watch?v=NF6O6HuNJ0k Contrast (Order and Chaos). M. C. Escher, 1950. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=29270 The Blob, Theatre Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=GODDLgM1gKo http://www.imdb.com/title/tt0051418/ Order and Chaos II. M. C. Escher, 1955. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=41980 Defect densities: • Capers Jones, Software Quality: Analysis and Guidelines for Success • Robert E. Park. Software Size Measurement: A Framework for Counting Source Statements. Technical Report CMU/SEI-92-TR-020. Software Engineering Institute, Carnegie Mellon University Spec Explorer. Microsoft http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9956b-d9bfa4902745 SpecFlow http://www.specflow.org/ The Blob, Final Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=0GEJgR-bX0w Spec Explorer Chat Model. Microsoft. All other content Copyright © Robert V. Binder, 2014. Model Driven Development: Taking BDD/ATDD to the Next Level 60