Slides from presentation at the Chicago Quality Assurance Association, February 25, 2014.
Acceptance Test Driven Development (ATDD) and Behavior Driven Development (BDD) are well-established Agile practices that rely on the knowledge and intuition of testers, product owners, and developers to identify and then translate statements into test suites. But the resulting test suites often cover only a small slice of happy-path behavior. And, as a BDD specification and its associated test code base grows over time, work to maintain it either crowds out new development and testing or, typically, is simply ignored. Either is high-risk. That’s how Agile teams get eaten by the testing BackBlob.Model Based Testing is a tool-based approach to automate the creation of test cases. This presentation will outline the techniques and benefits of MBT, and show how model-based testing can address both problems. A detailed demo of Spec Explorer, a free model-based testing tool shows how a model is constructed and used to create and maintain a test suite.
Axa Assurance Maroc - Insurer Innovation Award 2024
Model-based Testing: Taking BDD/ATDD to the Next Level
1. Model-based Testing:
Taking BDD/ATDD to the Next Level
Robert V. Binder
System Verification Associates
rvbinder@sysverif.com
#MoreModelsLessTests
Chicago Quality Assurance Association
February 25, 2014
4. Saved by Revealing Sub-domains
• Grains of sand to fill the
Grand Canyon
~ 1.68 × 1025
~ 10 septillion
• Trivial Program
–
–
–
–
Data space: 21024
Sequences: 21024
Execution space: 22048
1.34 × 10616
Large subsets of any program’s execution space are revealing: if a bug exists in any
member of that space and it is executed, an observable failure results.
But, we rarely know exactly which sub-domains are revealing until we test.
Model Driven Development: Taking BDD/ATDD to the Next Level
3
5. It’s release time.
Do you know where your bugs are?
Aerospace
Medical
IT
0.7
0.8
Other
Agile
1.9
1.8
4.4
5.0
6.0
6.0
Mobile
Best In Class
Average bugs per
thousand
equivalent source
code lines,
post-release
4.7
Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .
Model Driven Development: Taking BDD/ATDD to the Next Level
4
6. It’s release time.
Do you know where your bugs are?
Aerospace
Medical
IT
Best In Class
Other
325
375
Agile
Predicted total
latent bugs,
500 KLOC Java
925
875
2511
2202
2350
Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP
Model Driven Development: Taking BDD/ATDD to the Next Level
5
7. Driven to Distraction
• Test Driven Development
– TDD
• Behavior Driven Development
– BDD
• Acceptance Test Driven Development
• What is it?
• How does it work?
• What can it do for
me?
– ATDD
• Model-based Testing
– MBT
Model Driven Development: Taking BDD/ATDD to the Next Level
6
9. Where does the time go?
Total
Number
of Test
Cases
Sprint 1
Sprint 2
Sprint 3
Sprint 4
}
Tests/testing for
new features
Existing manual
Test Cases not
executed
Available
Dev/Test
Time
Existing automated
Test Cases not
maintained
Model Driven Development: Taking BDD/ATDD to the Next Level
8
10. The Attack of the BackBlob
http://www.youtube.com/watch?v=GODDLgM1gKo
12. The Chat Server
•
•
•
•
•
Logon
List active users
Post a message
Broadcast message
Logoff
!!!
Client
Endpoint
Client
Endpoint
Model Driven Development: Taking BDD/ATDD to the Next Level
Chat
Server
11
13. TDD – BDD – ATDD
• Test Driven
Development
• Behavior Driven
Development
• Acceptance Test Driven
Development
• Programming strategies
– Test suites are scaffolding
– Test each method/feature at
least once
– Rerun after any add/change
• Not testing strategies
Model Driven Development: Taking BDD/ATDD to the Next Level
12
14. The TDD Red-Green Cycle
• Imagine a class and its
responsibilities
• Code its API only
Logon
Logon
GetStatus
GetStatus
Logoff
Logoff
Model Driven Development: Taking BDD/ATDD to the Next Level
13
15. The Red-Green Cycle
• Imagine a class and its
responsibilities
• Code its API only
TestLogon
• Write and run a simple test – it
should fail
Logon
GetStatus
Logoff
Model Driven Development: Taking BDD/ATDD to the Next Level
14
16. The Red-Green Cycle
• Imagine a class and its
responsibilities
• Code its API only
TestLogon
• Write and run a simple test – it
should fail
Logon
GetStatus
• Code implementation
• Rerun all tests – expect pass
Model Driven Development: Taking BDD/ATDD to the Next Level
Logoff
15
17. The Red-Green Cycle
• Imagine a class and its
responsibilities
• Code its API only
• Write and run a simple test – it
should fail
TestLogon
Logon
TestGetStatus
GetStatus
• Code implementation
• Rerun all tests – expect pass
Logoff
• Repeat until feature complete
Model Driven Development: Taking BDD/ATDD to the Next Level
16
18. The Red-Green Cycle
• Imagine a class and its
responsibilities
• Code its API only
• Write and run a simple test – it
should fail
TestLogon
Logon
TestGetStatus
GetStatus
• Code implementation
• Rerun all tests – expect pass
TestLogoff
Logoff
• Repeat until feature complete
Model Driven Development: Taking BDD/ATDD to the Next Level
17
21. Acceptance Test Driven Development
Process,
Activities
Features,
Scenarios
UI Test
Cases
Feature Files
Scenario A
*DD Tool
Scenario B
Scenario C
Test Scripts
Test Objects
Test Tooling
*Unit
System
Under Test
Model Driven Development: Taking BDD/ATDD to the Next Level
20
22. Features and Scenarios
Feature: story
Scenario: instance of a story
Steps:
Given: pre-condition
When: trigger
Then: verification condition
Feature: Logon
In order to use the chat server,
As a validated chat client,
I want to submit my credentials and
establish a session
Scenario: Submit Good
Given A connection to
When I send the Login
Then I should see the
Model Driven Development: Taking BDD/ATDD to the Next Level
Credentials
the server is open
message
"Hi Chatter" message
21
23. Overview
Feature Files
Scenario A
SpecFlow
Scenario B
Scenario C
Test Objects
Nunit
Visual Studio
Components
http://www.specflow.org
• Enter features, scenarios
• Generates skeleton test
code for each step
– Extracts step names
– Can run immediately
• Developer adds step code
• Test run reports
• CI integration, more
Model Driven Development: Taking BDD/ATDD to the Next Level
22
24. 1. Create a
Feature File
• SpecFlow
generates a stub
file
• Edit as necessary
• One feature
• 1:N scenarios
Model Driven Development: Taking BDD/ATDD to the Next Level
23
25. 2. Enter Scenario
Steps
• Given
– Precondition
– Data State
• When
– Trigger
– API call, GUI action
• Then
– Verification
conditions
(expected results)
Model Driven Development: Taking BDD/ATDD to the Next Level
24
26. 3. Inspect Skeleton
Test Code
• SpecFlow parses
feature file
• Generates
skeleton test
code for each
scenario and step
Model Driven Development: Taking BDD/ATDD to the Next Level
25
27. 3. Add Test Case
Code
• User adds more
code as needed
to implement
each test case
Model Driven Development: Taking BDD/ATDD to the Next Level
26
28. 4. Run Test Suite
• The generated
test code can be
run immediately
• Supports the redgreen cycle
Model Driven Development: Taking BDD/ATDD to the Next Level
27
29. SpecFlow
•
BDD for Visual Studio
• Link Feature Files to test code
• Generates partial test code
Feature File contains statements in
Gherkin
–
Keywords + natural language
•
•
Generates test code skeleton from
steps
•
Feature
File
1:N Steps per Scenario
•
Feature
1:N Scenarios per Feature File
Steps may be parameterized
Steps
Rows
•
Variables and variable bindings in Rows
–
Scenarios
–
Test code generation using regular
expression patterns
Integration with TFS, CI, …
Model Driven Development: Taking BDD/ATDD to the Next Level
28
31. What is a Model?
• Focus
• Smaller, simpler
• In IT systems
– Behavior
– Structure
– Content
Model Driven Development: Taking BDD/ATDD to the Next Level
30
33. IT Models
• Notional (design)
• As built (documentation)
• Test Models
– Similar to notional and as-built
– Unique aspects
34. Test Models
• Focus on aspects that
support testing goals
• Behavior
– Sequence and time
• Transformation
– Input, output function
• Verification (oracle)
• Coverage
• Risk Management
Model Driven Development: Taking BDD/ATDD to the Next Level
33
35. Chat Server
• Logon
Chat Server
– Must be first
– Can’t be repeated
Logon
GetStatus
Logoff
• GetStatus
– Only after logon
– Returns number of calls since
logon
• Logoff
– Only after logon
• Typical RFC status codes
Model Driven Development: Taking BDD/ATDD to the Next Level
34
36. Simple Test Model
Logoff
nc = 0
GetStatus
System Under
Test (SUT)
Logon
Logon
Logon
GetStatus
assert (200)
assert (401)
assert (401)
assert (401)
Logoff
nc++
GetStatus
Logoff
assert (200)
Logoff
assert (200)
Model Driven Development: Taking BDD/ATDD to the Next Level
assert (200)
assert (n == nc)
35
37. Chat Server Message Sequence
Login a 1234
Hi Chatter a
Login b secret
Hi Chatter b
List Users
Users: a, b
!!!
Client Endpoint
Client Endpoint
Model Driven Development: Taking BDD/ATDD to the Next Level
Chat Server
36
38. Chat Server Message Sequence
List Users
Users: a, b
Post “Pumpkin Latte #mypics”
Ack a1 “Pumpkin Latte #mypics”
Bcast a1 “Pumpkin Latte #mypics”
Logoff b
Ack Logoff b
!!!
Client Endpoint
Client Endpoint
Model Driven Development: Taking BDD/ATDD to the Next Level
Chat Server
37
39. Spec Explorer Demo
• Model-based testing tool
– Microsoft Research
– Used to test all MSFT APIs
– Robust and stable
– Visual Studio “power tool”
• C# code, not cartoons
• Generates standalone
executable test suite
Model Driven Development: Taking BDD/ATDD to the Next Level
38
40. 1. Create Test
Model Actions
• [Rule] tags a
method for
exploration
• Requires defines
a exploration
precondition –
must be true to
explore
Model Driven Development: Taking BDD/ATDD to the Next Level
39
41. 2. Define Accepting
State(s)
• Tag condition(s)
that terminate
an exploration
Model Driven Development: Taking BDD/ATDD to the Next Level
40
42. 3. Create Cord File
• Set options
• Define machines
– Limits Actions used
in an exploration
• Similar to regular
expression
• Composable
• Very powerful!
Model Driven Development: Taking BDD/ATDD to the Next Level
41
43. 4. Explore the Model
• Select any
explorable machine
– Explore
– Generate test suite
– Run on-the-fly test
Model Driven Development: Taking BDD/ATDD to the Next Level
42
44. 5. Review
Exploration Graph
•
Shows action/event
sequence
•
Ellipse: model state
computed by model
program machine
•
Diamond: observation
“event” – get something
from SUT (input to the
model program)
•
Right click to inspect
any item
Model Driven Development: Taking BDD/ATDD to the Next Level
43
45. 6. Define machine(s)
to generate test
suites
• Set options to
generate test
code
• Use any machine
or composite
machine
Model Driven Development: Taking BDD/ATDD to the Next Level
44
46. 7. Explore the Test
Suite machine
• Each root to leaf
path becomes a
test suite
Model Driven Development: Taking BDD/ATDD to the Next Level
45
47. 8. Generate Test Code
• Explores the test
machine
• Writes out test
code using VS
testing
framework
Model Driven Development: Taking BDD/ATDD to the Next Level
46
48. 8. Generate Test Code
• Not intended to
be maintained
• Re-generate any
time
• Add helper code in
the model to
extend/customize
Model Driven Development: Taking BDD/ATDD to the Next Level
47
49. 9. Run the Test Suite
•
Uses built-in Visual
Studio test runner
•
Adapters
– Start/stop chat clients
and chat server
– Other SUT
setup/cleanup
•
Model Driven Development: Taking BDD/ATDD to the Next Level
Test code sends
messages, checks
replies
48
50. Model State
(test case inputs)
10. Review Test
Results
• Detailed run log
–
–
–
–
Event Handler
(pass/fail)
Requirement
Trace
VS test runner
States
Events
Requirements
• Add any other
programmable
function
Model Driven Development: Taking BDD/ATDD to the Next Level
49
51. Spec Explorer
Exploration
• Interactive model development
• Generate test suites
•
Machines
Cord
Declarations
Model
Program
Options
Machines define composable
Action sequences
•
Options control exploration and
data generation
•
Actions are explored
– Rules define exploration conditions
Model Data
Structures
Rules
Actions
Events
Adapters
– Events define expected SUT
response
•
Data structures support exploration
•
Adapters abstract, control, and
observe any interface, any platform
Model Driven Development: Taking BDD/ATDD to the Next Level
50
52. MBT Driven Development
Requirements, Stories, Scenarios...
Explore
Ambiguous, missing,
contradictory, incorrect,
redundant, incomplete
Test Model
Run
Generate
Model error, omission
Test Suite
Inputs
(Test Sequences)
Control
Missing,
incorrect
Coverage
Requirements
Model
Evaluate
Expected Outputs
(Test Oracle)
SUT
Observe
Model Driven Development: Taking BDD/ATDD to the Next Level
Bug
51
54. Model-based Testing in the Real World
• Windows protocol compliance (Web services, RPC, DCOM)
• Total effort: 250 person years (mostly junior SDETs)
• Saved 50 person years with model-based testing
Model-based Testing
1.4 Days/Requirement
42% Less
Time Per
Requirement
Traditional Testing
2.4 Days/Requirement
Requirements Study
Modeling
Test Coding
Adapter Coding
Model Driven Development: Taking
BDD/ATDD to the Next Level
Test Execution
53
55. Scope of Test Asset Maintenance
Model-based Testing
Behavior Driven Development
Model
How much MBTgenerated test code
must be maintained?
Test Code
None!
Adapters
Model Driven Development: Taking BDD/ATDD to the Next Level
54
56. More Models, Less Tests
• More Better Testing
• Icing the BackBlob
– Rule-based
– Maintain model
– Systematic and deep
exploration
– Regenerate test suites
– Generate 1000s of
scenarios in seconds
– Models grow much
more slowly than test
code
Model Driven Development: Taking BDD/ATDD to the Next Level
55
59. Learn More - MBT
•
UCCAT (User Conference on Advanced Automated Testing) Presentations
http://ucaat.etsi.org/2013/program_conf.html
•
2012 Model-based Testing User Survey
http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf
•
Open Source MBT tools
http://www.robertvbinder.com/open-source-tools-for-model-based-testing/
•
Overview of MBT, list of commercial MBT tools
http://www.slideshare.net/robertvbinder/model-basedtestingignite
•
Spec Explorer
http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745
http://msdn.microsoft.com/library/ee620411.aspx
•
Microsoft Open Protocol Project
http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#!
Model Driven Development: Taking
BDD/ATDD to the Next Level
58
60. Learn More: BDD, etc.
•
SpecFlow
http://www.specflow.org/
•
Using SpecFlow with WatiN
http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
•
SpecFlow Quick Start
http://www.youtube.com/watch?v=SVwBz-ZTliw
•
Writing Cucumber Feature Files
https://github.com/cucumber/cucumber/wiki
•
Using Lava Lamps to indicate Build Status
https://wiki.jenkins-ci.org/display/JENKINS/Hudson+Build+Status+Lava+Lamps
Model Driven Development: Taking
BDD/ATDD to the Next Level
59