Microsoft developed a large-scale model-based testing program to test 222 technical protocols over many years. They analyzed thousands of pages of documentation to identify requirements and convert them to testable assertions. Approximately 69% of tests were developed using model-based techniques, finding them 42% more efficient than traditional testing. Key factors in their success included tool support for modeling in common languages, extensive training for testers, and a review process to provide guidance. The large size and scope of the program provided empirical evidence of the benefits of model-based testing.

1. Microsoft’s Protocol DocumentationMicrosoft’s Protocol Documentation
Program:Program:
A Success Story for Model-Based TestingA Success Story for Model-Based Testing
Wolfgang Grieskamp, Software Architect
Windows Interoperability Engineering Team, Microsoft Corporation
Keynote given at A-MOST 2011, March 2011, Berlin, Germany.

2. Part 1: BackgroundPart 1: Background
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 2

3. MicrosoftMicrosoft
OpenOpen
ProtocolsProtocols
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
http://www.microsoft.com/protocols
9/4/2010 3

4. Technical DocumentTesting ProgramTechnical DocumentTesting Program
of Windows (as of 03/09)of Windows (as of 03/09)
 222 protocols/technical documents tested
 22,847 pages studied and converted into
requirements
 36,875 testable requirements identified and
converted into test assertions
 69% tested using MBT
 31% tested using traditional test automation
 66,962 person days (250+ years)
 Hyderabad: 250 test engineers
 Beijing: 100 test engineers
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 4

5. Formalization
Technical Document
Test Suite
Test Suite
English
Statements
(Requirements)
Assertions
Checked against Windows
(Accuracy)
• Validate
consistency
• Simulate
implementation
process
Test-Driven Document QualityTest-Driven Document Quality
AssuranceAssurance
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 5

6. Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Process: Managing a Project with 350 Test Engineers
Review after every phase
•High-profile reviewer team consisting of industry experts and Microsoft
senior engineers, build along the model of a program committee
•Review based on process report and test suite artifacts
•Verdict accept, conditional accept, reject
9/4/2010 6

7. RequirementsTrackingRequirementsTracking
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Technical
Document
Requirements
Spec
Model or Test
Design
Test Suite
Logs
Network
Captures
 Statements identified in
document and tracked
with requirement ID
 IDs referenced from
model or test assertion
 IDs logged at test
execution time
 IDs send to network as
beacon packages for
network monitor
capturing
manual
automated
9/4/2010 7

8. Typical Requirements Specification
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 8

9. Part 2: Model-BasedTest SuitePart 2: Model-BasedTest Suite
DevelopmentDevelopment
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 9

10. Model-Based Document AnalysisModel-Based Document Analysis
Model
Model
Implementation
Implementation
Expected Outputs
(Test Oracle)
Expected Outputs
(Test Oracle)Inputs
(Test Sequences)
Inputs
(Test Sequences)
Control
Document
Document
Generate
Observe
Feedback
Feedback
Author
Issue
Feedback
Verdict
Feedback
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 10

11. Process DrilldownProcess Drilldown
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 11

12. Gathering RequirementsGathering Requirements
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 12

13. Developing and Implementing AdaptersDeveloping and Implementing Adapters
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 13

14. Adapter Interface
Adapter Interface
TC
TC
Test Cases
(traditional or MBT)
TC
TC
TC
TC
TC
TC
TC
TC
TC
TC
Adapter Implementation
Adapter Implementation
SUT
(System under test)
• Abstracts SUT
functionality
• Contract between teams
• Test case team
• Adapter team
• Abstracts SUT
functionality
• Contract between teams
• Test case team
• Adapter team
• Pluggable
• Different server setups
• Different transports
• Pluggable
• Different server setups
• Different transports
Test Adapters
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 14

15. Develop Model and GenerateTestsDevelop Model and GenerateTests
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
Potential
dependency
on data types
9/4/2010 15

16. Spec Explorer 2010TechnologySpec Explorer 2010Technology
BreakdownBreakdown Model programs
 Guarded state update rules
 Rich object-oriented model state (collections, object graphs)
 Language agnostic (Based on .Net intermediate language
interpretation)
 Trace patterns
 Regular style language to represent scenarios
 Slicing of model program by composition
 Symbolic state exploration and test generation
 Expands parameters using combinatorial interaction testing
 Extracts a finite interface automaton (IA) from composed model
 Traverses IA to generate standalone test code –or-
 Runs on-the-fly tests from IA
 Integrated intoVisual Studio 2010
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 16

17. Spec Explorer 2010 Look & FeelSpec Explorer 2010 Look & Feel
Explore
Analyze
Generate
Execute
C# ModelC# Model
(or other .Net(or other .Net
Language)Language)
Model GraphModel Graph
Test SuiteTest Suite
VSTT ResultVSTT Result
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 17

18. Spec ExplorerWalkthrough: Chat ExampleSpec ExplorerWalkthrough: Chat Example
 Users can
 Enter the session
 Exit the session
 List all session users
 Broadcast a message
 Received by all session users
18
Chat Room Server
User1
SUT
User2 User3
What happens if two users broadcast a message at virtually the same
time?

19. Chat RequirementsChat Requirements
19
 R1: User MUST receive response for logon request
 R2: User MUST receive response for logoff request
 R3: User MUST receive response for list request
 R4: List response MUST contain the list of logged-on users if
successful
 R5: All logged-on users MUST receive broadcast message
 R6: Messages from one sender MUST be received in order

20. Chat tracesChat traces
Is the trace correct?
(Assume two users (user1, user2) are logged on)
20
T1
Broadcast(user1,”1”)
Broadcast(user2,”2”)
BroadcastAck(user2,”1”)
BroadcastAck(user1,”2”)
BroadcastAck(user1,”1”)
BroadcastAck(user2,”2”)
T2
Broadcast(user1,”1”)
Broadcast(user2,”2”)
BroadcastAck(user1,”2”)
BroadcastAck(user1,”1”)
BroadcastAck(user2,”1”)
BroadcastAck(user2,”2”)
T3
Broadcast(user1,”1”)
Broadcast(user2,”2”)
BroadcastAck(user1,”2”)
BroadcastAck(user2,”2”)
BroadcastAck(user1,”1”)
BroadcastAck(user2,”1”)
If each user sends one message, any receiving order is correct!

21. Chat tracesChat traces
Is the trace correct?
(Assume two users (user1, user2) are logged on)
21
T4
Broadcast(user1,”1a”)
Broadcast(user1,”1b”)
BroadcastAck(user2,”1a”)
BroadcastAck(user1,”1a”)
BroadcastAck(user1,”1b”)
BroadcastAck(user2,”1b”)
T5
Broadcast(user1,”1a”)
Broadcast(user1,”1b”)
BroadcastAck(user1,”1a”)
BroadcastAck(user2,”1b”)
BroadcastAck(user1,”1b”)
BroadcastAck(user2,”1a”)
Local order consistency:
messages sent by one user must be received in order

22. 22

23. 23

24. Spec Explorer’s Conformance NotionSpec Explorer’s Conformance Notion
Alternating simulation:
 SUT must “simulate” all stimuli of model
 Model must “simulate” all responses of SUT (responses are buffered)
24
Broadcast(1,”1”)
Broadcast(1,”2”)
Model SUT
Broadcast(1,”1”)
Broadcast(1,”2”)
BroadcastAck(1,”1”
)
BroadcastAck(2,”1”)
BroadcastAck(1,”1”
)
BroadcastAck(2,”1”)
BroadcastAck(2,”1”)
BroadcastAck(1,”2”
)

25. Spec Explorer’s Approach to State ExplosionSpec Explorer’s Approach to State Explosion
Slicing techniques:
 Parameter selection (Constraints, Pairwise, etc.)
 State filtering
 Trace patterns
 Requirement coverage
Slicing requires human intervention
Smart slicing is an art
Spec Explorer tailored for slicing design
25

26. Where aTrace can end: Accepting StateWhere aTrace can end: Accepting State
ConditionCondition
 Is the following a valid word?
Micr
 Does the following trace represents a useful test?
Broadcast(1,”1”); Broadcast(1,”1”)
 An Accepting state condition characterizes those
states in which a trace can end
 Used to ensure that a trace does not stop at arbitrary points
 Used to ensure that a test leaves the system in a good state
 Accepting state condition for Chat:
 All messages have been delivered to recipients
26

27. Demo: Chat ModelDemo: Chat Model
What the demo shows
 OO model state
 Using helper methods in models
 Modeling with events
 Slicing

28. Demo: Server Message Block ProtocolDemo: Server Message Block Protocol
v2v2
What the demo shows:
Modeling a key Microsoft product
Using C# to model state transition systems
Using trace patterns to slice test purposes
from models
Generating and executing tests
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 28

29. Part 3: EvaluationPart 3: Evaluation
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 33

30. Comparison MBT vsTraditionalComparison MBT vsTraditional
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
• In % of total effort per
requirement, normalizing
individual vendor
performance
• Vendor 2 modeled 85% of
all test suites, performing
relatively much better than
Vendor 1
9/4/2010 35

31. Remarks on ComparisonRemarks on Comparison
 Numbers backed up by correlation analysis
conducted by empirics expert
 Engineers applying MBT were lay folks, most
of them freshly hired from college
 Few power modelers achieve much higher
efficiency rates
 Though model authoring takes only a smaller
part of overall effort per requirement,
modeling approach seems to determine
efficiency
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 36

32. Factors for SuccessFactors for Success
 Tool support
 Modeling in mainstream languages and
environments
 Visualization of exploration and actual generated
test suite (no magic as in online testing)
 Dealing with state explosion by scenario slicing
 Training and Guidance
 One week modeling classes for every new hire
 Size of project generated critical mass for
community help
 Reviewing process added additional guidance
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 37

33. ConclusionsConclusions
 Biggest application of behavioral model-
based testing known so far
 Probe big enough for relevant empiric results,
proving efficiency gain of 42%
 Will lead to more momentum on modeling in
general inside of Microsoft
Model-based testing rocks!
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 38

34. Advertisement…Advertisement…
 Microsoft offers Spec Explorer 2010 aVisual
Studio Powertool via
 Licensing allows for commercial use
 Academic institutions can join MSDN
Academic Alliance for free access toVisual
Studio 2010
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 39

35. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should
not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 40