Microsoft's Protocol Documentation Program Success Story Model-Based Testing

Microsoft’s Protocol DocumentationMicrosoft’s Protocol Documentation
Program:Program:
A Success Story for Model-Based TestingA Success Story for Model-Based Testing
Wolfgang Grieskamp, Software Architect
Windows Interoperability Engineering Team, Microsoft Corporation
Keynote given at A-MOST 2011, March 2011, Berlin, Germany.

Part 1: BackgroundPart 1: Background
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 2

MicrosoftMicrosoft
OpenOpen
ProtocolsProtocols
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
http://www.microsoft.com/protocols
9/4/2010 3

Technical DocumentTesting ProgramTechnical DocumentTesting Program
of Windows (as of 03/09)of Windows (as of 03/09)
 222 protocols/technical documents tested
 22,847 pages studied and converted into
requirements
 36,875 testable requirements identified and
converted into test assertions
 69% tested using MBT
 31% tested using traditional test automation
 66,962 person days (250+ years)
 Hyderabad: 250 test engineers
 Beijing: 100 test engineers

Formalization
Technical Document
Test Suite
Test Suite
English
Statements
(Requirements)
Assertions
Checked against Windows
(Accuracy)
• Validate
consistency
• Simulate
implementation
process
Test-Driven Document QualityTest-Driven Document Quality
AssuranceAssurance

Process: Managing a Project with 350 Test Engineers
Review after every phase
•High-profile reviewer team consisting of industry experts and Microsoft
senior engineers, build along the model of a program committee
•Review based on process report and test suite artifacts
•Verdict accept, conditional accept, reject
9/4/2010 6

RequirementsTrackingRequirementsTracking
Technical
Document
Requirements
Spec
Model or Test
Design
Test Suite
Logs
Network
Captures
 Statements identified in
document and tracked
with requirement ID
 IDs referenced from
model or test assertion
 IDs logged at test
execution time
 IDs send to network as
beacon packages for
network monitor
capturing
manual
automated
9/4/2010 7

Typical Requirements Specification

Part 2: Model-BasedTest SuitePart 2: Model-BasedTest Suite
DevelopmentDevelopment

Model-Based Document AnalysisModel-Based Document Analysis
Model
Model
Implementation
Implementation
Expected Outputs
(Test Oracle)
Expected Outputs
(Test Oracle)Inputs
(Test Sequences)
Inputs
(Test Sequences)
Control
Document
Document
Generate
Observe
Feedback
Feedback
Author
Issue
Feedback
Verdict
Feedback

Process DrilldownProcess Drilldown
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 11

Gathering RequirementsGathering Requirements
Gather Requirements
Model & Explore
Run Tests
9/4/2010 12

Developing and Implementing AdaptersDeveloping and Implementing Adapters
Gather Requirements
Model & Explore
Run Tests
9/4/2010 13

Adapter Interface
Adapter Interface
TC
TC
Test Cases
(traditional or MBT)
TC
TC
TC
TC
TC
TC
TC
TC
TC
TC
Adapter Implementation
Adapter Implementation
SUT
(System under test)
• Abstracts SUT
functionality
• Contract between teams
• Test case team
• Adapter team
• Abstracts SUT
functionality
• Contract between teams
• Test case team
• Adapter team
• Pluggable
• Different server setups
• Different transports
• Pluggable
• Different server setups
• Different transports
Test Adapters

Develop Model and GenerateTestsDevelop Model and GenerateTests
Gather Requirements
Model & Explore
Run Tests
Potential
dependency
on data types
9/4/2010 15

Spec Explorer 2010TechnologySpec Explorer 2010Technology
BreakdownBreakdown Model programs
 Guarded state update rules
 Rich object-oriented model state (collections, object graphs)
 Language agnostic (Based on .Net intermediate language
interpretation)
 Trace patterns
 Regular style language to represent scenarios
 Slicing of model program by composition
 Symbolic state exploration and test generation
 Expands parameters using combinatorial interaction testing
 Extracts a finite interface automaton (IA) from composed model
 Traverses IA to generate standalone test code –or-
 Runs on-the-fly tests from IA
 Integrated intoVisual Studio 2010

Spec Explorer 2010 Look & FeelSpec Explorer 2010 Look & Feel
Explore
Analyze
Generate
Execute
C# ModelC# Model
(or other .Net(or other .Net
Language)Language)
Model GraphModel Graph
Test SuiteTest Suite
VSTT ResultVSTT Result

Spec ExplorerWalkthrough: Chat ExampleSpec ExplorerWalkthrough: Chat Example
 Users can
 Enter the session
 Exit the session
 List all session users
 Broadcast a message
 Received by all session users
18
Chat Room Server
User1
SUT
User2 User3
What happens if two users broadcast a message at virtually the same
time?

Chat RequirementsChat Requirements
19
 R1: User MUST receive response for logon request
 R2: User MUST receive response for logoff request
 R3: User MUST receive response for list request
 R4: List response MUST contain the list of logged-on users if
successful
 R5: All logged-on users MUST receive broadcast message
 R6: Messages from one sender MUST be received in order

Chat tracesChat traces
Is the trace correct?
(Assume two users (user1, user2) are logged on)
20
T1
Broadcast(user1,”1”)
BroadcastAck(user2,”1”)
T2
T3
If each user sends one message, any receiving order is correct!

Chat tracesChat traces
Is the trace correct?
(Assume two users (user1, user2) are logged on)
21
T4
Broadcast(user1,”1a”)
Broadcast(user1,”1b”)
BroadcastAck(user2,”1a”)
BroadcastAck(user1,”1b”)
T5
Broadcast(user1,”1a”)
Broadcast(user1,”1b”)
Local order consistency:
messages sent by one user must be received in order

Spec Explorer’s Conformance NotionSpec Explorer’s Conformance Notion
Alternating simulation:
 SUT must “simulate” all stimuli of model
 Model must “simulate” all responses of SUT (responses are buffered)
24
Broadcast(1,”1”)
Model SUT
BroadcastAck(1,”1”
)
BroadcastAck(2,”1”)
)
)

Spec Explorer’s Approach to State ExplosionSpec Explorer’s Approach to State Explosion
Slicing techniques:
 Parameter selection (Constraints, Pairwise, etc.)
 State filtering
 Trace patterns
 Requirement coverage
Slicing requires human intervention
Smart slicing is an art
Spec Explorer tailored for slicing design
25

Where aTrace can end: Accepting StateWhere aTrace can end: Accepting State
ConditionCondition
 Is the following a valid word?
Micr
 Does the following trace represents a useful test?
Broadcast(1,”1”); Broadcast(1,”1”)
 An Accepting state condition characterizes those
states in which a trace can end
 Used to ensure that a trace does not stop at arbitrary points
 Used to ensure that a test leaves the system in a good state
 Accepting state condition for Chat:
 All messages have been delivered to recipients
26

Demo: Chat ModelDemo: Chat Model
What the demo shows
 OO model state
 Using helper methods in models
 Modeling with events
 Slicing

Demo: Server Message Block ProtocolDemo: Server Message Block Protocol
v2v2
What the demo shows:
Modeling a key Microsoft product
Using C# to model state transition systems
Using trace patterns to slice test purposes
from models
Generating and executing tests

Part 3: EvaluationPart 3: Evaluation

Comparison MBT vsTraditionalComparison MBT vsTraditional
• In % of total effort per
requirement, normalizing
individual vendor
performance
• Vendor 2 modeled 85% of
all test suites, performing
relatively much better than
Vendor 1
9/4/2010 35

Remarks on ComparisonRemarks on Comparison
 Numbers backed up by correlation analysis
conducted by empirics expert
 Engineers applying MBT were lay folks, most
of them freshly hired from college
 Few power modelers achieve much higher
efficiency rates
 Though model authoring takes only a smaller
part of overall effort per requirement,
modeling approach seems to determine
efficiency

Factors for SuccessFactors for Success
 Tool support
 Modeling in mainstream languages and
environments
 Visualization of exploration and actual generated
test suite (no magic as in online testing)
 Dealing with state explosion by scenario slicing
 Training and Guidance
 One week modeling classes for every new hire
 Size of project generated critical mass for
community help
 Reviewing process added additional guidance

ConclusionsConclusions
 Biggest application of behavioral model-
based testing known so far
 Probe big enough for relevant empiric results,
proving efficiency gain of 42%
 Will lead to more momentum on modeling in
general inside of Microsoft
Model-based testing rocks!

Advertisement…Advertisement…
 Microsoft offers Spec Explorer 2010 aVisual
Studio Powertool via
 Licensing allows for commercial use
 Academic institutions can join MSDN
Academic Alliance for free access toVisual
Studio 2010

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should
not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft's Protocol Documentation Program Success Story Model-Based Testing

Recommended

Recommended

More Related Content

Similar to Microsoft's Protocol Documentation Program Success Story Model-Based Testing

Similar to Microsoft's Protocol Documentation Program Success Story Model-Based Testing (20)

Recently uploaded

Recently uploaded (20)

Microsoft's Protocol Documentation Program Success Story Model-Based Testing