In December 2008 when How We Test Software at Microsoft was first published, the software community appreciated the insight into many testing activities and processes popular at Microsoft. Six and a half years later, many companies—including Microsoft—have evolved and changed in a variety of ways, and now much of the book is outdated or obsolete. New products, new ideas, and new strategies for releasing software have emerged. Alan Page explores Microsoft’s current approaches to software testing and quality. He digs into new practices, describes changing roles, rants about long-lived ideas kicked to the curb in the past seven years―and might even share a few tidbits not fit for print and wide-scale distribution. To give organizations food for thought and ideas for growth, Alan reveals what’s new in quality approaches, developer to tester ratios, agile practices, tools, tester responsibilities—and lessons he’s learned along the way.

1. !
!
K1#
Keynote!
6/24/2015! !
8:30!AM!
!
!
!
!
“How#We#NOW#Test#Software#at#Microsoft”##
Presented#by:#
Alan#Page#
Microsoft#
#
#
#
#
#
Brought#to#you#by:#
#
#
#
#
#
#
340!Corporate!Way,!Suite!300,!Orange!Park,!FL!32073!
888D268D8770!E!904D278D0524!E!sqeinfo@sqe.com!E!www.sqe.com!
!
!
!

2. !
!
!!!!! !
!!!!
Alan Page is a principal SDET—a fancy name for tester—on the Xbox console team at
Microsoft. Edging up on twenty years in software testing, Alan has previously worked on
a variety of Microsoft products including Windows, Windows CE, Internet Explorer, and
Office Lync. He spent some time as Microsoft’s director of test excellence where he
developed and ran technical training programs for testers throughout the company. Alan
was the lead author of How We Test Software at Microsoft and contributed chapters on
large-scale test automation to Beautiful Testing and Experiences of Test Automation:
Case Studies of Software Test Automation. You can follow Alan on his blog or on
Twitter @alanpage..
Alan Page
Microsoft

3. 5/18/15%
1%
NOW%
Can$we$improve$the$way$we$make$
so2ware?$

4. 5/18/15%
2%
Can$we$be$more$efficient?$

5. 5/18/15%
3%
2006

6. 5/18/15%
4%

7. 5/18/15%
5%
HWTSAM$=$The$Beginning$

8. 5/18/15%
6%

9. 5/18/15%
7%
Nobody’s$Perfect$
1) In the intro, I state that Ken approached me in the fall of 2007. On reflection, it was actually
late fall 2006. I began writing the first chapter in January 2007 (chapter 4 to be exact).
Minor issue, but an Author Bug nonetheless.
2) The table on p. 19 is missing a line between USA (California) and Hyderabad (Production
Bug)
3) On page 95, where the header says. “3BV”, it should say “3BC”
4) There's a bit of weirdness with the code coverage data on page 119. Bj explains it in detail
on his blog here. (Author bug)
5) On page 120, the explanation refers to variables names myString and myCharacter – but we
use different variable names in the code (s and c). (Author Bug).
6) On page 123, the title of figure 5-6 is incorrect (the function is SimpleGetNT5ClientVersion).
7) On page 125, the text refers to the Int.Parse() function, but it doesn’t exist in the code
fragment (oops - Author Bug)
8) On page 202, the text reads “Total fixed found / total bugs fixed”, but it should read “Total
bugs fixed / total bugs found”
9) On page 221, the “Positively False” sidebar is not a sidebar, and in fact the box around it
encompasses the additional points (Supported platforms, Complexity, and Other factors)
and section (We don’t have time to automate this) that aren’t part of that sidebar (Author
Bug)
10) On page 226, the sample using InvokePattern could use some more documentation (Author
Oversight)
11) In the code sample on the bottom of page 235, I have mismatched quotes in two of the
lines. (Author Bug).

10. 5/18/15%
8%

11. 5/18/15%
9%
People$and$Processes$

12. 5/18/15%
10%
Senior%Vice%President%
GM%or%VP%of%Dev%
Dev%Managers% Architects%
GM%or%VP%of%Test%
Test%Managers% Test%Architects%
GM%or%VP%of%Program%
Mgmt%
PM%Managers% Etc.%
Vice%President%
Engineering%Manager% Engineering%Manager%
Engineer% Engineer% Engineer% Engineer% Engineer% Engineer% Engineer% Engineer% Engineer%
Engineering%Manager%

13. 5/18/15%
11%
2008
Processes%iteraIve,%yet%slow%
Customer%behavior,%feedback%measured%postSrelease%
%
2015
Most%teams%use%agileSish%processes%
Customer%data%is%cri$cal'
2008
More%than%90,000%Employees%World%Wide%
More%than%38,000%SoTware%Engineers%%
Nearly%%10,000%Testers%
Test%to%Developer%RaIo%–%(almost)%1:1%
2015
120,000%Employees%World%Wide%
40,000%SoTware%Engineers%%
Testers?%

14. 5/18/15%
12%
I$Know$What$You’re$Thinking…$

15. 5/18/15%
13%

16. 5/18/15%
14%
%The%implementaIon%described%
above%is%risky%and%invites%failure.%

17. 5/18/15%
15%

18. 5/18/15%
16%

19. 5/18/15%
17%
FlightingRIPre CheckinDev SessionRealtime

20. 5/18/15%
18%

21. 5/18/15%
19%

22. 5/18/15%
20%
Who$“owns”$Quality?$
What'Is'Quality?$

23. 5/18/15%
21%
“Quality*is*value*to*some*person”%
S%Jerry%Weinberg%
Who%“owns”%creaIng%value%for%the%
customer?%

24. 5/18/15%
22%
Dev%“stuff”%

25. 5/18/15%
23%
Dev%“stuff”%
Project%
Management%
“stuff%
Dev%“stuff”%
Project%
Management%
“stuff%
The%
“Remainder”%

26. 5/18/15%
24%
Data%Science%
Exploratory$TesPng$
“The%Remainder”%

27. 5/18/15%
25%
Diversity$is$Important$CriPcal$
Specializing%Generalists%
Generalizing%Specialists%
Everyone%doesn’t%do%everything…%
…but%the%team%contains%“all%of%the%skills”%

28. 5/18/15%
26%
By%staffing%a%team%with$people$who$have$an$
area$of$experPse,$but$can$do$anything,$you$
can$maximize$the$value$of$each$delivery$
cycle%
Nothing$is$easy…$

29. 5/18/15%
27%
“That’s the way we’ve always done it.”
“That won’t work for us.”
“We tried that once, and it didn’t work.”
Things%I%Hear…%a%Lot%
“The old way won’t work here because….”
“This can work for us, because…”
“This will work for us, because…”
Things%I%Say…%a%Lot%

30. 5/18/15%
28%
Before$I$Go…$

31. 5/18/15%
29%

32. 5/18/15%
30%
You’re$Not$As$Different$As$You$Think$

33. 5/18/15%
31%
Thank$You$
$
hVp://angryweasel.com$
hVp://angryweasel.com/ABTesPng$
$