Testers frequently say that they are the voice of the customer or the customer advocate for their organization’s products. In some situations this can be a helpful mindset, but no matter how hard he tries, a software tester is not the customer. In fact, there is no one better suited to evaluate customer experience than the actual customer of your software. However, getting actionable feedback from customers can be time-consuming, difficult, and often too late to have any meaningful impact on the product. Alan Page shares his thoughts and a number of examples of how to get customer feedback quickly, how to make that feedback actionable, and how to use customer data to drive better software development and testing on any team—and for any product. In this fast-paced session of information and fun, Alan discusses product instrumentation, analysis techniques, reporting, A/B testing, and many other facets of customer feedback.

1. !
!
W10$
Session!
6/24/2015! !
1:30!PM!
!
!
!
!
“Inside$the$Mind$of$the$21st
$Century$Customer”$$
Presented$by:$
Alan$Page$
Microsoft$
$
$
$
$
$
Brought$to$you$by:$
$
$
$
$
$
$
340!Corporate!Way,!Suite!300,!Orange!Park,!FL!32073!
888C268C8770!D!904C278C0524!D!sqeinfo@sqe.com!D!www.sqe.com!
!
!
!

2. !
!
!!!!! !
!!!!
Alan Page is a principal SDET—a fancy name for tester—on the Xbox console team at
Microsoft. Edging up on twenty years in software testing, Alan has previously worked on
a variety of Microsoft products including Windows, Windows CE, Internet Explorer, and
Office Lync. He spent some time as Microsoft’s director of test excellence where he
developed and ran technical training programs for testers throughout the company. Alan
was the lead author of How We Test Software at Microsoft and contributed chapters on
large-scale test automation to Beautiful Testing and Experiences of Test Automation:
Case Studies of Software Test Automation. You can follow Alan on his blog or on
Twitter @alanpage..
Alan Page
Microsoft

3. 6/8/15&
1&
Inside the Mind of the 21st
Century Customer
Alan&Page&

4. 6/8/15&
2&

5. 6/8/15&
3&
FlightingRIPre CheckinDev SessionRealtime

6. 6/8/15&
4&
Understanding+the+“Center”+of+your+data
• Mean:&The&average&value&
• Outliers&can&throw&it&off&
• Usage:&Hard&core&gamers&
• CompleFon&Fme:&Failures&may&be&very&quick&
• Consider&using&percenFles&instead.&
• Median:&The&central&value&
• Cannot&be&used&to&reason&about&the&magnitude&of&the&group.&
• Can&mask&outliers.&
• Mode:&The&most&commonly&occurring&value&
• Best&to&use&when&values&are&not&numeric&

7. 6/8/15&
5&
Types+of+Measures
“It$is$a$capital$mistake$to$theorize$before$one$has$data.”$$
–$Sherlock$Holmes,&A&Study&in&ScarleQ&(Arthur&Conan&Doyle)&
&
Types+of+Measurement
&
&
Customer)
problem)
descrip0on)
&
happy)
outcome)
descrip0on)
funcFonal&
error&free&
%&success&runs&
#&Fme&between&runs&
%&Fme&lost&due&to&errors&
performant&
speed&vs&baseline&
resources&uFlizaFon&
consistency&
nonXfuncFonal&
usable&
#&Fme&to&discover&
#&Fme&to&complete&it&
%&users&abandoning&
useful&
scenario&adopFon&
Repeat&users&
Session&length&
Scenario$exposure:$

8. 6/8/15&
6&
Measurements+Priori:es
Outcome&Focused&
Scenario&
&
ERROR&FREE&
PERFORMANT&
USEFUL&
USABLE&
Start&Menu&helps&me&
find&the&apps&I&need&
&
ERROR&FREE&
PERFORMANT&
USEFUL&
USABLE&
The&user&experience&
is&opFmized&for&a&
variety&of&networks&
ERROR&FREE&
PERFORMANT&
USEFUL&
USABLE&
Users&get&the&same&
look&and&feel&for&cloud&
apps&as&for&naFve&
ERROR&FREE&
PERFORMANT&
USEFUL&
USABLE&
Measurement$types$will$rank$up$with$different$prioriBes$depending$on$scenario.$
Not$every$scenario$will$need$every$type$of$measurement.$
User+Percep:on+Session+Analysis
Goal:&Detect&and&understand&user’s&bad&experience&and&root&causes&
CollecFon&and&CombinaFon&
of&Logs&
VisualizaFon&of&Session&Logs& Log&Analysis&Scripts&
Latency&
Breakdown&

9. 6/8/15&
7&
EvaluaFng&a&Scenario&
Ease&of&use&
Responsiveness&
Usefulness&
Visual&Appeal&
Example:
Showcase (marketing speak)
Scenario (project manager speak)
End-to-End Test (something actionable)
(Optional) Variations
Xbox Knows You
Better Identity
Step in front of the console, get recognized and see my curated content
Light/Dark settings
Child/Adult/Gender/Height/Size/Apparel
Curated content

10. 6/8/15&
8&
Ease&of&Use&
• Am&I&able&to&complete&the&scenario?&
• Is&it&complicated?&Do&I&need&to&perform&extraordinary&
steps&to&get&what&I&need&done?&
• Are&there&glitches&in&the&system&that&make&it&difficult?&
• Was&it&hard&to&find&how&to&execute&this&scenario?&Are&
the&features&hidden?&
• Is&the&experience&consistent?&
Responsiveness&
• Did I feel like the scenario was fast and fluid?
&
• At any point did I feel like I had to wait a
frustrating amount of time?
• Was I effectively distracted while waiting for an
action to take place? (ex. Movie or animation
while I wait)

11. 6/8/15&
9&
Usefulness&&
• Would you use this scenario yourself to
accomplish this specific outcome?
• Does this scenario meet a need for our
consumer?
• Does this let me do something that I want to do?
Visual&Appeal&
• Is the experience exciting to see and hear?
• Is the UI polished?
• Does the UX make the experience
enjoyable?

12. 6/8/15&
10&
Evaluation Scale
5 & Love It! – You love it so much you’d shout it out at the top of
your lungs through a bullhorn from every rooftop you encounter &
4& Like It – You like it a lot and might mention it during a lull in
conversation at a dinner party&
3& Meh – You can live with it and neither like it or hate it. It’s
nothing special. There are some improvements that can be
made&
2& Don’t Like It – The experience leaves a bad taste in your
mouth. You’d use it if you really had to otherwise, you’d stay
away from it.&
1& Hate it! – You hate it so much that you would only use it if you
were under a Hogwarts compulsion spell&
0& Not Implemented&
Scenario) Love)It) Like)It) Meh) Don't)Like) Hate)
Do&this& 43& 10& 1& 1& 1&
Do&that& 3& 2& 9& 14& 20&
Do&the&other&thing& 40& 14& 5& 0& 1&
Do&something&else& 20& 3& 5& 7& 15&

13. 6/8/15&
11&
Experimenta0on)
Controlled+“A/B”+Experiments+
Concept:&
• Randomly&split&traffic&
between&two&(or&more)&
versions&
• A&(Control)&
• B&(Treatment)&
• Collect&metrics&of&
interest&
• Analyze&&&
100%
Users
50%
Users
50%
Users
Control:
Existing System
Treatment:
Existing System
with Feature X
Users interactions instrumented,
analyzed & compared
Analyze at the end of the
experiment

14. 6/8/15&
12&
Let’s&play&a&game&
• Three&choices&are:&
• A&wins&&&
• B&wins&
• A&and&B&are&approximately&the&same&&
&
(*)&results&based&on&staFsFcally&significant&data&
&
MSN+Real+Estate
“Find&a&house”&widget&variaFons&
Revenue&to&Microsom&generated&every&Fme&a&user&clicks&
search/find&buQon&
• Raise&your&lem&hand&if&you&think&A&Wins&
• Raise&your&right&hand&if&you&think&B&Wins&
• Don’t&raise&your&hand&if&you&think&they’re&about&the&same&
A& B&

15. 6/8/15&
13&
MSN+Real+Estate
• If&&you&did&not&raise&a&hand,&please&sit&down&
• If&you&raised&your&right&hand,&please&sit&down&
• A&was&8.5%&beQer&
Office+Online
Test&new&design&for&Office&Online&homepage&&
A&
Clicks&on&revenue&generaFng&
links&(red&below)&
• Raise&your&lem&hand&if&you&think&A&Wins&
• Raise&your&right&hand&if&you&think&B&Wins&
• Don’t&raise&your&hand&if&you&think&they’re&about&the&same&
B&

16. 6/8/15&
14&
+Office+Online
If&&you&did&not&raise&a&hand,&
please&sit&down&
If&you&raised&your&right&hand,&
please&sit&down&
A&was&64%&beQer&
The&Office&Online&team&wrote&
A/B$tesBng$is$a$fundamental$and$criBcal$Web$services…$
consistent$use$of$A/B$tesBng$could$save$the$company$
millions$of$dollars$$
MSN+Home+Page+Search+Box
&&&ClickXthrough&rate&for&Search&box&and&popular&searches&
A&
B&
Differences:&&
A&has&taller&search&box&(overall&size&is&the&same),&has&magnifying&glass&icon,&
“popular&searches”&&
B&has&big&search&buQon&
• Raise&your&lem&hand&if&you&think&A&Wins&
• Raise&your&right&hand&if&you&think&B&Wins&
• Don’t&raise&your&hand&if&they&are&the&about&the&same&

17. 6/8/15&
15&
+Search+Box
If&you&raised&any&hand,&please&sit&down&
&
Don’t&change&for&the&sake&of&change&
&
SomeFmes&what&you&have&is&enough&
&
29&

18. 6/8/15&
16&
hQp://kadavy.net/blog/posts/aaXtesFng/&
“Given$the$baseline$conversion$rate$on$
opens,$the$sample$size$simply$isn’t$large$
enough$to$get$a$reliable$result.”$

19. 6/8/15&
17&
Tracing&
(Build)&
Collect&
Logs&&&
upload&
(Measure)&
Analyze&
(Learn)&
startTime)=)GetTime());)
CoolActivity();)
…)
Trace(ID,)ACTIVITY_LENGTH,)GetTime())A)startTime))
0:00:00)
12:00:00)
24:00:00)
36:00:00)
48:00:00)
0) 200) 400) 600) 800) 1000) 1200)
Ac0vity)Length)
If)You)Only)Do)One)Thing…)

20. 6/8/15&
18&
There&are&no&Silver&Bullets&
Customer&data&can$help&
Use&both&qualitaFve&and&quanFtaFve&
data&
Customer&data&can&replace&some&tesFng&
Data&does&not&replace&excellent&human&
tesFng&

21. 6/8/15&
19&
Ques0ons?)
hQp://angryweasel.com )))hQp://angryweasel.com/ABTes0ng)