Many organizations use the wrong methods to understand their users. Michael will reveal the critical success factors he has honed in a decade of user research at Scient, eBay Enterprise, and Nielsen Norman Group. He will show real examples from his current work helping PayPal study user behaviors. Whether you are an expert, or just getting started — you will leave this session with concrete actions you can take to get immediate results.

1. 4 Ways PayPal Gets More out
of Usability Research
Michael Summers, PayPal

3. 1
2
3

4. a
b
c

5. MEANWHILE0–0LATER0IN0THE0FUNNEL:
Zombie0“accounts”0don’t0help0–0meat0&0potatoes0usability0II0FORM0LABELING0is0sMll0with0
us!!!0
00
play0video:00In_Field_Form_Labels.mp40000000000might_say_forget_it.avi
• Disappearing0Labels:00link0a0new0CC0(web),0users0enter0date0in0the0format0on0card,007/18;0but0
system0required007/2018;0error0message0did0not0teach0the0required0format0of0MM/YYYY
• ReducMon0in0Errors:00system0now0accepts0MM/YY0or0MM/YYYY;0error0message0corrected
I In0the030months0leading0up0to0the0fix,0on0average027%0of0users0experienced0an0error0
when0agempMng0to0link0a0card
I In0the030months0post0fix,0on0average08.2%0of0users0experienced0an0error

9. selec%ve'collec%on'of'evidence'
memory;''''primacy;''''sta.c/narrow/no2context'prototype'
'watch'the'videos!'
'UI'creator''≠''UI'evaluator'
'
'
selec%ve'interpreta%on'of'evidence'
show'the'videos!'
'
'
selec%ve'recall'of'evidence'
Sanitloso,'“Mo.vated'recruitment'of'autobiographical'
memories”,'Journal'of'Personality'and'Social'Psychology'
confirma%on'bias:'
Kunda,'“Direc.onal'ques.ons'direct'self5
concep.ons”,'Journal'of'Experimental'Social'
Psychology'
“…a'phenomenon'wherein'
decision'makers'have'been'
shown'to'ac.vely'seek'out'and'
assign'more'weight'to'evidence'
that'confirms'their'hypothesis,'
and'ignore'or'underweigh'
evidence'that'could'disconfirm'
their'hypothesis.”'
Plous,'ScoG'(1993),'The'Psychology'of'Judgment'
and'Decision'Making,'McGraw5Hill,'
ISBN'97850507505047756,'OCLC'26931106'

10. hGp://www.youtube.com/watch?v=QZV1Rkxd4wc'

12. behavioral data…

14. preference data…

15. • right'recrui.ng:'“real”'users'
• right'methodology:'“real”'behavior,'“real”'car'
– user2directed,'NOT'moderator2contrived'
– doing;'not'talking'about'doing'
– don’t'distort'mo.va.on;'aGen.on'
• don’t'study'the'cruise'control'lever;'watch'them'drive'
• right'analysis:'
– “right'stuff:”'a)'which'behaviors,'b)'how'design'
should'change'
• Experience:'observe'1000s'of'hours'of'user'behaviors'with'wide'
variety'of'devices'
• Watch'100%'of'footage'–'get'past'obvious'
– independent:'findings'not'filtered,'or'“previewed”'by'
any'stakeholders'
• right'point'in'the'process:'
– did'we'plan'.me'to'do'something'with'the'results?'
UI'='fast;''talking'='slow:''eye_faster_than_UI.avi'
did ya’? did ya’?: thats_a_lie.avi
read%more%at%USERresearch.com%
the'an.dote:'
attention & motivation: infobox_trigger.avi
bob''|''angie''|''gum''|''burberry'
“the%4%R’s”%

16. WHO0IS0THE0REAL0USA0CONSUMER?
vs.'

17. Observing non-leading/natural behavior
requires a much more robust prototype:
EARLY
Lower cost of
Making changes
LATE
Higher cost of
Making changes
Lower Confidence in
usability conclusions
Higher Confidence in
usability conclusions

18. APPENDIX

19. N'(19(19L)'n')'
Nielsen, Jakob, and Landauer, Thomas K.: "A mathematical model of the finding of usability
problems," Proceedings of ACM INTERCHI'93 Conference (Amsterdam, The Netherlands, 24-29
April 1993), pp. 206-213.

20. Nielsen, Jakob, and Landauer, Thomas K.: "A mathematical model of the finding of usability
problems," Proceedings of ACM INTERCHI'93 Conference (Amsterdam, The Netherlands, 24-29
April 1993), pp. 206-213.

21. N'='2,400,000'
1936:''Landon'57%;'Roosevelt,'43%;'''the'Literary'Digest,'had'correctly'predicted'the'outcomes'of'the'1916,'
1920,'1924,'1928,'and'1932'elec.ons'by'conduc.ng'polls.'
Non5response'bias;'selec.on'bias'

22. N'='3,250'
1948;'quota'sampling;'selec.on'bias'