International Business
User Research:
Methods and Tools
Zhang, Yong
About me
张勇 (Zhang,Yong)
 Vanderbilt University
 M.S. Management of Technology
 Alibaba.com
 User Research / Data Anal...
Contents
Qualitative Research on International
Business Users
• How to interview (business) users
• Cross-cultural user re...
Planning Requirements Design Implementation Test & Measure Post Release
Lead user study User surveys Design guidelines Sty...
More (advanced) methods and tools
All pictures from Google image search
But
If most of your users are international
and your only have 2 weeks, at most
cannot find users to come to your lab
……
A...
You may want to consider: Online
(Remote) User Research
Instant Message/Phone Interview
Instant Message/Online Chat Room/C...
How to choose from the Remote
User Research Toolbox
Automated
Simple Tasks
A/B Tests
Various Forms
Of Online Surveys
Mouse...
Most popular user research methods
Methods Solve N
Min.
Time
Pros Cons
Online Interview /
Focus Group
Explore
concept &
so...
User Interviews:F2F, Phone, online
 User researchers and designers (sometimes product managers)
go to interview users tog...
How to interview (business) users
 Aim to achieve rapport (in sync moment) in conversation
 Let user start telling stori...
Cross-cultural user research
 沉默是金:Silence is Gold
 Using silence to elicit participants to talk is an art in interview
...
Cross-cultural user research
 Think aloud in user testing
• Definition: In a thinking aloud test, you ask test participan...
Cross-cultural user research
 East: Interdependence,
Relationships,
Panoramic
 West: Independent,
Individualistic, zoom ...
Poor man’s eye tracking Heatmap
Based on: http://sourceforge.net/projects/clickheat/
User mouse click tracking and in-page
web analysis tools
 The Problem:
 Designers think best in pictures, they
need bett...
Clickheat
(flat log file)
Filter by user
types
Marmot* @
Taobao, Baidu
(hadoop)
The evolution of Clickheat
*http://www.sli...
SMT2: Simple mouse tracking
https://code.google.com/p/smt2/
Mouse movement tracking
SMT video replay
Moderated remote user testing
Remote user testing tools and methods
(Example test: main user flow)
Recruiting participants
Participants registration and...
Step 1:Live Recruiting
Kiva.org Vs.
Amazon gift
certificates
Note:Used Ethnio Recruiting tool first, developed our own too...
Step 2:Participants Registration and Screening
Simple questionnaire
for making appointment
and screening
Live Recruiting
s...
Step 3:Invite participants to testing
Limited by facilitator resources, tests conducted one by one
Basic Function:LimeSurv...
Step 4:Participants setup screen
TechSmith UserVue
Participant’s screen in testing session
TechSmith UserVue
Tasks and questions communicated via telephone or IM
Moderator’s Screen
Remote User Testing Analysis:Morae Manager
Remote Vs. Traditional User Testing
 Pros:
 Testing on Real users (incl. 1st time visitors)
 Testing on users’ own comp...
Remote User Testing Requirements
 Need to work with
designers and PM to
confirm user testing
tasks.
 Recruiting particip...
Alternative remote testing tools tried:
 User download/run remote desktop sharing
applications
 Online Conference tools:...
Automated simple tasks
Task 2 of 4: Where would you go to sign up for email notifications of the latest products?
Task 3 o...
Online card sorting
Solving IA problems
Understand a different
point of view.
Using OptimalWorkshop OptimalSort, for menu ...
R shiny web app for card sorting
analysis (WIP)
plot(hclust(dist(Data)))
Structured Online Survey: based on Limesurvey
Problem:
Many quick small research
requested
Solution:
Use panel, survey ads...
Marketing models
 Biplot (MCA, MDS, MDPREF)
Marketing Research: Uncovering Competitive Advantages,
Warren F. Kuhfeld, SAS...
A nice exit survey example
Voice from users
 Problem in collecting user feedbacks
 Customer satisfaction surveys: limited coverage
and long time la...
Amazon has been collecting all kinds of
specific user feedback
Feedback
on Ads
Feedback
on Search
Taobao is collecting specific feedback on
search
Feedback
on Search
Experience
Alibaba.com Key User Flow Feedbacks
Feedback
on Search
Experience
Feedback
on
Product
Details
But, the value lies in how w...
Suggestion Box 2.0
User feedbacks with
votes
Service providers:
•Salesforce.com
•Uservoice.com
•Getsatisfaction.com
User feedback system solution
 Main functions and
scenarios
1. User post suggestions
2. User interacts on priorities
and ...
Team and process supports
 Additional problems
 Privacy protection: anonymous and closed feedback
 Spam control, User a...
AB-Testing at different levels
•JavaScript to randomize display of elements
•Cookie-based randomization (simple implement)...
The Hippos Vs. Jaguar Process
Set
Success
Measure
(Manage
ment)
Collect
Ideas
(From
Everyone)
Esp.
customers
Select
(Appro...
•User feedbacks
•Systems approach,
standardized process
•User input for priority
•Feedback on feedback
Understand real
use...
Kaizen
 Continuous Improvement
改善
http://en.wikipedia.org/wiki/Kaizen
Kaizen (Japanese for "improvement" or
"change for t...
Upcoming SlideShare
Loading in …5
×

International Business User Research: Methods and Tools

1,611 views

Published on

Qualitative Research on International Business Users
How to interview (business) users
Cross-cultural user research
Online (remote) User Research
Mouse tracking heatmap
Remote user testing
User feedback system

Published in: Design, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,611
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • 解决问题,优缺点,资源需求
  • 概念 <--> 行为人工 <--> 自动http://www.slideshare.net/boltpeters/remote-research-at-ixd10Concept FeedbackChalkmarkClickheatClickTaleClixpyCrazy EggGoogle AnalyticsGoogle Website OptimizerEthnio recruiting tool only nowFeng-GUIFive Second TestFeedback Armypanel&toolLoop11 toolMechanical TurkMoraeOpen Hallway toolSimple Mouse TrackingUsabilla toolUserFlyUserTesting.companel&toolTrymyui.com provide panelist using OpenHallway toolUserbilitytest.com tool
  • Concept FeedbackChalkmarkClickheatClickTaleClixpyCrazy EggEthnioFeng-GUIFive Second TestFeedback ArmyLoop11Mechanical TurkMoraeOpen HallwaySilverbackSimple Mouse TrackingUsabillaUserFlyUserTesting.comUserVueGoogle AnalyticsGoogle Website Optimizer
  • And there’s the international user challenge, one of the interesting experience to a global user researcher would be cultural differences in silence.
  • given an intent / opportunity to click in the current user activity, the mouse is much more likely to be close to the eye. This is a basic rationality that says "If I might click, I might as well keep the mouse close to my eyes." Where there's no potential to click, either because the user is in an evaluative mode or the content of interest is devoid of links, the mouse and eye diverge. The scientific literature is closing in on this interpretation with results which show that mouse prediction is predictive of gaze, sometimes. How this partial mapping can best be turned into value is to be determined
  • Demo: http://test.ued.alibaba.com/product.htmlhttp://test.ued.alibaba.com/advanced-posting/index.htmlReplay:http://test.ued.alibaba.com/smt2/admin/ext/tracking-report/track.php?id=194&api=js
  • Demo: http://test.ued.alibaba.com/product.htmlhttp://test.ued.alibaba.com/advanced-posting/index.htmlReplay:http://test.ued.alibaba.com/smt2/admin/ext/tracking-report/track.php?id=194&api=js
  • http://yong.optimalworkshop.com/chalkmark/survey/alibaba-1
  • The happy customer smile
  • International Business User Research: Methods and Tools

    1. 1. International Business User Research: Methods and Tools Zhang, Yong
    2. 2. About me 张勇 (Zhang,Yong)  Vanderbilt University  M.S. Management of Technology  Alibaba.com  User Research / Data Analysis  5 yrs of business user research covering both International Buyers and China Suppliers  Explored and implemented various user research methods and tools: Online Survey, Remote User Testing, Focus Group, Interview, AB-Testing, Persona and other Data mining projects
    3. 3. Contents Qualitative Research on International Business Users • How to interview (business) users • Cross-cultural user research Online (remote) User Research • Mouse tracking heatmap • Remote user testing • User feedback system
    4. 4. Planning Requirements Design Implementation Test & Measure Post Release Lead user study User surveys Design guidelines Style guides Performance testing Data analysis Stakeholder meeting (Brainstorming) Interviews Paper prototyping Rapid prototyping Diagnostic evaluation Subjective assessment Competitor analysis Contextual inquiry Heuristic evaluation Cognitive walkthrough User surveys Data analysis Field study User observation Parallel design A/B testing Remote evaluation Secondary research Focus group Storyboarding Persona Wizard of Oz Evaluating existing system Interface design patterns Card sorting Affinity diagramming Scenarios of use (User diary) Task analysis User research methods in product development lifecycle: Quantitative – Qualitative – Quantitative Concept FBRD Specs Checklist Review (from The Persona Lifecycle: Keeping People in Mind Throughout Product Design )
    5. 5. More (advanced) methods and tools All pictures from Google image search
    6. 6. But If most of your users are international and your only have 2 weeks, at most cannot find users to come to your lab …… As of December 31, 2012, the platform had around 36.7 million registered users from more than 240 countries and regions and showcased more than 2.8 million supplier storefronts. http://news.alibaba.com/specials/aboutalibaba/aligroup/index.html
    7. 7. You may want to consider: Online (Remote) User Research Instant Message/Phone Interview Instant Message/Online Chat Room/Conference Call Focus Group http://www.slideshare.net/boltpeters/remote-research-at-ixd10
    8. 8. How to choose from the Remote User Research Toolbox Automated Simple Tasks A/B Tests Various Forms Of Online Surveys Mouse Tracking Heatmap Online Focus Group or Phone/IM interviews Remote User Testing Project Progress
    9. 9. Most popular user research methods Methods Solve N Min. Time Pros Cons Online Interview / Focus Group Explore concept & solutions 6-12 6 ~10 days Rich insight, in depth Appointment for Int’l Focus Group Remote user testing Specific interaction solutions 6-12 6 ~10 days Effective in finding usability problems Appointment for Int’l Focus Group Online tests: (card sorting, simple tasks) Solution prototypes 50+ 10 days Quick concept proof, Effective in IA Limited testable prototype Online survey Specific questions 100+ 10 days Statistical Significant Questionnaire design Mouse Tracking HeatMap Launched product 200+ 3 days Visual, intuitive Single page analysis A/B test Launched product 200++ 15 days Accurate, Statistical Significant May need more time for results
    10. 10. User Interviews:F2F, Phone, online  User researchers and designers (sometimes product managers) go to interview users together, to enrich shared understanding of our users.  Interviewees:  China Suppliers  International Buyers  Methods:  Factory visits, tradeshows  Phone interview, conference calls  Online focus group Member: Mr. Zhou Company: Zhejiang Dale Garments Co., Ltd. Jane F. Ivanov President, Eve Alexander Maternity/Nursing Lingerie Location: USA Industry: Apparel Member since 2005
    11. 11. How to interview (business) users  Aim to achieve rapport (in sync moment) in conversation  Let user start telling stories, (where the richest insights lie)  Active listening and humility is the most effective way to build trust and rapport  Make user the expert and you are the novice, (vice-versa)  Ask the participants to play the teacher role help to articulate the details  Sometimes you need to show some areas of expertise to make them want to talk to you  Near the end of the interview is a great opportunity to ask more audacious questions (about their business)  Check your own worldview at the door, ready to embrace user’s mental model(Empathy)  Don’t fixate on what you expect to learn, cultivate your own general curiosity (about roles in business decisions) *Interviewing Users: How to Uncover Compelling Insights (Portigal, Steve)
    12. 12. Cross-cultural user research  沉默是金:Silence is Gold  Using silence to elicit participants to talk is an art in interview  Three kinds of silences in Japanese conversations*  Setting-the-stage silence:  Indicate readiness for a shared experience  Show mutual respect and humility  Interviewer need to match with natural responses and thought pace  Effort silence:  Show efforts in thinking and a desire to contribute  Interviewer should allow room to think, use silence to encourage the efforts  Failure silence:  Indicate confusion or resistance  Interviewer should be attentive to this silence and step in to help *Interviewing Users: How to Uncover Compelling Insights (Portigal, Steve)
    13. 13. Cross-cultural user research  Think aloud in user testing • Definition: In a thinking aloud test, you ask test participants to use the system while continuously thinking out loud — that is, simply verbalizing their thoughts as they move through the user interface. • In Asia, it is impolite to tell someone they have a bad design. • It is embarrassing to admit that you cannot find something.  May consider ―Retrospective Think-aloud protocol‖** after task completion  East Asians need more encouragement, be attentive to silence  Sometimes, young/inexperienced facilitator may be able to collect more negative feedbacks  ―Teach me how to do it‖, asking user to play the teacher’s role * Thinking Aloud: The #1 Usability Tool http://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/ ** Kuusela, H., & Paul, P. (2000). "A comparison of concurrent and retrospective verbal protocol analysis". American Journal of Psychology (University of Illinois Press) 113 (3): 387–404.
    14. 14. Cross-cultural user research  East: Interdependence, Relationships, Panoramic  West: Independent, Individualistic, zoom in  ―Westerners focus on a main or dominant foreground object, while people from East Asia pay more attention to context and background. (East Asian people who grow up in the West show the Western pattern.)”  ―When shown complex, busy scenes, Asian- Americans and non- Asian-Americans recruited different brain regions. The Asians showed more activity in areas that process figure-ground relations— holistic context—while the Americans showed more activity in regions that recognize objects.‖  How we see it: Culturally different eye movement patterns over visual scenes, Julie E. Boland, Hannah Faye Chua, & Richard E. Nisbett  Differences Between East and West Discovered in People’s Brain Activity  Carey Goldberg, http://tech.mit.edu/V128/N9/culture.html  Sharon Begley: West Brain, East Brain May explain this!
    15. 15. Poor man’s eye tracking Heatmap Based on: http://sourceforge.net/projects/clickheat/
    16. 16. User mouse click tracking and in-page web analysis tools  The Problem:  Designers think best in pictures, they need better visual way to see users’ interaction on our page elements  Eye tracking is too expensive in $ and time  Background:  Given a high intent/opportunity to click, the mouse move and click are much more likely to be closely related to the eye movement* * Summary of research on eye - mouse syn available from: Edmonds, A., White, R., Morris, D., Drucker, S. Instrumenting the Dynamic Web. Journal of Web Engineering (JWE), Vol. 6, No. 3 (2007), 243-260. Solution: •Embed click tracking JavaScript to collect user interactions on page. •Real-time adjustment of sampling ratio and/or targeting conditions, i.e. certain types of users, etc.
    17. 17. Clickheat (flat log file) Filter by user types Marmot* @ Taobao, Baidu (hadoop) The evolution of Clickheat *http://www.slideshare.net/taobaoued/marmot
    18. 18. SMT2: Simple mouse tracking https://code.google.com/p/smt2/
    19. 19. Mouse movement tracking SMT video replay
    20. 20. Moderated remote user testing
    21. 21. Remote user testing tools and methods (Example test: main user flow) Recruiting participants Participants registration and screening Inviting participant to testing Participant accept testing Record and review testing sessions Live Recruiting Ads* Online Registration Forms Phone call or IM TechSmith UserVUE UserVUE Morae Manager
    22. 22. Step 1:Live Recruiting Kiva.org Vs. Amazon gift certificates Note:Used Ethnio Recruiting tool first, developed our own tool later. Basic Function:Pop-in Live Recruiting Ads. Real time control On/Off, Pop-in ratio, etc. Advanced Function:Conditioning on member types, reg. date, last visit time, etc. Real time control On/Off, Ratio, Conditions
    23. 23. Step 2:Participants Registration and Screening Simple questionnaire for making appointment and screening Live Recruiting support Basic Function:Used Open Source LimeSurvey system for Registration form and survey Advanced:Embed chat for answering questions and live recruiting support
    24. 24. Step 3:Invite participants to testing Limited by facilitator resources, tests conducted one by one Basic Function:LimeSurvey responses analysis for screening participants
    25. 25. Step 4:Participants setup screen TechSmith UserVue
    26. 26. Participant’s screen in testing session TechSmith UserVue Tasks and questions communicated via telephone or IM
    27. 27. Moderator’s Screen
    28. 28. Remote User Testing Analysis:Morae Manager
    29. 29. Remote Vs. Traditional User Testing  Pros:  Testing on Real users (incl. 1st time visitors)  Testing on users’ own computer  Interactive feedback and communications on major process  User Screen Recording and live broadcasting to Observers  Can do additional user background interview, possible to do longitude follow-up studies  No travel, low cost and low equipment requirements.  Minimum requirement for users.  Cons:  No facial or body movement video if no webcam on user side.  Possible interruption on user side, may be limited by Internet speed.  Downloading and running application on user’s computer can be a problem. http://www.boltpeters.com/articles/versus.html http://www.techsmith.com/uservue/whyremotetest.asp
    30. 30. Remote User Testing Requirements  Need to work with designers and PM to confirm user testing tasks.  Recruiting participants need to be representative to target user group.  From testing plan to pilot testing, close involvement of designer and PM are essential for successful tests.
    31. 31. Alternative remote testing tools tried:  User download/run remote desktop sharing applications  Online Conference tools: WebX, GotoMeeting, TeamViewer, etc.  IMs:QQ remote assistance, MSN etc.  VNC: RealVNC, UltraVNC, etc.  RDP:Windows Remote Desktop  No client applications, Remote desktop in Browser  Guacamole: HTML5 remote desktop  Flash-based  Cloud-based Virtual Machine
    32. 32. Automated simple tasks Task 2 of 4: Where would you go to sign up for email notifications of the latest products? Task 3 of 4: How to find out the supplier's contact information? Task 4 of 4: How to check out more information about a product you are interested in? Make interaction prototype testable online: •Used OptimalWorkshop Chalkmark for simple IA tests; •Developed single page interaction prototype tests based on limesurvey and Clickheat mouse tracking
    33. 33. Online card sorting Solving IA problems Understand a different point of view. Using OptimalWorkshop OptimalSort, for menu design and IA problems.
    34. 34. R shiny web app for card sorting analysis (WIP) plot(hclust(dist(Data)))
    35. 35. Structured Online Survey: based on Limesurvey Problem: Many quick small research requested Solution: Use panel, survey ads, email survey invitations to quickly collect large amount of target users feedbacks. Many classical marketing research model can provide deep insight for user research, U&A, perception model, etc. Questionnaire design with analysis in mind is the key to successful online survey. Be sure, not sorry
    36. 36. Marketing models  Biplot (MCA, MDS, MDPREF) Marketing Research: Uncovering Competitive Advantages, Warren F. Kuhfeld, SAS Institute Inc.
    37. 37. A nice exit survey example
    38. 38. Voice from users  Problem in collecting user feedbacks  Customer satisfaction surveys: limited coverage and long time lag  Online customer service and feedback form: manual processing low efficiency, no priority  No transparency in user feedback handling, low motivation in participation  Many user suggestions are not specific, not actionable  Operation and management need more valuable voice from customers …… Solution: User feedback systems with product feature/function setting
    39. 39. Amazon has been collecting all kinds of specific user feedback Feedback on Ads Feedback on Search
    40. 40. Taobao is collecting specific feedback on search Feedback on Search Experience
    41. 41. Alibaba.com Key User Flow Feedbacks Feedback on Search Experience Feedback on Product Details But, the value lies in how we process these feedbacks
    42. 42. Suggestion Box 2.0 User feedbacks with votes Service providers: •Salesforce.com •Uservoice.com •Getsatisfaction.com
    43. 43. User feedback system solution  Main functions and scenarios 1. User post suggestions 2. User interacts on priorities and solutions 3. Company participate in feedback community, provide feedback on feedback  Advantages  Facilitate multiple product lines, configurable feedback types  Integrated user feedback channel  Unified feedback interface and community  Provide long term customer satisfaction metrics  Interaction among users:  Tag and vote on feedbacks  Share and discuss solutions  Interaction with company:  Transparent processing of user feedback  Tracking feedback status on user feedbacks, increased user satisfaction and motivation
    44. 44. Team and process supports  Additional problems  Privacy protection: anonymous and closed feedback  Spam control, User admin, Voting limits, Scores, etc.  Search and suggestion to consolidate duplicates  Pre-defined feedback types and product lines  Customer service team handle questions and complains  Support Ticket tracking, integrate with existing BOPS、 CS support email system  Satisfaction rating on CS provider  Suggestions are handled by designated PM  PM handle TOP suggestions of related product lines, tracking, organizing, replying Top feedbacks  User research provide integrated summary reports  Provide user satisfaction metrics by product lines  Follow-up studies on Top user feedbacks User feedback Customer service: Reply (if needed) Question & Complains User interaction and CS reply CS handling of question & complains Issue support ticket tracking Suggestions Users vote on priorities PM handle TOP suggestions User track feedback status
    45. 45. AB-Testing at different levels •JavaScript to randomize display of elements •Cookie-based randomization (simple implement) •Not suitable for complex tests, affect page loads •Reverse engineering of Website Optimizer metrics Frontend •Low level code for random assignments •Multiple template sets •Complete multivariate test solutions Template •Low level code for random tests •Useful in algorithm test: search ranking, p4p rules, etc. Backend Simple in-page design Interaction process Workflow, algorithm
    46. 46. The Hippos Vs. Jaguar Process Set Success Measure (Manage ment) Collect Ideas (From Everyone) Esp. customers Select (Approver) Execute (Agile team) Cross- functional Test & Evaluation (Iterate) Traps: •Management not aligned •No resources – fully booked Traps: •Ownership issues •Technically impossible Traps: •Endless debate Traps: •Perfection Traps: •Opps…forget about it Highest Paid Person’s Opinion (HiPPO) Joint (JIT) Agile Users Aggregated Responses (JAgUAR) AB-Testing User Feedbacks
    47. 47. •User feedbacks •Systems approach, standardized process •User input for priority •Feedback on feedback Understand real users need •User research •Validate design ASAP •Avoid HiPPO’s design •User-Centered Interface/interaction Design effective solutions •Agile Development •Cross-functional small teams •Bottom-up communication •Fast iterations Fast iterative product development •Evaluating performance •Fast, actionable testing results •Multivariate tests •Clear and concise metrics Metric-based validation Collaborative teams and integrated process User Research Customer service User Research Product Management User Research Design and Development
    48. 48. Kaizen  Continuous Improvement 改善 http://en.wikipedia.org/wiki/Kaizen Kaizen (Japanese for "improvement" or "change for the better") refers to a philosophy or practices that focus upon continuous improvement of processes in manufacturing, engineering, supporting business processes, and management. It has been applied in healthcare, government, banking, and many other industries. When used in the business sense and applied to the workplace, kaizen refers to activities that continually improve all functions, and involves all employees from the CEO to the assembly line workers.

    ×