• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
International Business User Research: Methods and Tools

International Business User Research: Methods and Tools



Qualitative Research on International Business Users ...

Qualitative Research on International Business Users
How to interview (business) users
Cross-cultural user research
Online (remote) User Research
Mouse tracking heatmap
Remote user testing
User feedback system



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds


Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • 解决问题,优缺点,资源需求
  • 概念 行为人工 自动http://www.slideshare.net/boltpeters/remote-research-at-ixd10Concept FeedbackChalkmarkClickheatClickTaleClixpyCrazy EggGoogle AnalyticsGoogle Website OptimizerEthnio recruiting tool only nowFeng-GUIFive Second TestFeedback Armypanel&toolLoop11 toolMechanical TurkMoraeOpen Hallway toolSimple Mouse TrackingUsabilla toolUserFlyUserTesting.companel&toolTrymyui.com provide panelist using OpenHallway toolUserbilitytest.com tool
  • Concept FeedbackChalkmarkClickheatClickTaleClixpyCrazy EggEthnioFeng-GUIFive Second TestFeedback ArmyLoop11Mechanical TurkMoraeOpen HallwaySilverbackSimple Mouse TrackingUsabillaUserFlyUserTesting.comUserVueGoogle AnalyticsGoogle Website Optimizer
  • And there’s the international user challenge, one of the interesting experience to a global user researcher would be cultural differences in silence.
  • given an intent / opportunity to click in the current user activity, the mouse is much more likely to be close to the eye. This is a basic rationality that says "If I might click, I might as well keep the mouse close to my eyes." Where there's no potential to click, either because the user is in an evaluative mode or the content of interest is devoid of links, the mouse and eye diverge. The scientific literature is closing in on this interpretation with results which show that mouse prediction is predictive of gaze, sometimes. How this partial mapping can best be turned into value is to be determined
  • Demo: http://test.ued.alibaba.com/product.htmlhttp://test.ued.alibaba.com/advanced-posting/index.htmlReplay:http://test.ued.alibaba.com/smt2/admin/ext/tracking-report/track.php?id=194&api=js
  • Demo: http://test.ued.alibaba.com/product.htmlhttp://test.ued.alibaba.com/advanced-posting/index.htmlReplay:http://test.ued.alibaba.com/smt2/admin/ext/tracking-report/track.php?id=194&api=js
  • http://yong.optimalworkshop.com/chalkmark/survey/alibaba-1
  • The happy customer smile

International Business User Research: Methods and Tools International Business User Research: Methods and Tools Presentation Transcript

  • International Business User Research: Methods and Tools Zhang, Yong
  • About me 张勇 (Zhang,Yong)  Vanderbilt University  M.S. Management of Technology  Alibaba.com  User Research / Data Analysis  5 yrs of business user research covering both International Buyers and China Suppliers  Explored and implemented various user research methods and tools: Online Survey, Remote User Testing, Focus Group, Interview, AB-Testing, Persona and other Data mining projects
  • Contents Qualitative Research on International Business Users • How to interview (business) users • Cross-cultural user research Online (remote) User Research • Mouse tracking heatmap • Remote user testing • User feedback system
  • Planning Requirements Design Implementation Test & Measure Post Release Lead user study User surveys Design guidelines Style guides Performance testing Data analysis Stakeholder meeting (Brainstorming) Interviews Paper prototyping Rapid prototyping Diagnostic evaluation Subjective assessment Competitor analysis Contextual inquiry Heuristic evaluation Cognitive walkthrough User surveys Data analysis Field study User observation Parallel design A/B testing Remote evaluation Secondary research Focus group Storyboarding Persona Wizard of Oz Evaluating existing system Interface design patterns Card sorting Affinity diagramming Scenarios of use (User diary) Task analysis User research methods in product development lifecycle: Quantitative – Qualitative – Quantitative Concept FBRD Specs Checklist Review (from The Persona Lifecycle: Keeping People in Mind Throughout Product Design )
  • More (advanced) methods and tools All pictures from Google image search
  • But If most of your users are international and your only have 2 weeks, at most cannot find users to come to your lab …… As of December 31, 2012, the platform had around 36.7 million registered users from more than 240 countries and regions and showcased more than 2.8 million supplier storefronts. http://news.alibaba.com/specials/aboutalibaba/aligroup/index.html
  • You may want to consider: Online (Remote) User Research Instant Message/Phone Interview Instant Message/Online Chat Room/Conference Call Focus Group http://www.slideshare.net/boltpeters/remote-research-at-ixd10
  • How to choose from the Remote User Research Toolbox Automated Simple Tasks A/B Tests Various Forms Of Online Surveys Mouse Tracking Heatmap Online Focus Group or Phone/IM interviews Remote User Testing Project Progress
  • Most popular user research methods Methods Solve N Min. Time Pros Cons Online Interview / Focus Group Explore concept & solutions 6-12 6 ~10 days Rich insight, in depth Appointment for Int’l Focus Group Remote user testing Specific interaction solutions 6-12 6 ~10 days Effective in finding usability problems Appointment for Int’l Focus Group Online tests: (card sorting, simple tasks) Solution prototypes 50+ 10 days Quick concept proof, Effective in IA Limited testable prototype Online survey Specific questions 100+ 10 days Statistical Significant Questionnaire design Mouse Tracking HeatMap Launched product 200+ 3 days Visual, intuitive Single page analysis A/B test Launched product 200++ 15 days Accurate, Statistical Significant May need more time for results
  • User Interviews:F2F, Phone, online  User researchers and designers (sometimes product managers) go to interview users together, to enrich shared understanding of our users.  Interviewees:  China Suppliers  International Buyers  Methods:  Factory visits, tradeshows  Phone interview, conference calls  Online focus group Member: Mr. Zhou Company: Zhejiang Dale Garments Co., Ltd. Jane F. Ivanov President, Eve Alexander Maternity/Nursing Lingerie Location: USA Industry: Apparel Member since 2005
  • How to interview (business) users  Aim to achieve rapport (in sync moment) in conversation  Let user start telling stories, (where the richest insights lie)  Active listening and humility is the most effective way to build trust and rapport  Make user the expert and you are the novice, (vice-versa)  Ask the participants to play the teacher role help to articulate the details  Sometimes you need to show some areas of expertise to make them want to talk to you  Near the end of the interview is a great opportunity to ask more audacious questions (about their business)  Check your own worldview at the door, ready to embrace user’s mental model(Empathy)  Don’t fixate on what you expect to learn, cultivate your own general curiosity (about roles in business decisions) *Interviewing Users: How to Uncover Compelling Insights (Portigal, Steve)
  • Cross-cultural user research  沉默是金:Silence is Gold  Using silence to elicit participants to talk is an art in interview  Three kinds of silences in Japanese conversations*  Setting-the-stage silence:  Indicate readiness for a shared experience  Show mutual respect and humility  Interviewer need to match with natural responses and thought pace  Effort silence:  Show efforts in thinking and a desire to contribute  Interviewer should allow room to think, use silence to encourage the efforts  Failure silence:  Indicate confusion or resistance  Interviewer should be attentive to this silence and step in to help *Interviewing Users: How to Uncover Compelling Insights (Portigal, Steve)
  • Cross-cultural user research  Think aloud in user testing • Definition: In a thinking aloud test, you ask test participants to use the system while continuously thinking out loud — that is, simply verbalizing their thoughts as they move through the user interface. • In Asia, it is impolite to tell someone they have a bad design. • It is embarrassing to admit that you cannot find something.  May consider ―Retrospective Think-aloud protocol‖** after task completion  East Asians need more encouragement, be attentive to silence  Sometimes, young/inexperienced facilitator may be able to collect more negative feedbacks  ―Teach me how to do it‖, asking user to play the teacher’s role * Thinking Aloud: The #1 Usability Tool http://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/ ** Kuusela, H., & Paul, P. (2000). "A comparison of concurrent and retrospective verbal protocol analysis". American Journal of Psychology (University of Illinois Press) 113 (3): 387–404.
  • Cross-cultural user research  East: Interdependence, Relationships, Panoramic  West: Independent, Individualistic, zoom in  ―Westerners focus on a main or dominant foreground object, while people from East Asia pay more attention to context and background. (East Asian people who grow up in the West show the Western pattern.)”  ―When shown complex, busy scenes, Asian- Americans and non- Asian-Americans recruited different brain regions. The Asians showed more activity in areas that process figure-ground relations— holistic context—while the Americans showed more activity in regions that recognize objects.‖  How we see it: Culturally different eye movement patterns over visual scenes, Julie E. Boland, Hannah Faye Chua, & Richard E. Nisbett  Differences Between East and West Discovered in People’s Brain Activity  Carey Goldberg, http://tech.mit.edu/V128/N9/culture.html  Sharon Begley: West Brain, East Brain May explain this!
  • Poor man’s eye tracking Heatmap Based on: http://sourceforge.net/projects/clickheat/
  • User mouse click tracking and in-page web analysis tools  The Problem:  Designers think best in pictures, they need better visual way to see users’ interaction on our page elements  Eye tracking is too expensive in $ and time  Background:  Given a high intent/opportunity to click, the mouse move and click are much more likely to be closely related to the eye movement* * Summary of research on eye - mouse syn available from: Edmonds, A., White, R., Morris, D., Drucker, S. Instrumenting the Dynamic Web. Journal of Web Engineering (JWE), Vol. 6, No. 3 (2007), 243-260. Solution: •Embed click tracking JavaScript to collect user interactions on page. •Real-time adjustment of sampling ratio and/or targeting conditions, i.e. certain types of users, etc.
  • Clickheat (flat log file) Filter by user types Marmot* @ Taobao, Baidu (hadoop) The evolution of Clickheat *http://www.slideshare.net/taobaoued/marmot
  • SMT2: Simple mouse tracking https://code.google.com/p/smt2/
  • Mouse movement tracking SMT video replay
  • Moderated remote user testing
  • Remote user testing tools and methods (Example test: main user flow) Recruiting participants Participants registration and screening Inviting participant to testing Participant accept testing Record and review testing sessions Live Recruiting Ads* Online Registration Forms Phone call or IM TechSmith UserVUE UserVUE Morae Manager
  • Step 1:Live Recruiting Kiva.org Vs. Amazon gift certificates Note:Used Ethnio Recruiting tool first, developed our own tool later. Basic Function:Pop-in Live Recruiting Ads. Real time control On/Off, Pop-in ratio, etc. Advanced Function:Conditioning on member types, reg. date, last visit time, etc. Real time control On/Off, Ratio, Conditions
  • Step 2:Participants Registration and Screening Simple questionnaire for making appointment and screening Live Recruiting support Basic Function:Used Open Source LimeSurvey system for Registration form and survey Advanced:Embed chat for answering questions and live recruiting support
  • Step 3:Invite participants to testing Limited by facilitator resources, tests conducted one by one Basic Function:LimeSurvey responses analysis for screening participants
  • Step 4:Participants setup screen TechSmith UserVue
  • Participant’s screen in testing session TechSmith UserVue Tasks and questions communicated via telephone or IM
  • Moderator’s Screen
  • Remote User Testing Analysis:Morae Manager
  • Remote Vs. Traditional User Testing  Pros:  Testing on Real users (incl. 1st time visitors)  Testing on users’ own computer  Interactive feedback and communications on major process  User Screen Recording and live broadcasting to Observers  Can do additional user background interview, possible to do longitude follow-up studies  No travel, low cost and low equipment requirements.  Minimum requirement for users.  Cons:  No facial or body movement video if no webcam on user side.  Possible interruption on user side, may be limited by Internet speed.  Downloading and running application on user’s computer can be a problem. http://www.boltpeters.com/articles/versus.html http://www.techsmith.com/uservue/whyremotetest.asp
  • Remote User Testing Requirements  Need to work with designers and PM to confirm user testing tasks.  Recruiting participants need to be representative to target user group.  From testing plan to pilot testing, close involvement of designer and PM are essential for successful tests.
  • Alternative remote testing tools tried:  User download/run remote desktop sharing applications  Online Conference tools: WebX, GotoMeeting, TeamViewer, etc.  IMs:QQ remote assistance, MSN etc.  VNC: RealVNC, UltraVNC, etc.  RDP:Windows Remote Desktop  No client applications, Remote desktop in Browser  Guacamole: HTML5 remote desktop  Flash-based  Cloud-based Virtual Machine
  • Automated simple tasks Task 2 of 4: Where would you go to sign up for email notifications of the latest products? Task 3 of 4: How to find out the supplier's contact information? Task 4 of 4: How to check out more information about a product you are interested in? Make interaction prototype testable online: •Used OptimalWorkshop Chalkmark for simple IA tests; •Developed single page interaction prototype tests based on limesurvey and Clickheat mouse tracking
  • Online card sorting Solving IA problems Understand a different point of view. Using OptimalWorkshop OptimalSort, for menu design and IA problems.
  • R shiny web app for card sorting analysis (WIP) plot(hclust(dist(Data)))
  • Structured Online Survey: based on Limesurvey Problem: Many quick small research requested Solution: Use panel, survey ads, email survey invitations to quickly collect large amount of target users feedbacks. Many classical marketing research model can provide deep insight for user research, U&A, perception model, etc. Questionnaire design with analysis in mind is the key to successful online survey. Be sure, not sorry
  • Marketing models  Biplot (MCA, MDS, MDPREF) Marketing Research: Uncovering Competitive Advantages, Warren F. Kuhfeld, SAS Institute Inc.
  • A nice exit survey example
  • Voice from users  Problem in collecting user feedbacks  Customer satisfaction surveys: limited coverage and long time lag  Online customer service and feedback form: manual processing low efficiency, no priority  No transparency in user feedback handling, low motivation in participation  Many user suggestions are not specific, not actionable  Operation and management need more valuable voice from customers …… Solution: User feedback systems with product feature/function setting
  • Amazon has been collecting all kinds of specific user feedback Feedback on Ads Feedback on Search
  • Taobao is collecting specific feedback on search Feedback on Search Experience
  • Alibaba.com Key User Flow Feedbacks Feedback on Search Experience Feedback on Product Details But, the value lies in how we process these feedbacks
  • Suggestion Box 2.0 User feedbacks with votes Service providers: •Salesforce.com •Uservoice.com •Getsatisfaction.com
  • User feedback system solution  Main functions and scenarios 1. User post suggestions 2. User interacts on priorities and solutions 3. Company participate in feedback community, provide feedback on feedback  Advantages  Facilitate multiple product lines, configurable feedback types  Integrated user feedback channel  Unified feedback interface and community  Provide long term customer satisfaction metrics  Interaction among users:  Tag and vote on feedbacks  Share and discuss solutions  Interaction with company:  Transparent processing of user feedback  Tracking feedback status on user feedbacks, increased user satisfaction and motivation
  • Team and process supports  Additional problems  Privacy protection: anonymous and closed feedback  Spam control, User admin, Voting limits, Scores, etc.  Search and suggestion to consolidate duplicates  Pre-defined feedback types and product lines  Customer service team handle questions and complains  Support Ticket tracking, integrate with existing BOPS、 CS support email system  Satisfaction rating on CS provider  Suggestions are handled by designated PM  PM handle TOP suggestions of related product lines, tracking, organizing, replying Top feedbacks  User research provide integrated summary reports  Provide user satisfaction metrics by product lines  Follow-up studies on Top user feedbacks User feedback Customer service: Reply (if needed) Question & Complains User interaction and CS reply CS handling of question & complains Issue support ticket tracking Suggestions Users vote on priorities PM handle TOP suggestions User track feedback status
  • AB-Testing at different levels •JavaScript to randomize display of elements •Cookie-based randomization (simple implement) •Not suitable for complex tests, affect page loads •Reverse engineering of Website Optimizer metrics Frontend •Low level code for random assignments •Multiple template sets •Complete multivariate test solutions Template •Low level code for random tests •Useful in algorithm test: search ranking, p4p rules, etc. Backend Simple in-page design Interaction process Workflow, algorithm
  • The Hippos Vs. Jaguar Process Set Success Measure (Manage ment) Collect Ideas (From Everyone) Esp. customers Select (Approver) Execute (Agile team) Cross- functional Test & Evaluation (Iterate) Traps: •Management not aligned •No resources – fully booked Traps: •Ownership issues •Technically impossible Traps: •Endless debate Traps: •Perfection Traps: •Opps…forget about it Highest Paid Person’s Opinion (HiPPO) Joint (JIT) Agile Users Aggregated Responses (JAgUAR) AB-Testing User Feedbacks
  • •User feedbacks •Systems approach, standardized process •User input for priority •Feedback on feedback Understand real users need •User research •Validate design ASAP •Avoid HiPPO’s design •User-Centered Interface/interaction Design effective solutions •Agile Development •Cross-functional small teams •Bottom-up communication •Fast iterations Fast iterative product development •Evaluating performance •Fast, actionable testing results •Multivariate tests •Clear and concise metrics Metric-based validation Collaborative teams and integrated process User Research Customer service User Research Product Management User Research Design and Development
  • Kaizen  Continuous Improvement 改善 http://en.wikipedia.org/wiki/Kaizen Kaizen (Japanese for "improvement" or "change for the better") refers to a philosophy or practices that focus upon continuous improvement of processes in manufacturing, engineering, supporting business processes, and management. It has been applied in healthcare, government, banking, and many other industries. When used in the business sense and applied to the workplace, kaizen refers to activities that continually improve all functions, and involves all employees from the CEO to the assembly line workers.