Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Turn Stakeholders into DataCollectors to Uncover TestingResults… QuicklySusan Rice & Lisa SpitzUX Center of Excellence, Gl...
Company Confidential Vistaprint © 2013Overview6/3/2013 2• Who We Are• Identify the Problem• Describe the Idea• Initial App...
Company Confidential Vistaprint © 2013Who We Are6/3/2013 3
Company Confidential Vistaprint © 2013Vistaprint N.V. (Nasdaq: VPRT)6/3/2013 4We empower more than15 million micro busines...
Company Confidential Vistaprint © 2013Company Facts6/3/2013 5• 25 localized websites• 4,100 employees• 3 state of artmanuf...
Company Confidential Vistaprint © 2013 6/3/2013 6UX Center of Excellence
Company Confidential Vistaprint © 2013The Problem6/3/2013 7
Company Confidential Vistaprint © 2013 6/3/2013 8BUILDDESIGNTHINKCHECK
Company Confidential Vistaprint © 2013 6/3/2013 9Session Notes Usability Report
Company Confidential Vistaprint © 2013 6/3/2013 10
Company Confidential Vistaprint © 2013 6/3/2013 11
Company Confidential Vistaprint © 2013The Idea6/3/2013 12
Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 13Stakeholders Process Saved Time!
Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 14Stakeholders Process Saved Time!
Company Confidential Vistaprint © 2013 6/3/2013 15Don’t just sit there… Make yourself useful!
Company Confidential Vistaprint © 2013Why Stakeholders?6/3/2013 16CollaborationEngagementOwnership
Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 17Stakeholders Process Saved Time!
Company Confidential Vistaprint © 2013Data Logging Practices6/3/2013 18• Problem Coding• Event Description• Event Descript...
Company Confidential Vistaprint © 2013Not Collaborative6/3/2013 19
Company Confidential Vistaprint © 2013Tomer’s Rainbow Spreadsheet6/3/2013 20http://uxdesign.smashingmagazine.com/2013/04/1...
Company Confidential Vistaprint © 2013Too High Tech, Mental Overload6/3/2013 21
Company Confidential Vistaprint © 2013Our Low-Tech Solution6/3/2013 22
Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 23Stakeholders Process Saved Time!?
Company Confidential Vistaprint © 2013Initial Approaches6/3/2013 24
Company Confidential Vistaprint © 2013 6/3/2013 25
Company Confidential Vistaprint © 2013Pilot 1: Approach6/3/2013 26• All stakeholders recordobservations on yellowsticky no...
Company Confidential Vistaprint © 2013Pilot 1: Lessons Learned6/3/2013 27• Errors• Inconsistencies• Redundancy• Too long t...
Company Confidential Vistaprint © 2013Pilot 2: Approach6/3/2013 28• Stakeholders still recordobservations on stickies• Sti...
Company Confidential Vistaprint © 2013Pilot 2: Lessons Learned6/3/2013 29• Still too much redundancy• Stakeholders didn’t ...
Company Confidential Vistaprint © 2013Pilot 3: Approach6/3/2013 30• Stakeholders still recordobservations on stickies• UX ...
Company Confidential Vistaprint © 2013Pilot 3: Lessons Learned6/3/2013 31• Requires one person tobe there to managestickie...
Company Confidential Vistaprint © 2013What Worked6/3/2013 32• People were engaged• Stimulated discussion• Immediate readou...
Company Confidential Vistaprint © 2013What Didn’t Work6/3/2013 33• Errors in participantidentification• Inconsistent docum...
Company Confidential Vistaprint © 2013
Company Confidential Vistaprint © 2013Revised Approach6/3/2013 35
Company Confidential Vistaprint © 2013 6/3/2013 36One size fits all NOT!
Company Confidential Vistaprint © 2013Product Development Phases6/3/2013 37http://www.nngroup.com/articles/which-ux-resear...
Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 38Simple SiteFlowsComplexInteractionsAll Studies
Company Confidential Vistaprint © 2013All Studies6/3/2013 39AllStakeholders
Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 40Simple SiteFlowsComplexInteractionsAll Studies
Company Confidential Vistaprint © 2013Simple Site Flows6/3/2013 41
Company Confidential Vistaprint © 2013Simple Site Flows6/3/2013 42Key Takeaway:WhatWorked?Key Takeaway:What Didn’tWork?
Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 43Simple SiteFlowsComplexInteractionsAll Studies
Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 44
Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 45Key Takeaway:What Worked?Key Takeaway:What Didn’tWork?
Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 46P1 P2 P3 P4 P51. First they did …2.3.4.5.1 Notetaker
Company Confidential Vistaprint © 2013Questions + Discussion6/3/2013 47
Company Confidential Vistaprint © 2013 6/3/2013 48General questions?Your test questions?Note taking ideas?Unique challenges?
Company Confidential Vistaprint © 2013Thanks!6/3/2013 49Special thanks to the UX CoE team for their fantasticediting exper...
Upcoming SlideShare
Loading in …5
×

Turn Stakeholders into Data Collectors to to Uncover Testing Results… Quickly

949 views

Published on

With all the data that usability testing produces, it can be difficult and time consuming to uncover the golden nuggets, and unfortunately you don't always have enough time to wait for the formal usability test report to impact the design process. Sound familiar? At Vistaprint, we've been piloting new ways of capturing and transforming test data into actionable insights to speed up the iterative design process. As a result we're finding that it not only increases our reaction time but improves collaboration and engagement across stakeholders.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Turn Stakeholders into Data Collectors to to Uncover Testing Results… Quickly

  1. 1. Turn Stakeholders into DataCollectors to Uncover TestingResults… QuicklySusan Rice & Lisa SpitzUX Center of Excellence, Global
  2. 2. Company Confidential Vistaprint © 2013Overview6/3/2013 2• Who We Are• Identify the Problem• Describe the Idea• Initial Approaches• Revised Approach• Q&A + Discussion
  3. 3. Company Confidential Vistaprint © 2013Who We Are6/3/2013 3
  4. 4. Company Confidential Vistaprint © 2013Vistaprint N.V. (Nasdaq: VPRT)6/3/2013 4We empower more than15 million micro businessesand consumers annuallywith affordable, professionaloptions to make animpression.
  5. 5. Company Confidential Vistaprint © 2013Company Facts6/3/2013 5• 25 localized websites• 4,100 employees• 3 state of artmanufacturing facilities• 21 offices worldwide
  6. 6. Company Confidential Vistaprint © 2013 6/3/2013 6UX Center of Excellence
  7. 7. Company Confidential Vistaprint © 2013The Problem6/3/2013 7
  8. 8. Company Confidential Vistaprint © 2013 6/3/2013 8BUILDDESIGNTHINKCHECK
  9. 9. Company Confidential Vistaprint © 2013 6/3/2013 9Session Notes Usability Report
  10. 10. Company Confidential Vistaprint © 2013 6/3/2013 10
  11. 11. Company Confidential Vistaprint © 2013 6/3/2013 11
  12. 12. Company Confidential Vistaprint © 2013The Idea6/3/2013 12
  13. 13. Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 13Stakeholders Process Saved Time!
  14. 14. Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 14Stakeholders Process Saved Time!
  15. 15. Company Confidential Vistaprint © 2013 6/3/2013 15Don’t just sit there… Make yourself useful!
  16. 16. Company Confidential Vistaprint © 2013Why Stakeholders?6/3/2013 16CollaborationEngagementOwnership
  17. 17. Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 17Stakeholders Process Saved Time!
  18. 18. Company Confidential Vistaprint © 2013Data Logging Practices6/3/2013 18• Problem Coding• Event Description• Event Description with Problem Coding• Event Description with Problem Coding &Video Time Stampshttp://www.userfocus.co.uk/articles/dataloggingtools.htmlUPA 2004, Dr. David Dayton (Southern Polytechnic State University)
  19. 19. Company Confidential Vistaprint © 2013Not Collaborative6/3/2013 19
  20. 20. Company Confidential Vistaprint © 2013Tomer’s Rainbow Spreadsheet6/3/2013 20http://uxdesign.smashingmagazine.com/2013/04/11/rainbow-spreadsheet-collaborative-ux-research-tool/A Collaborative Lean UX Research Tool By Tomer Sharon
  21. 21. Company Confidential Vistaprint © 2013Too High Tech, Mental Overload6/3/2013 21
  22. 22. Company Confidential Vistaprint © 2013Our Low-Tech Solution6/3/2013 22
  23. 23. Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 23Stakeholders Process Saved Time!?
  24. 24. Company Confidential Vistaprint © 2013Initial Approaches6/3/2013 24
  25. 25. Company Confidential Vistaprint © 2013 6/3/2013 25
  26. 26. Company Confidential Vistaprint © 2013Pilot 1: Approach6/3/2013 26• All stakeholders recordobservations on yellowsticky notes identified byparticipant #• At end of session,organize stickies by + / -according to site area
  27. 27. Company Confidential Vistaprint © 2013Pilot 1: Lessons Learned6/3/2013 27• Errors• Inconsistencies• Redundancy• Too long to collate andmake sense of the data
  28. 28. Company Confidential Vistaprint © 2013Pilot 2: Approach6/3/2013 28• Stakeholders still recordobservations on stickies• Stickies were color codedby participant
  29. 29. Company Confidential Vistaprint © 2013Pilot 2: Lessons Learned6/3/2013 29• Still too much redundancy• Stakeholders didn’t feelthey were adding value
  30. 30. Company Confidential Vistaprint © 2013Pilot 3: Approach6/3/2013 30• Stakeholders still recordobservations on stickies• UX actively cullsobservations during andbetween sessions• Identify # of participantsmaking similar comments• Duplicate stickies aretossed
  31. 31. Company Confidential Vistaprint © 2013Pilot 3: Lessons Learned6/3/2013 31• Requires one person tobe there to managestickies• Need more time betweensessions• Redundancy is still anissue, but is managed onthe fly
  32. 32. Company Confidential Vistaprint © 2013What Worked6/3/2013 32• People were engaged• Stimulated discussion• Immediate readout ofobservations• Richer set ofobservations• Creating the usabilitytest report was easierbecause the data wasalready compiled
  33. 33. Company Confidential Vistaprint © 2013What Didn’t Work6/3/2013 33• Errors in participantidentification• Inconsistent documentation• Tons of redundancy• No consistent way to obtainquantitative feedback• Observations were toogeneral• We didn’t know what torecord• Not saving enough time!
  34. 34. Company Confidential Vistaprint © 2013
  35. 35. Company Confidential Vistaprint © 2013Revised Approach6/3/2013 35
  36. 36. Company Confidential Vistaprint © 2013 6/3/2013 36One size fits all NOT!
  37. 37. Company Confidential Vistaprint © 2013Product Development Phases6/3/2013 37http://www.nngroup.com/articles/which-ux-research-methods/Strategize Optimize AssessGoal: Inspire, explore andchoose new directionsand opportunitiesInform and optimizedesigns in order to reducerisk and improve usabilityMeasure productperformance against itselfor its competitionApproach: Qualitative andQuantitativeMainly Qualitative(formative)Mainly Quantitative(summative)TypicalMethods:Ethnographic field studies,focus groups, diary studies,surveys, data mining oranalyticsCardsorting, field studies,participatory design, paperprototype and usabilitystudies, desirability studies,customer emailsUsability benchmarking,online assessments,surveys, A/B testing
  38. 38. Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 38Simple SiteFlowsComplexInteractionsAll Studies
  39. 39. Company Confidential Vistaprint © 2013All Studies6/3/2013 39AllStakeholders
  40. 40. Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 40Simple SiteFlowsComplexInteractionsAll Studies
  41. 41. Company Confidential Vistaprint © 2013Simple Site Flows6/3/2013 41
  42. 42. Company Confidential Vistaprint © 2013Simple Site Flows6/3/2013 42Key Takeaway:WhatWorked?Key Takeaway:What Didn’tWork?
  43. 43. Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 43Simple SiteFlowsComplexInteractionsAll Studies
  44. 44. Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 44
  45. 45. Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 45Key Takeaway:What Worked?Key Takeaway:What Didn’tWork?
  46. 46. Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 46P1 P2 P3 P4 P51. First they did …2.3.4.5.1 Notetaker
  47. 47. Company Confidential Vistaprint © 2013Questions + Discussion6/3/2013 47
  48. 48. Company Confidential Vistaprint © 2013 6/3/2013 48General questions?Your test questions?Note taking ideas?Unique challenges?
  49. 49. Company Confidential Vistaprint © 2013Thanks!6/3/2013 49Special thanks to the UX CoE team for their fantasticediting expertise… and for gathering supporting examplesfor the presentation.Susan Rice leads UX Designsrice@vistaprint.comLisa Spitz is a Senior UX Designerlspitz@vistaprint.com

×