• Like
  • Save
Turn Stakeholders into Data Collectors to to Uncover Testing Results… Quickly
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Turn Stakeholders into Data Collectors to to Uncover Testing Results… Quickly

  • 338 views
Published

With all the data that usability testing produces, it can be difficult and time consuming to uncover the golden nuggets, and unfortunately you don't always have enough time to wait for the formal …

With all the data that usability testing produces, it can be difficult and time consuming to uncover the golden nuggets, and unfortunately you don't always have enough time to wait for the formal usability test report to impact the design process. Sound familiar? At Vistaprint, we've been piloting new ways of capturing and transforming test data into actionable insights to speed up the iterative design process. As a result we're finding that it not only increases our reaction time but improves collaboration and engagement across stakeholders.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
338
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • To change the layout of any slide: Right click on the slide and scroll down to layout Choose the appropriate layout option If images and text need to be reset, right click and choose “reset slide”Slide Name: Title Slide
  • Who We Are – 2 min (never starts on time)Identify the Problem – 3 minDescribe the Idea – 7 minShare Our Results – 13 minQuestions + Discussion – 15 min
  • Vistaprint N.V. (Nasdaq: VPRT) empowers more than 15 million micro businesses and consumers annually with affordable, professional options to make an impression.
  • UX CoE - Created in October 20117 UX Designers with over 75 years of combined UX experience2 UX ResearchersWe focus on delighting our customers by creating an effortless enjoyable experience based on user centered design, customer research, and iterative improvements.
  • As our design process became more iterative, it meant that we needed customer feedback… FAST!
  • We could no longer wait for the formal usability report, although still useful for documentation, executive buy in, etc.
  • We needed design direction immediately to stay on schedule. Yet, each UX designer didn’t have the luxury of watching all 10 usability sessions.
  • What to do…?
  • High Level before getting into the details Utilize people and processes to save time in the iterative design processPeopleLow tech processCapture dataSort dataCapture trends across participants
  • High Level before getting into the details Utilize people and processes to save time in the iterative design processPeopleLow tech processCapture dataSort dataCapture trends across participants
  • We already have a lab at VPWe already have a lot of interest from stakeholders in observing usability testingSO why not make better use of their time!
  • Collaboration – Sharing the workload to obtain resultsEngagement – Stakeholders are more involved in the usability testing processOwnership – Everyone owns the findingsSpeed! – Faster, better, more comprehensive feedback that can be used in an iterative design process – cuz we’re still a lean UX team!
  • High Level before getting into the details Utilize people and processes to save time in the iterative design processPeopleLow tech processCapture dataSort dataCapture trends across participants
  • To be clear, we are not the first ones to think about this. There is a lot of information out there on Data Logging techniques. Many leading to digital tools for quickly capturing data and turning it into a usability test report. There are different techniques for logging Usability Test Data in general. Often done by the usability practitioner him/herselfAt UPA 2004, Dr. David Dayton presented details of how 6 different practitioners (including himself) logged usability data. These were the 4 most popular types of Data Logging PracticesProblem Coding Record predictable events and sort them on the fly into one or more categories. Analyse the resulting quantitative data with statistical methods and compare to pre-defined benchmarks to assess the usability of the product. Event Description Records free-form handwritten notes to capture significant events and/or usability problems. Analyze notes post-test, group and categorize events, and rate their severity. Event Description with Problem Coding Code events into certain pre-set categories, and enter descriptive notes for later team review and discussion of the most significant problems.Event Description with Problem Coding & Video Time Stamps Capture the "story" of a test session in shorthand notes.
  • With varying degrees on pros and cons depending on which approach taken
  • Then there’s the Rainbow Speadsheet by Tomer Sharon – hi tech approach which uses a Google Doc for capturing test data. Each test stakeholder has access to and is responsible for entering observations. With it, you will be able to collaboratively observe UX research sessions with team members (or clients). You will be able to conduct research that involves the entire product team, with results that are turned around quickly and that team members will be committed to acting on. And all of this without writing a formal exhaustive research report that no one wants to read.Requires “homework” for familiarity and access to Google Docs. Ultimately, we wanted something low tech with less mental overload for stakeholders.
  • But we were hoping for something more visible and interactive – where we could see trends throughout the course of the test sessions and quickly organize the visible data into a set of priorities
  • High Level before getting into the details Utilize people and processes to save time in the iterative design processPeopleLow tech processCapture dataSort dataCapture trends across participants
  • We took an iterative approach to field testing some ways of capturing data in a low-tech manner
  • Let all stakeholders record all observations on yellow sticky notes identified by participant #Easy for stakeholdersAt end of session, organize stickies by + and – by site area
  • Inconsistencies – example: funny quote documented by all, something deemed important only documented by UX Not consistently documenting across participants
  • One example where we separated features and had stakeholders document observations and comments by + and -. Another example where we had observers document by step in the flow.
  • Weren’t as engaged over time – due to redundancies
  • One example where we separated features and had stakeholders document observations and comments by + and -. Another example where we had observers document by step in the flow.
  • One example where we separated features and had stakeholders document observations and comments by + and -. Another example where we had observers document by step in the flow.
  • So now what? How do we learn from our pilot tests to remove the redundancy and improve our time spent with the data yet add value for stakeholders? How do we squeeze more out of them?
  • There are many different kinds of testsHere are a couple examples. We are mainly focused here in the middleYou may need to change your approach based on the kind of test, the research questions, and your place in the process
  • Kind of test, research questions, place in process
  • Kind of test, research questions, place in process
  • Kind of test, research questions, place in process

Transcript

  • 1. Turn Stakeholders into DataCollectors to Uncover TestingResults… QuicklySusan Rice & Lisa SpitzUX Center of Excellence, Global
  • 2. Company Confidential Vistaprint © 2013Overview6/3/2013 2• Who We Are• Identify the Problem• Describe the Idea• Initial Approaches• Revised Approach• Q&A + Discussion
  • 3. Company Confidential Vistaprint © 2013Who We Are6/3/2013 3
  • 4. Company Confidential Vistaprint © 2013Vistaprint N.V. (Nasdaq: VPRT)6/3/2013 4We empower more than15 million micro businessesand consumers annuallywith affordable, professionaloptions to make animpression.
  • 5. Company Confidential Vistaprint © 2013Company Facts6/3/2013 5• 25 localized websites• 4,100 employees• 3 state of artmanufacturing facilities• 21 offices worldwide
  • 6. Company Confidential Vistaprint © 2013 6/3/2013 6UX Center of Excellence
  • 7. Company Confidential Vistaprint © 2013The Problem6/3/2013 7
  • 8. Company Confidential Vistaprint © 2013 6/3/2013 8BUILDDESIGNTHINKCHECK
  • 9. Company Confidential Vistaprint © 2013 6/3/2013 9Session Notes Usability Report
  • 10. Company Confidential Vistaprint © 2013 6/3/2013 10
  • 11. Company Confidential Vistaprint © 2013 6/3/2013 11
  • 12. Company Confidential Vistaprint © 2013The Idea6/3/2013 12
  • 13. Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 13Stakeholders Process Saved Time!
  • 14. Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 14Stakeholders Process Saved Time!
  • 15. Company Confidential Vistaprint © 2013 6/3/2013 15Don’t just sit there… Make yourself useful!
  • 16. Company Confidential Vistaprint © 2013Why Stakeholders?6/3/2013 16CollaborationEngagementOwnership
  • 17. Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 17Stakeholders Process Saved Time!
  • 18. Company Confidential Vistaprint © 2013Data Logging Practices6/3/2013 18• Problem Coding• Event Description• Event Description with Problem Coding• Event Description with Problem Coding &Video Time Stampshttp://www.userfocus.co.uk/articles/dataloggingtools.htmlUPA 2004, Dr. David Dayton (Southern Polytechnic State University)
  • 19. Company Confidential Vistaprint © 2013Not Collaborative6/3/2013 19
  • 20. Company Confidential Vistaprint © 2013Tomer’s Rainbow Spreadsheet6/3/2013 20http://uxdesign.smashingmagazine.com/2013/04/11/rainbow-spreadsheet-collaborative-ux-research-tool/A Collaborative Lean UX Research Tool By Tomer Sharon
  • 21. Company Confidential Vistaprint © 2013Too High Tech, Mental Overload6/3/2013 21
  • 22. Company Confidential Vistaprint © 2013Our Low-Tech Solution6/3/2013 22
  • 23. Company Confidential Vistaprint © 2013Our Hypothesis6/3/2013 23Stakeholders Process Saved Time!?
  • 24. Company Confidential Vistaprint © 2013Initial Approaches6/3/2013 24
  • 25. Company Confidential Vistaprint © 2013 6/3/2013 25
  • 26. Company Confidential Vistaprint © 2013Pilot 1: Approach6/3/2013 26• All stakeholders recordobservations on yellowsticky notes identified byparticipant #• At end of session,organize stickies by + / -according to site area
  • 27. Company Confidential Vistaprint © 2013Pilot 1: Lessons Learned6/3/2013 27• Errors• Inconsistencies• Redundancy• Too long to collate andmake sense of the data
  • 28. Company Confidential Vistaprint © 2013Pilot 2: Approach6/3/2013 28• Stakeholders still recordobservations on stickies• Stickies were color codedby participant
  • 29. Company Confidential Vistaprint © 2013Pilot 2: Lessons Learned6/3/2013 29• Still too much redundancy• Stakeholders didn’t feelthey were adding value
  • 30. Company Confidential Vistaprint © 2013Pilot 3: Approach6/3/2013 30• Stakeholders still recordobservations on stickies• UX actively cullsobservations during andbetween sessions• Identify # of participantsmaking similar comments• Duplicate stickies aretossed
  • 31. Company Confidential Vistaprint © 2013Pilot 3: Lessons Learned6/3/2013 31• Requires one person tobe there to managestickies• Need more time betweensessions• Redundancy is still anissue, but is managed onthe fly
  • 32. Company Confidential Vistaprint © 2013What Worked6/3/2013 32• People were engaged• Stimulated discussion• Immediate readout ofobservations• Richer set ofobservations• Creating the usabilitytest report was easierbecause the data wasalready compiled
  • 33. Company Confidential Vistaprint © 2013What Didn’t Work6/3/2013 33• Errors in participantidentification• Inconsistent documentation• Tons of redundancy• No consistent way to obtainquantitative feedback• Observations were toogeneral• We didn’t know what torecord• Not saving enough time!
  • 34. Company Confidential Vistaprint © 2013
  • 35. Company Confidential Vistaprint © 2013Revised Approach6/3/2013 35
  • 36. Company Confidential Vistaprint © 2013 6/3/2013 36One size fits all NOT!
  • 37. Company Confidential Vistaprint © 2013Product Development Phases6/3/2013 37http://www.nngroup.com/articles/which-ux-research-methods/Strategize Optimize AssessGoal: Inspire, explore andchoose new directionsand opportunitiesInform and optimizedesigns in order to reducerisk and improve usabilityMeasure productperformance against itselfor its competitionApproach: Qualitative andQuantitativeMainly Qualitative(formative)Mainly Quantitative(summative)TypicalMethods:Ethnographic field studies,focus groups, diary studies,surveys, data mining oranalyticsCardsorting, field studies,participatory design, paperprototype and usabilitystudies, desirability studies,customer emailsUsability benchmarking,online assessments,surveys, A/B testing
  • 38. Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 38Simple SiteFlowsComplexInteractionsAll Studies
  • 39. Company Confidential Vistaprint © 2013All Studies6/3/2013 39AllStakeholders
  • 40. Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 40Simple SiteFlowsComplexInteractionsAll Studies
  • 41. Company Confidential Vistaprint © 2013Simple Site Flows6/3/2013 41
  • 42. Company Confidential Vistaprint © 2013Simple Site Flows6/3/2013 42Key Takeaway:WhatWorked?Key Takeaway:What Didn’tWork?
  • 43. Company Confidential Vistaprint © 2013Two Types of Tests6/3/2013 43Simple SiteFlowsComplexInteractionsAll Studies
  • 44. Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 44
  • 45. Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 45Key Takeaway:What Worked?Key Takeaway:What Didn’tWork?
  • 46. Company Confidential Vistaprint © 2013Complex Interactions6/3/2013 46P1 P2 P3 P4 P51. First they did …2.3.4.5.1 Notetaker
  • 47. Company Confidential Vistaprint © 2013Questions + Discussion6/3/2013 47
  • 48. Company Confidential Vistaprint © 2013 6/3/2013 48General questions?Your test questions?Note taking ideas?Unique challenges?
  • 49. Company Confidential Vistaprint © 2013Thanks!6/3/2013 49Special thanks to the UX CoE team for their fantasticediting expertise… and for gathering supporting examplesfor the presentation.Susan Rice leads UX Designsrice@vistaprint.comLisa Spitz is a Senior UX Designerlspitz@vistaprint.com