User Studies Are a Waste of Money! …if you know what you are doing J Risto Sarvas Futurice & Aalto University 6/12/12 INUSE seminar
› Head of UX & Service Design at Futurice. › Docentship at Aalto University (Media Technology). › A decade at HIIT.
Futurice in brief TAMPERE HELSINKI LONDON BERLIN DÜSSELDORF 1 Founded in 150 2000
We oﬀer tailored solutions done with agile methods.
› How to spend the UX budget? Overview › What to focus on if not on user studies? › Discussion points
20 000 € …to make a newsreader for iOS. How should we spend it?
› Development › Estimated 12k€ Budget › Management breakdown › Estimated 3k€ › UX design & research › What is left is… 5k€.
Typical choices for user research. » Focus groups » 1 FG, 2 desgnrs, 4 days of work = 6400€ » Observations » 5 usrs, 1 designer, 5 days of work = 4000€ » User interviews » 5 usrs, 1 designer, 5days of work = 4000€ » PD workshops » 1 workshop, 2 designers, 3 days = 4800€ » Stakeholder workshops » 1 workshop, 2 designers, 3 days = 4800€ » etc. » Our UX budget is 5000€ and above we have allocated no design yet! One person day = 800€ in budget
› Which is more critical a “Let’s cut the working software or UX? Really. developmet budget?” › Interviewing 5 people would be 33% of the development budget.
› Sure… but does that really save money here? “Let’s scope it › User studies do not scale down: Running focus groups down to make or workshops takes few days it possible” each anyway. › We have to scope this down anyway to do it only with 20k €.
› Yes the UX budget is too small. “The UX budget › Nevertheless, that is what is too small!” the customer was willing to pay. This is their budget. “We should not › Should we take the project, then? accept this › Can we make the project!” newsreader with a 5k€ UX budget? With no user research! Futurice 6/12/12 11
› Is this the ﬁrst newsreader app in the world? › Have we done newsreader apps before? Should we do › Does the customer have this already the UX for 5k€? done on some other platform? › Are we users ourselves? In ~6 days? › Is this a familiar customer? › What features does the customer want? How simple is it? › Are we familiar with the platform UI guidelines?
World of a diﬀerence: ~5k€ ~50k€ Porting an existing app. Creating from scratch. “You guys did a great job with the Android newsreader for our company. “We want something previously unseen in the world of newsreaders. We do not Could you do the same reader for iOS know what it is. It will be a tool for stock as well? It will be for public use to market analysts and investors. This accompany our newspaper.” would be a good ﬁrst project between our companies.”
Our 25 latest mobile cosnumer apps are rated 4.2 in average. Amount of user studies varies, but it is never what schoolbooks would expect.
How is it possible to make successful software without emphasizing user studies?
• Observation • Literature • Existing studies • Focus groups Eliciting • Design for myself user reqs • Common sense User • Social distance participa Designer as user • Lead users tion What shapes • Participatory design the product, • Professional • User testing and its UX experience design? Designer Existing • Existing trends as • Shared visions among cultural • UI genres professio designers • Metaphores and genres Tech & nal associations • Use practices compone nts • Existing UI components • Existing platform APIs • Existing content Adapted from Hyysalo 2011.
Bechmark competition and reverse engineer user reqs. Ask the customer to tell about the users. Eliciting Do we have speciﬁc user domain user reqs knowledge? How much does it cost? User Does the customer know their Is it worth it? participa Designer customers? User testing is often worth it. as user Are we gathering user data tion What shapes after launch?. the product, and its UX How experienced is our designer? design? Designer Existing Have we done these kinds of as products/services? Is this rocket science? cultural professio Are we learning enough? What is the “metaphor” of the genres design? Tech & nal Are we using well-‐known compone conventions? nts What is the platform? What is the back-‐end? What are the UI guidelines? What are the customer’s brand Adapted from Hyysalo 2011. guidelines?
New ideas Familiar Consumers Plenty of Customer Complex Large Strategy domain user data new to agile service organization documents 5 5 4 4 3 3 2 2 1 1 0 0 1 1 2 2 3 3 4 4 5 5 Concept Unknown Speciﬁc No user Customer Single Small Gfx for ready domain users data knows agile functionality organization coding 18
New ideas Familiar Consumers Plenty of Customer Complex Large Strategy domain user data new to agile service organization documents 5 p” 5 4 “Mobile ap 4 3 3 2 2 1 1 0 0 1 1 2 2 “ERP 3 app” 3 4 4 5 5 Concept Unknown Speciﬁc No user Customer Single Small Gfx for ready domain users data knows agile functionality organization coding 19
New ideas Familiar Consumers Plenty of Customer Complex Large Strategy domain user data new to agile service organization documents 5 5 4 4 3 3 2 2 1 1 0 0 1 1 2 2 3 3 4 4 5 5 Concept Unknown Speciﬁc No user Customer Single Small Gfx for ready domain users data knows agile functionality organization coding The 20k€ newsreader 20
› Is the designer also a user? › Has the designer done newsreaders? › Has the designer worked with this customer? › Does iOS have clear guidelines? › Does the customer have clear visual & brand guidelines? › Is a newsreader app a common product? › Can we re-‐use the design from existing products? › Can the customer give us good feedback on behalf of the end-‐users? › Can we do the UX design for 5000€? › Sure we can!
› User studies are expensive. They are not always worth the cost. › Take advantage of other factors shaping the UX: › The designer as a user. Key points › The designer as a professional with experience. › Existing UI patterns, guidelines & components. › Obvious use patterns and practices. › Consulting other people who know the users (e.g., product owner)
User Studies are a Waste of Money… …under speciﬁc circumstances.
Discussion point 1: We need more understanding on what shapes the design of technology and how to apply that into design.
The Multi-‐Dimensional Space Experience of designer Enterprise system for XY ERP for X Mobile app for ZZ Online service for WZ Web page for G ERP for XXY # of featur Enterprise system for XX es Domain familiarity Mobile app for XY Online service for X ERP for XXY Mobile app for ZZ ERP for Z
Theory Practice • Social construction of technology • Ethnography, ethnomethods • Qualitative & quantitative approaches • Use practices, domestication of tech • Design research This should be a rapid cycle! • Empirical data on projects • Case studies • Real world context • Views of professionals • Reality check
Discussion point 2: Real user knowledge is created after launch. From waterfall to agile/lean.
User studies as guesswork Business Research Develop Mainten objectives & Design ment ance x months Several user studies 6/12/12 based on actual use 29