For some UX/design teams, anything that comes out of analytics, research, and testing is suspect – like a mugger that could rob us of creativity and innovation. There are traps to avoid so that data doesn’t imprison design. So how do we avoid the traps and instead use data for good? Come find out how NPR thinks about the intersection of data and design, and how we are using lean, data-informed processes to experiment with the future of digital listening.
28. 28
Yes, this sounds like Lean Startup thinking
Assumption/hypoth
esis
Minimum Viable
Product
Analytics, researc
h, testing
29. 29
Designing choice in the new NPR app
Original design: Give
users more
control/choices
Hypothesis to test:
Fewer immediate
choices + simplicity
= longer listening
Results: Listening
time up 8%
UX and design community: fundamental fear and skepticism of research & metricsCan feel like data’s purpose is to mug design and steal creativity from us, restrict what we dohttp://www.flickr.com/photos/spencerbbclark/6483073463/
False:Research = doing what users tell you to doDriving directions, prescriptive, restrictiveLycos marketing survey on colors
Sony boombox colorWhat users say isn’t always what they do
Or measuring only one thingEverything hinges on what you decide to measurehttp://www.adprint.ro/files/executions/8/356_topinterior_mag_sofa_low.jpg_900.jpg
Wisconsin grocery chain decided to use IPM (items per minute) as key metric for cashier productivity. Faster checkout should mean shorter lines, happier customers, more profit.But IPM is only tracked when a lane is open and a cashier’s terminal is ready. So cashiers learned to put their terminals in secure state to stop the clock – between customers, while bagging, etc. IPM score goes up, but terminal on and off slows down actual checkout times.*Unintended consequences*http://innovationedge.com/2010/05/10/dangerous-metrics-for-innovation/
Former Soviet Union: managers of glass plants rewarded by tons of sheet glass produced. Most plants produced glass so thick it was nearly opaque. Changed to square meters of glass. Led to glass so thin that it was easily broken.For us: Session conversion vs longer-term loyalty effectsUpsellsLeads to false optimization, taking us in the wrong directionhttp://compliance.saiglobal.com/community/know/blogs/item/4392-unintended-consequences-and-perverse-incentives
Ifsomethingcanbe counted, it’smorelikely to be used tomeasuresuccessWe choosewhat we measure – and whatmetrics we lookat and use the mosthttp://www.flickr.com/photos/60852569@N00/2204909532/
Incentives drive behavior change and cultureThey affect what we work on and who we becomeWhat gets counted in your organization?What numbers do you praise people for or chastise them for?How does that change behavior and impact what you do and who you are?http://www.flickr.com/photos/jon_tucker/2446567442/
Instead of using our creative skills, it’s just about optimization and making very small adjustmentsBecoming machinists, tending to the robots that make decisionsCult of A/B testingDoug at Google on testing blueshttp://www.flickr.com/photos/seattlemunicipalarchives/4089729743/
What is it about A/B testing that makes it so easy to fall into this trap?
It’s the baseline we start from, not the endIt provides context and constraints that focus us before freeing ushttp://www.flickr.com/photos/bfishadow/4407857237/
Opportunity: reaching new audience where they liveChallenge: More competitors for “earshare” than ever before
Channel usage by segmentLikelihood to consider public radioBrand perception and affinity:NPRvs station vsprogramBUT all this data doesn’t tell us exactly what to do
Smartphone app works with Ford Sync, voice control
App in private betaMobile version of NPR Infinite PlayerRemixing public radio content, algorithms and editorial curation for a lean-back or lean-forward experience
Measure how high we've jumped (and whether it's worth continuing investment) It should inspire us to reach higherWithout overly limiting our design choiceshttp://www.flickr.com/photos/10710442@N08/8265087493/in/photolist-dAmJyv-aoYjU9-aoYk3q-LJAUR
Initially: manual process – concierge MVP
Mindset: the goal is learningThere is no “finished”http://www.deviantart.com/morelikethis/artists/198956566?view_mode=2
Flips the product development model on its head (build first, then figure out what to measure)
Findability and discoverabilityTesting alone didn’t answer the questionBut it gave us insights so WE could answer it
Falling into the cult of datavsFalling into the cult of egoWithout data, design is just guessingWhy is it so hard for teams to walk the narrow path?
Data doesn’t give us the answers, it helps us ask better questions