How to design with science and not destroy the magic

@mrjoe How to design with science January 8, 2015 Joe Leech Waving einstein How to design with science and not destroy the magic @mrjoe

Hi, I’m @mrjoe but you can call me Joe

@mrjoe ⅓ ⅓⅓ Research DesignDigital Strategy

@mrjoe [Newton] has destroyed all the poetry of the rainbow, by reducing it to the prismatic colours. – John Keats

@mrjoe The product goes live and it's lost it’s magic The team / client continually questioning your choices Being micromanaged by your boss / client Ever heard these issues? Hands-up? I had the ﬁrst one so badly for about a year, I called it the curse of Joe.

@mrjoe It started with a need 6 http:// www.cxpartner s.co.uk/cxblog/ the_myth_of_th e_page_fold_ev idence_from_us er_testing/ It started with a need. I had heard the same thing again and again. I wrote this, have you read it? Almost half a million of you have.

@mrjoe Here’s the data, from an eyetracking study. It shows 2 designs, one to the left with stuﬀ crammed above the fold. The one on the right where things are spaced out more. People explored more when there was less stuﬀ crammed above the fold. It showed me there was a need for evidence, reasoning and science to support the design process.

@mrjoe Design is not so much a design issue as a power struggle. – Alan Cooper Alan cooper, created visual basic amongst other things, teaches design leadership. Harsh words! So what do we do?

@mrjoe Design is the process of choosing and organising code words, images and messages into a form that communicates and inﬂuences its audience – @mrjoe + The UK Design council

Talk stories.

@mrjoe Here’s some research I did many years ago for the trainline. What does this photo show? This one photo saved countless time for thousands of people.

@mrjoe 2. 1. It tells us two things. 1. When people travel they are often carrying loads of stuﬀ 2. most people have a phone and are looking for the conformation email. They don’t print it out. We took the conﬁrmation number and put it as the email subject line.

@mrjoe I didn’t see them [the asterisks]. There’s nothing that explains what they mean. How did you hear about us? * How old are you? * I’ve heard this a few times in user research. What does it mean? Not enough people to make it statistically signiﬁcant but enough to change my approach to forms.

@mrjoe How did you hear about us? How old are you? Optional We don’t mark mandatory, we mark optional. It’s enough to get me to change my design approach. Is there enough evidence to prove it from just a handful of users? No, it requires another set of supporting data. Which is why we need to use each approach in combination with another.

Talk business.

@mrjoe h‫מּ‬p://www.slate.com/articles/technology/technology/2014/06/ online_advertising_effectiveness_for_large_brands_online_ads_may_be_worthless.single.html? utm_content=bufferfa367&utm_medium=social&utm_source=twi‫מּ‬er.com&utm_campaign=buffer http://carolinavega.com/web%20documents%20and%20pics/oreo.jpg In 2003 Mel Karmazin, a Viacom executive. Viacom are a huge media company where the bulk of their income is from advertising, visited the Google offices. He met Sergey and Larry who preceded to tell him all the ways they could measure and track advertising online. His one comment:

@mrjoe You’re f---ing with the magic! – Mel Karmazin, Viacom executive h‫מּ‬p://www.slate.com/articles/technology/technology/2014/06/ online_advertising_effectiveness_for_large_brands_online_ads_may_be_worthless.single.html? utm_content=bufferfa367&utm_medium=social&utm_source=twi‫מּ‬er.com&utm_campaign=buffer http://carolinavega.com/web%20documents%20and%20pics/oreo.jpg

@mrjoe http:// justinmrao.com/ lewis_rao_nearim possibility.pdf On the Near Impossibility of Measuring the Returns to Advertising⇤ Randall A. Lewis Google, Inc. ralewis@google.com Justin M. Rao Microsoft Research justin.rao@microsoft.com April 23, 2013 Abstract Classical theories of the firm assume access to reliable signals to measure the causal impact of choice variables on profit. For advertising expenditure we show, using twenty-five online field experiments (representing $2.8 million) with major U.S. retailers and brokerages, that this assumption typically does not hold. Statistical evidence from the randomized trials is very weak because individual-level sales are incredibly volatile relative to the per capita cost of a campaign—a “small” impact on a noisy dependent variable can generate positive returns. A concise statistical argument shows that the required sample size for an experiment to generate su ciently informative confidence intervals is typically in excess of ten million person-weeks. This also implies that heterogeneity bias (or model misspecification) unaccounted for by ob- servational methods only needs to explain a tiny fraction of the variation in sales to severely bias estimates. The weak informational feedback means most firms cannot even approach profit maximization. This paper was published. The study was undertaken written when Lewis and Rao were at Yahoo. What does this tell us? There is no proof this works. It’s built on faith.

@mrjoe The 5 basic business models. 19 http://mrjoe.uk Advertising Aﬃliate feesSelling Subscription / Licensing Market place What does that mean for you. You need to understand some business basics. I wrote an article. more coming soon. These are the 5 basic income streams for organisations. Understand how they generate money and you have more science to base your designs on. Read more at http://mrjoe.uk

@mrjoe Strong ideas, weakly held. http://mrjoe.uk/strong-ideas-weakly-held/

Talk data.

@mrjoe 22 5 years ago. I was a single man. At a point in life where it was becoming harder to meet new people. Someone suggested internet dating. I thought wow, ok, I can do this. My UX background is perfect for meeting the perfect person.

@mrjoe http://blog.okcupid.com/ index.php/the-4-big-myths-of- profile-pictures/ I went in search of data, OK Cupid has a blog. Full of great insights, like what is the perfect profile picture. I set about getting all UX on it, user researching my profile. At one point I emailed OK Cupid and asked if I could drop a google analytics tag my page. I learned a lot about how data can inform design. I also met my wife.

@mrjoe Classic -Conversion (sales / visits) -Product page conversion (sales / visits to product page) -Funnel analysis -Basket abandonment -Bounce rate -Sales -Leads -Subscribers -Unique visitors -Returning visitors -Page views per visit -Visit to order ratio -Load time -Registrations -Visit / session length -Page views per visit -Time on page -Time on site -Form abandonment -Failed internal search -Referring pages / links -Geographic locations -Print page Banners: -Click through rate -Impressions Financial / sales / business -Average order value -Basket value -Profit margin -Average sales price -Cross sell -Gross margin -Category margin -Cost per lead -Customer acquisition cost -Lifetime customer value -Average customer value -Membership / subscription churn RSS -Feedburner subscriptions -Shares on Google Reader Call centre calls -Average call length -Support vs sales calls -Inbound vs outbound calls -Web generated calls (unique number on website) -Web fulfilled information calls SEO -SEM keyword value -SEO positioning -Changes in SERP results/ rankings -Top entry pages -Number of keywords triggering results for your site -Number of clicks to your site from keywords -Google trends -Inbound links (back link discovery) -Percentage share of each engine -Branded vs non-branded searches -Affiliate links -Affiliate fees Social media -Facebook referrals -Incoming Twitter links -Facebook sends/shares/ mentions -Facebook likes -Facebook fans -Facebook fan rates -Tweets -Retweets -@s on Twitter -Twitter followers -Twitter follow rate -Google +1s -Bookmarks on Delicious -StumbleUpon thumbs up -StumpleUpon reviews -Diggs -Google BlogSearch links -Blog comments -Blog articles -Video views -Youtube favourites -Youtube channel subscriptions -Youtube channel comments -Youtube video reviews -Slideshare views -Forum mentions -Thread size -Online review mentions -Stars in reviews -Bit.ly / URL shorteners usage / clicks Third party / benchmarks -Comscore -Hitwise -Alexa -Compete (US) Email & campaigns -Email newsletter churn -Email sign-up -Emails sent -Emails bounced (bad address) -Email forwards -Email campaigns -Open rate -Delivery rate (sent - bounces) -Click through rate (CTR) -Email related to conversion / other metric -Unsubscribe rate -SMS subscribers -Via print publication / 3rd party (unique URLs) Internal search -Search no results -# Search 1 - 10 results -# Search 10 - 25 results -# Search over 50 results IA: -GOMS technique -Time to content -Clicks to content -Back button clicks -Task completion -Errors Testing metrics: -Words Recognition Rate -Reported expectations and performance -Facial reaction -Number of back presses -Gap satisfaction -Path's taken as a measure of scent -Work-flow matches mental model or not Satisfaction measurements: -Net Promoter Score -ASQ PDF: After Scenario Questionnaire (3 Questions) -NASA-TLX : NASA's task load index is a measure of mental effort (5 Questions) -SMEQPDF: Subjective Mental Effort Questionnaire -UMEPDF : Usability Magnitude Estimation -SEQ PDF: Single Ease Question A bewildering amount of metrics. Here’s some I collected over a couple of years. This can be dangerous, organisations can be tied to data, refusing to make a decision without it.

@mrjoe I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that… … I won’t miss a design philosophy that lives or dies strictly by the sword of data. – Doug Bowmen http://stopdesign.com/archive/2009/03/20/goodbye-google.html This is taken from Doug Bowmen’s blog, after he left Google, he was their ﬁrst visual designer. Pretty damning.

@mrjoe Classic -Conversion (sales / visits) -Product page conversion (sales / visits to product page) -Funnel analysis -Basket abandonment -Bounce rate -Sales -Leads -Subscribers -Unique visitors -Returning visitors -Page views per visit -Visit to order ratio -Load time -Registrations -Visit / session length -Page views per visit -Time on page -Time on site -Form abandonment -Failed internal search -Referring pages / links -Geographic locations -Print page Banners: -Click through rate -Impressions Financial / sales / business -Average order value -Basket value -Profit margin -Average sales price -Cross sell -Gross margin -Category margin -Cost per lead -Customer acquisition cost -Lifetime customer value -Average customer value -Membership / subscription churn RSS -Feedburner subscriptions -Shares on Google Reader Call centre calls -Average call length -Support vs sales calls -Inbound vs outbound calls -Web generated calls (unique number on website) -Web fulfilled information calls SEO -SEM keyword value -SEO positioning -Changes in SERP results/ rankings -Top entry pages -Number of keywords triggering results for your site -Number of clicks to your site from keywords -Google trends -Inbound links (back link discovery) -Percentage share of each engine -Branded vs non-branded searches -Affiliate links -Affiliate fees Social media -Facebook referrals -Incoming Twitter links -Facebook sends/shares/ mentions -Facebook likes -Facebook fans -Facebook fan rates -Tweets -Retweets -@s on Twitter -Twitter followers -Twitter follow rate -Google +1s -Bookmarks on Delicious -StumbleUpon thumbs up -StumpleUpon reviews -Diggs -Google BlogSearch links -Blog comments -Blog articles -Video views -Youtube favourites -Youtube channel subscriptions -Youtube channel comments -Youtube video reviews -Slideshare views -Forum mentions -Thread size -Online review mentions -Stars in reviews -Bit.ly / URL shorteners usage / clicks Third party / benchmarks -Comscore -Hitwise -Alexa -Compete (US) Email & campaigns -Email newsletter churn -Email sign-up -Emails sent -Emails bounced (bad address) -Email forwards -Email campaigns -Open rate -Delivery rate (sent - bounces) -Click through rate (CTR) -Email related to conversion / other metric -Unsubscribe rate -SMS subscribers -Via print publication / 3rd party (unique URLs) Internal search -Search no results -# Search 1 - 10 results -# Search 10 - 25 results -# Search over 50 results IA: -GOMS technique -Time to content -Clicks to content -Back button clicks -Task completion -Errors Testing metrics: -Words Recognition Rate -Reported expectations and performance -Facial reaction -Number of back presses -Gap satisfaction -Path's taken as a measure of scent -Work-flow matches mental model or not Satisfaction measurements: -Net Promoter Score -ASQ PDF: After Scenario Questionnaire (3 Questions) -NASA-TLX : NASA's task load index is a measure of mental effort (5 Questions) -SMEQPDF: Subjective Mental Effort Questionnaire -UMEPDF : Usability Magnitude Estimation -SEQ PDF: Single Ease Question Measure 2 to 3 things. Each has: 1. Timescale 2. Benchmark 3. Reason to be reported 4. Associated action http://mrjoe.uk/ux-and-roi-what-to-measure-and-what-to-expect/ Truth is it’s important to measure. It shows confidence in your design. Here’s some tips to get it right. See a talk by me on this here http://mrjoe.uk/ ux-and-roi-what-to-measure-and-what-to-expect/

@mrjoe 27 Data shows conﬁdence in the design

@mrjoe http://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_squirrel/ The squirrel evolved bit by bit to fly. Small incremental improvements that allowed him to jump further, land better, and eventually to glide. The squirrel has reached a point where he isn’t going to get better at flying (called a local maximum).

@mrjoe The soaring eagle. Elegantly drifting through the air with little effort. The eagle also evolves, little by little to its own maxim. It’s a superb flyer, soaring to 15,000 feet instead of making long leaps between trees. The eagle started from a better set of basics: front limbs to wings, reptile scales to feathers. Good design gives a good starting point for evolution. MVT can get you to a maxim, but it can’t help you choose the best starting point to get to the highest maxim. A strong design leads to elegant flight. A weak design leads to hopping between branches. We need to combine MVT with craft. Read more: http://www.cxpartners.co.uk/ cxblog/ multi_variant_testing_and_the_flying_

@mrjoe Sore like an eagle don’t leap like a squirrel http://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_ﬂying_squirrel/

Talk theory.

@mrjoe James Vicary, in 1957, ﬂashed up hungry eat popcorn, thirsty drink coke as single frames during a movie showing.

@mrjoe 58%A reported 58% increase in sales of popcorn. But it was later found out to be a hoax. He made it up to prop up his failing advertising business.

@mrjoe Some examples, but there is no proof. http://www.badabingrecords.com/2011/03/the-subliminal- scares-a-short-history-of-an-american-obsession/ http://www.businessinsider.com/subliminal-ads-2011-5?op=1

@mrjoe Strikingly, there was no significant effect. http://www.bbc.co.uk/news/magazine-30878843 The media’s obsession. They were shown the same three- minute clip from the BBC/Kudos drama Spooks twice, but each time one group wore the blindfolds. The clip watched by the group with red blindfolds contained a 10-millisecond flash, every five seconds, of the word “Lipton", http:// www.bbc.co.uk/news/magazine-30878843 So what can we do?

@mrjoe There are resources out there that help with theory. This is from Ribot, alumni of my psychology for designers workshop. http://coglode.com/ Workshop: http://mrjoe.uk/psychology-design-ux- workshop/

@mrjoe PsychologyForDesigners.com Or you can buy my book, I show you how to ﬁnd a theory and apply it to design. How to use psychology to advocate design. PsychologyForDesigners.com

@mrjoe Title Text 38 A designer who doesn't understand psychology is going to be no more successful than an architect who doesn't understand physics Image: http://victorenrich.com/archives/155 @mrjoe

@mrjoe The biggest myth ever perpetuated in the design ﬁeld is that good design sells   itself. – Mike   Montiero

Talk objectively.

@mrjoe 'I feel the design, minimalist in style, reﬂects a modern design aesthetic’ - The Designer

@mrjoe ‘Well I feel more that it’s a website and not a f#cking coﬀee table.’ - The Product Manager

@mrjoe 'I feel the design, minimalist in style, reﬂects a modern design aesthetic’ - The Designer What was wrong with that statement? Anytime you say I feel, it leaves the door open for the ‘Well I feel’. It becomes a battle of wills, there is no subjectivity to judge the design against.

@mrjoe http://www.gensleron.com/storage/post-images/cities/reinventingaclassicreuniontowerobservationdeckreopens/1.jpg Which reminds me, I was once looking over a beautiful view, but I got annoyed at the poor information design of the support panel. My wife simply said, chill the f$ck out and enjoy the view, the view is the focus not the design of the information panel.

@mrjoe Art: …works to be appreciated primarily for their beauty or emotional power. – Merriam Webster Dictionary

@mrjoe http:// www.ise.bgu.ac.il/ faculty/noam/papers/ 00_nt_ask_di_iwc.pdf Art does support design. Pretty things are easier to use.

@mrjoe What is beautiful is usable.

1. Talk stories 2. Talk business 3. Talk data 4. Talk theory 5. Talk objectively

@mrjoe mrjoe.uk