1.
Data-driven Design Making Software & Websites Better Through Use of Statistics Patrick McKenzie April 15th, 2010 Creative Commons Licensed CC-BY. Please feel free to share.

2.
Outline <ul><li>Introduction </li></ul><ul><li>Art & Science </li></ul><ul><li>Rationale </li></ul><ul><li>Instrumentation </li></ul><ul><li>Funnel Analysis </li></ul><ul><li>A/B Testing </li></ul><ul><li>Conclusion </li></ul><ul><li>自己紹介 </li></ul><ul><li>芸術と 技術 </li></ul><ul><li>目的 </li></ul><ul><li>動作監視 </li></ul><ul><li>ファネル分析 </li></ul><ul><li>A/Bテスト </li></ul><ul><li>まとめ </li></ul>

3.
D:Downloads2264763977_fbeb2e34ba.jpg The Plan

4.
What Happened D:DropboxMy DropboxPhotosSakura Season 2010IMG_0348.JPG

5.
D:Downloads3452417503_88c32d1397.jpg What Happened Next

6.
Started Business

7.
Art into Science 芸術から技術へ Supplement the vision and craft of designers with the logic, regimentation, and process of science. デザイナーの芸術を技術の倫理、数学、体験方式で拡張する。

8.
Rationale 目的 <ul><li>Decrease Costs </li></ul><ul><li>Decrease Uncertainty </li></ul><ul><li>Increase Satisfaction </li></ul><ul><li>Increase Success </li></ul><ul><li>Increase Profits </li></ul><ul><li>開発工数ダウン </li></ul><ul><li>不確実性ダウン </li></ul><ul><li>ユーザ満足度アップ </li></ul><ul><li>成功率アップ </li></ul><ul><li>利益アップ </li></ul>

9.
Designing With Data <ul><li>Get Facts to Answer The Questions: </li></ul><ul><ul><li>What are users doing? 何をしているか？ </li></ul></ul><ul><ul><li>Which users are doing things? 誰がしているか？ </li></ul></ul><ul><ul><li>Where are they doing things? 何処にしているか？ </li></ul></ul><ul><ul><li>(Maybe) Why are they doing it? 何故しているか？ </li></ul></ul>

10.
Types of Instrumentation <ul><li>Analytics software </li></ul><ul><li>Heatmapping </li></ul><ul><li>Surveys </li></ul><ul><li>Ad hoc analysis (Excel, etc) </li></ul><ul><li>Custom software </li></ul>

11.
CrazyEgg

12.
Landing Page <ul><li>Photo of problem solved goes here </li></ul>

13.
Landing Page (Fixed)

14.
Paving The Trails <ul><li>Picture of paving a path </li></ul>

15.
Instrumenting Software

16.
Instrumenting Software #2

17.
Funnel Analysis ファネル分析 <ul><li>Many applications have several steps followed by a goal. </li></ul><ul><li>Only a percentage of users clear each step. </li></ul><ul><li>Funnel analysis finds where folks are falling out, to concentrate efforts on improvement </li></ul>Viewed Landing Page Signed Up For Trial Did Work $$

18.
Software ソフト <ul><li>Google Analytics </li></ul><ul><ul><li>Free but limited </li></ul></ul><ul><ul><li>Good for starter </li></ul></ul><ul><li>Mixpanel </li></ul><ul><ul><li>Cheap </li></ul></ul><ul><ul><li>API </li></ul></ul><ul><ul><li>Harder to use </li></ul></ul>

19.
Define Funnel <ul><li>Signs In </li></ul><ul><li>Opens Word List </li></ul><ul><li>Saves Word List </li></ul><ul><li>Customizes Cards </li></ul><ul><li>Creates Cards </li></ul><ul><li>Downloads Card </li></ul>

20.
Collect Data <ul><li>Signs In: 260 </li></ul><ul><li>Opens Word List: 218 (83%) </li></ul><ul><li>Saves Word List: 191 (87%) </li></ul><ul><li>Customizes Cards 161 (84%) </li></ul><ul><li>Creates Cards: 131 (81%) </li></ul><ul><li>Downloads Card 126 (95%) </li></ul>Total Conversion Rate 126 / 260 = ~48.4%

21.
Identify Areas for Improvement <ul><li>Signs In: 260 </li></ul><ul><li>Opens Word List: 218 (83%) </li></ul><ul><li>Saves Word List: 191 (87%) </li></ul><ul><li>Customizes Cards 161 (84%) </li></ul><ul><li>Creates Cards: 131 (81%) </li></ul><ul><li>Downloads Card 126 (95%) </li></ul>Total Conversion Rate 126 / 260 = ~48.4%

22.
Implement Improvements <ul><li>Open Word List needlessly complicated: make easier. </li></ul><ul><li>Save Button was unclear: tweak microcopy, button design. </li></ul><ul><li>Customize Cards needlessly complicated: show less options. </li></ul>

23.
Before

24.
After

25.
Collect Metrics <ul><li>Signs In: 194 </li></ul><ul><li>Opens Word List: 155 (79%) </li></ul><ul><li>Saves Word List: 151 (97%) </li></ul><ul><li>Customizes Cards: 127 (84%) </li></ul><ul><li>Creates Cards: 112 (88%) </li></ul><ul><li>Downloads Card 126 (98%) </li></ul>Up from 83% Up from 81% Total Conversion Rate goes from 48.5% to 56.7% 16.6% lift in customer success / revenue for 1 hour of work .

26.
http://farm1.static.flickr.com/21/33346213_e5d5842024.jpg Pretend You Own A Bar Salty Peanut by EuroMagic. Half of Customers Get... Half of Customers Get...

27.
A/B Testing <ul><li>Divide similar population into two groups randomly. </li></ul><ul><li>Treat the groups differently. </li></ul><ul><li>Measure difference in outcomes. </li></ul><ul><li>Scientific Method applied to design . Eliminates bias, randomness, poor test design. </li></ul>

28.
Actual A/B Test Half of Users See... Half of Users See...

29.
Software Tools <ul><li>Google Website Optimizer </li></ul><ul><ul><li>Terrible. Do not use. </li></ul></ul><ul><li>Visual Website Optimizer </li></ul><ul><ul><li>visualwebsiteoptimizer.com </li></ul></ul><ul><ul><li>WYSIWYG alternative creation. Very nice. </li></ul></ul><ul><ul><li>Free for now. </li></ul></ul><ul><li>A/Bingo or similar </li></ul><ul><ul><li>Rails A/B testing library. </li></ul></ul><ul><ul><li>Programmer friendly, powerful, easy to use. </li></ul></ul>

30.
Core A/B Testing Loop <ul><li>Identify conversion and instrument. </li></ul><ul><li>Create alternatives. </li></ul><ul><li>Assign users randomly to alternatives </li></ul><ul><li>Watch behavior. </li></ul><ul><li>Identify winning alternative. </li></ul><ul><li>Iterate, iterate, iterate... </li></ul>

31.
Landing Pages experianvasm experianvbsm Source: whichtestwon.com

32.
Signup Forms http://www.bingocardcreator.com/blog-images/signup-ab-tests/old-boring-signup.png

33.
Signup Forms http://www.bingocardcreator.com/blog-images/signup-ab-tests/new-hotness-signup.png

34.
Signup Forms Vast contact dealer form

35.
New Features http://farm3.static.flickr.com/2422/3981364314_d4b30cb739.jpg <ul><ul><li>60 ~ 80% of features do not change user behavior </li></ul></ul><ul><ul><li>Eric Ries </li></ul></ul>

36.
You can quickly create bingo cards using our included wizards! Old Version

37.
C:UserspatrickPictures
ew-version.png Spot The New Feature

38.
Core A/B Testing Discipline <ul><li>Accept that you do not know what is right. ( Hardest lesson. ) </li></ul><ul><li>Test frequently, as a matter of routine. (Helps to have one-line testing.) </li></ul><ul><li>Test changes in isolation. </li></ul><ul><li>Use evidence from tests to inform your marketing and development plans. </li></ul>

39.
Conclusion <ul><li>What we learned today: </li></ul><ul><ul><li>Importance of tracking user interaction. </li></ul></ul><ul><ul><li>Funnel analysis – measuring user success and goal completion. </li></ul></ul><ul><ul><li>A/B tests – improving marketing message and product design. </li></ul></ul>

40.
<ul><li>Thanks for listening. ご清聴、ありがとうございました。 </li></ul><ul><li>[email_address] ← My email. </li></ul><ul><li>http://www.kalzumeus.com ← My blog. </li></ul><ul><li>http://www.abingo.org ← A/Bingo software (OSS) </li></ul><ul><li>http://www.startuplessonslearned.com ← Eric Ries, very useful. </li></ul>

41.
Microsoft logo: http://www.flickr.com/photos/scobleizer/2264763977/ Sleeping Salaryman: http://www.flickr.com/photos/kansai/3452417503/ Edamame (Green Soybeans): http://www.flickr.com/photos/kankan/33346213/ Salty Peanuts: http://www.flickr.com/photos/euromagic/2351628831/ Facepalm Statue: http://www.flickr.com/photos/cesarastudillo/3981364314/ Photo Credits