Data-Driven Software Design

Data-driven Design Making Software & Websites Better Through Use of Statistics Patrick McKenzie April 15th, 2010 Creative Commons Licensed CC-BY. Please feel free to share.

Outline
Introduction
Art & Science
Rationale
Instrumentation
Funnel Analysis
A/B Testing
Conclusion
自己紹介
芸術と技術
目的
動作監視
ファネル分析
A/Bテスト
まとめ

D:Downloads2264763977_fbeb2e34ba.jpg The Plan

What Happened D:DropboxMy DropboxPhotosSakura Season 2010IMG_0348.JPG

D:Downloads3452417503_88c32d1397.jpg What Happened Next

Started Business

Art into Science 芸術から技術へ Supplement the vision and craft of designers with the logic, regimentation, and process of science. デザイナーの芸術を技術の倫理、数学、体験方式で拡張する。

Rationale 目的
Decrease Costs
Decrease Uncertainty
Increase Satisfaction
Increase Success
Increase Profits
開発工数ダウン
不確実性ダウン
ユーザ満足度アップ
成功率アップ
利益アップ

Designing With Data
Get Facts to Answer The Questions:
What are users doing? 何をしているか？
Which users are doing things? 誰がしているか？
Where are they doing things? 何処にしているか？
(Maybe) Why are they doing it? 何故しているか？

Types of Instrumentation
Analytics software
Heatmapping
Surveys
Ad hoc analysis (Excel, etc)
Custom software

CrazyEgg

Landing Page
Photo of problem solved goes here

Landing Page (Fixed)

Paving The Trails
Picture of paving a path

Instrumenting Software

Instrumenting Software #2

Funnel Analysis ファネル分析
Many applications have several steps followed by a goal.
Only a percentage of users clear each step.
Funnel analysis finds where folks are falling out, to concentrate efforts on improvement
Viewed Landing Page Signed Up For Trial Did Work $$

Software ソフト
Google Analytics
Free but limited
Good for starter
Mixpanel
Cheap
API
Harder to use

Define Funnel
Signs In
Opens Word List
Saves Word List
Customizes Cards
Creates Cards
Downloads Card

Collect Data
Signs In: 260
Opens Word List: 218 (83%)
Saves Word List: 191 (87%)
Customizes Cards 161 (84%)
Creates Cards: 131 (81%)
Downloads Card 126 (95%)
Total Conversion Rate 126 / 260 = ~48.4%

Identify Areas for Improvement
Signs In: 260
Customizes Cards 161 (84%)
Total Conversion Rate 126 / 260 = ~48.4%

Implement Improvements
Open Word List needlessly complicated: make easier.
Save Button was unclear: tweak microcopy, button design.
Customize Cards needlessly complicated: show less options.

Before

Collect Metrics
Signs In: 194
Customizes Cards: 127 (84%)
Up from 83% Up from 81% Total Conversion Rate goes from 48.5% to 56.7% 16.6% lift in customer success / revenue for 1 hour of work .

http://farm1.static.flickr.com/21/33346213_e5d5842024.jpg Pretend You Own A Bar Salty Peanut by EuroMagic. Half of Customers Get... Half of Customers Get...

A/B Testing
Divide similar population into two groups randomly.
Treat the groups differently.
Measure difference in outcomes.
Scientific Method applied to design . Eliminates bias, randomness, poor test design.

Actual A/B Test Half of Users See... Half of Users See...

Software Tools
Google Website Optimizer
Terrible. Do not use.
Visual Website Optimizer
visualwebsiteoptimizer.com
WYSIWYG alternative creation. Very nice.
Free for now.
A/Bingo or similar
Rails A/B testing library.
Programmer friendly, powerful, easy to use.

Core A/B Testing Loop
Identify conversion and instrument.
Create alternatives.
Assign users randomly to alternatives
Watch behavior.
Identify winning alternative.
Iterate, iterate, iterate...

Landing Pages experianvasm experianvbsm Source: whichtestwon.com

Signup Forms http://www.bingocardcreator.com/blog-images/signup-ab-tests/old-boring-signup.png

Signup Forms http://www.bingocardcreator.com/blog-images/signup-ab-tests/new-hotness-signup.png

Signup Forms Vast contact dealer form

New Features http://farm3.static.flickr.com/2422/3981364314_d4b30cb739.jpg
60 ~ 80% of features do not change user behavior
Eric Ries

You can quickly create bingo cards using our included wizards! Old Version

C:UserspatrickPictures ew-version.png Spot The New Feature

Core A/B Testing Discipline
Accept that you do not know what is right. ( Hardest lesson. )
Test frequently, as a matter of routine. (Helps to have one-line testing.)
Test changes in isolation.
Use evidence from tests to inform your marketing and development plans.

Conclusion
What we learned today:
Importance of tracking user interaction.
Funnel analysis – measuring user success and goal completion.
A/B tests – improving marketing message and product design.

Thanks for listening. ご清聴、ありがとうございました。
[email_address] ← My email.
http://www.kalzumeus.com ← My blog.
http://www.abingo.org ← A/Bingo software (OSS)
http://www.startuplessonslearned.com ← Eric Ries, very useful.

Microsoft logo: http://www.flickr.com/photos/scobleizer/2264763977/ Sleeping Salaryman: http://www.flickr.com/photos/kansai/3452417503/ Edamame (Green Soybeans): http://www.flickr.com/photos/kankan/33346213/ Salty Peanuts: http://www.flickr.com/photos/euromagic/2351628831/ Facepalm Statue: http://www.flickr.com/photos/cesarastudillo/3981364314/ Photo Credits

http://www.kalzumeus.com	610
http://fbindie.posterous.com	57
http://www.slideshare.net	14
http://static.slidesharecdn.com	8
http://www.bingocardcreator.com	7
http://posterous.com	1
http://webcache.googleusercontent.com	1
https://www.coursesites.com	1
http://dashboard.bloglines.com	1

Data-Driven Software Design

by Patrick McKenzie on Apr 22, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

9 Embeds 700

Statistics

Data-Driven Software Design — Presentation Transcript