This document summarizes a presentation by Luke Stevens on data-driven web design. [1] It discusses how performance metrics can be used to test different design variations and identify the best performing one using A/B testing. A case study is presented of redesigning PhotographyBLOG.com where testing found user performance was identical between two designs. The presentation advocates for using tools like Google Website Optimizer to serve different CSS files and measure results in Google Analytics to objectively test web designs.

1. Web Directions South 2009, Oct 8-9
Session: Data Driven Design
Speaker: Luke Stevens (@lukestevens)
Sign up for the book!
http://msrmnt.com/book/
http://twitter.com/msrmnt
Questions?
luke@msrmnt.com

2. Hey, I’m Luke

3. 1. Intro + The Problem
2. Quick case study
3. Cool technique

4. Data driven design

5. © Dominic Wilcox http://dominicwilcox.com

6. Who cares?

7. Data driven design
Performance
based design

9. No longer just an
aesthetic, usability
or experience
problem.

10. 1. Coaches + athletes
2. Doctors + patients

11. Coach + Athlete

13. Athletes:
- Have a speci c goal
- Goal = single number
- Data driven

15. Michael Phelps’ breakfast:
- Three fried-egg sandwiches
- Five-egg omelet
- Bowl of grits (porridge)
- Three slices of French toast
- Three choc-chip pancakes
- Two cups of coffee

16. Performance encourages:
- Creativity
- The unexpected
- Risk

17. Doctors

19. 1st priority?

20. NOT KILL YOU
(aka ‘do no harm’)

21. WARNING:
Side effects may include:
- Decreased sales
- Angry customers
- Lighter wallet

23. Getting harder
NOT to do harm

24. Design is maturing

25. http://www.microsoft.com/misc/features/features_ shbk.htm

26. 2009

27. Even Microsoft
gets design
Well, visual design. On the web. For their home page.

28. Design maturity over time
50
37.5
25
12.5
0
1994 1997 2000 2003 2006 2009

29. Do no harm

30. One problem.
Many solutions.

31. Who chooses?

32. Flip a coin?

33. Wired:
Extreme Makeover:
Craigslist Edition
http://wired.com/entertainment/theweb/magazine/17-09/ff_craigslist_makeover

34. Original - craigslist today

35. Khoi Vinh & colleagues, NYTimes.com

36. SimpleScott, former design director of BarackObama.com

37. Luke Hayman and Lisa Strausfeld, Pentagram

38. 1. Will it do harm?
2. Which is best?

39. “Your designs are
terrible, confusing and
you miss the point”

40. If we, as designers,
can’t pick the best
performer, then...?

41. THIS IS
THE PROBLEM

42. Out with the old

43. In with the new

44. Gutenberg Bible - 15th century

45. What we do is different

46. The most
measurable
form of design
in history

47. Recap - Part 1

48. 1. We have the most
measurable form of
design in history

49. 2. We should use that

50. Part II
Quick case study

51. Know your metric

52. What success looks like:

53. PhotographyBLOG.com - Original

54. ~1 MM visitors
~3 MM page views
~10 blog stories a day

55. Home page traffic?

56. 8%
(92% elsewhere)

57. Data > Assumptions

58. Know Google Analytics

59. Buy a book
advanced-web-metrics.com

60. Know web analytics

61. Read Avinash’s blog
kaushik.net/avinash/

62. PhotographyBLOG:
Camera reviews

63. Web analytics = strategy

64. PhotographyBLOG.com - v1

65. PhotographyBLOG.com - v2

66. PhotographyBLOG.com - v1 & v2

67. Test, don’t guess

68. A/B testing

69. Which design won?

70. v1 or v2? Pick one

71. Neither
Performance was almost identical

72. ANY result = learnings*
* ”learnings” is not a real word

73. E.g. Users did. not. care.

74. PhotographyBLOG.com - Original

75. PhotographyBLOG.com - Redesign

76. Performance based design:
Happy client &
Happy users

77. Testing is powerful

78. Testing will change the way
you think about design

79. But what to test?

80. YouTube - 1,024 (!)
combinations of
tiny details.
15% more sign ups

81. YouTube’s 1,024 combo experiment
using Google Website Optimizer
From: http://youtube-global.blogspot.com/2009/08/look-inside-1024-recipe-multivariate.html

82. 37signals tested
headlines.
30% more sign ups

83. Winner
30% lift
2nd place
27% lift
3rd place
15% lift
Original
(worst performer)
From: http://37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

84. Button text:
“Free Trial”
“Sign-up for Free Trial”
“See Plans and Pricing”

85. 37signals tested a
button -
Doubled clicks
From: http://carsoni ed.com/blog/business/how-to-increase-sign-ups-by-200-percent/

86. We barely know where
the landscape is,
let alone where we
should be on it.

87. We all need to test

88. - All design is testable
- Testing takes time
- Can only test a few things

89. We all need to test
and SHARE

90. Current clearing houses:
@kissmetrics
@abtests

91. Part III
A cool technique

92. Google Website Optimizer
google.com/websiteoptimizer

93. The web standards
movement has given us
an incredible,
unexpected gift.

94. HTML + CSS =
The perfect platform to
test design

95. Use GWO to serve up
alternate CSS les

96. (But conversions don’t
suit everyone...)

97. Measure the results in
Google Analytics
gwotricks.com

98. CSS + GWO + GA =
Testing trinity

99. Objective web design
is possible right now

100. Performance
based design
builds on
standards
based design

101. It’s the future
of web design

102. BOOK!
Sign up to get noti ed:
http://msrmnt.com/book
follow @msrmnt
luke@msrmnt.com | @lukestevens