1. Marrying Web Analytics
and User Experience
Louis Rosenfeld • 5 August 2009
Delve NYC • Brooklyn
1

2. Web Analytics?
User Experience?
2

3. Code “DELVE” for 25% off
at rosenfeldmedia.com
3

4. My recent
struggle
4

5. CONTRASTING WEB ANALYTICS AND USER EXPERIENCE
5

6. Who we are
How we do our work
What data we use
How we use that data
CONTRASTING WEB ANALYTICS AND USER EXPERIENCE
5

7. WHO WE ARE
ARE THE STEREOTYPES TRUE?
6

8. VIVE LA DIFFÉRENCE! (FROM MARKO HURST)
7

9. !"#$%&"'()*+),%(-).(%("-&/)0(1/*$%)
Behavioral / Eyetracking Data Mining/Analysis
A/B (Live) Testing
Usability Benchmarking (in lab)
/
Data Source
Usability Lab Studies Online User Experience Assessments
(“Vividence-like” studies)
Ethnographic Field Studies
mix
Diary/Camera Study
Message Board Mining
Participatory Design Customer feedback via email
Focus Groups Desirability studies Intercept Surveys
Attitudinal Phone Interviews Cardsorting Email Surveys
mix
Qualitative (direct) Approach Quantitative (indirect)
Key for Context of Product Use during data collection
Natural use of product De-contextualized / not using product
© 2008 Christian Rohrer Scripted (often lab-based) use of product Combination / hybrid
20
HOW USER EXPERIENCE PEOPLE SEE THEIR WORK
(FROM CHRISTIAN ROHRER)
8

10. !"#$%&"'()*+),%(-).(%("-&/)0(1/*$%)
Behavioral / Eyetracking Data Mining/Analysis
A/B (Live) Testing
Usability Benchmarking (in lab)
/
Data Source
Usability Lab Studies Online User Experience Assessments
(“Vividence-like” studies)
Ethnographic Field Studies
mix
Diary/Camera Study
Message Board Mining
Participatory Design Customer feedback via email
Focus Groups Desirability studies Intercept Surveys
Attitudinal Phone Interviews Cardsorting Email Surveys
mix
Qualitative (direct) Approach Quantitative (indirect)
Key for Context of Product Use during data collection
Natural use of product De-contextualized / not using product
© 2008 Christian Rohrer Scripted (often lab-based) use of product Combination / hybrid
20
HOW USER EXPERIENCE PEOPLE SEE THEIR WORK
(FROM CHRISTIAN ROHRER)
8

11. HOW WEB ANALYTICS PEOPLE SEE THEIR WORK
(FROM AVINASH KAUSHIK)
9

12. HOW WEB ANALYTICS PEOPLE SEE THEIR WORK
(FROM AVINASH KAUSHIK)
9

13. The data that
drives our decisions
10

14. The data that
drives our decisions
Web Analytics User Experience
behavioral attitudinal
quantitative qualitative
high fidelity artificial
high volume high quality
This data is about WHAT This data is about WHY
10

15. The data that
drives our decisions
Web Analytics User Experience
behavioral attitudinal
quantitative qualitative
high fidelity artificial
high volume high quality
This data is about WHAT This data is about WHY
10

16. The data that
drives our decisions
Web Analytics User Experience
behavioral attitudinal
quantitative qualitative
high fidelity artificial
high volume high quality
This data is about WHAT This data is about WHY
10

17. The data that
drives our decisions
Web Analytics User Experience
behavioral attitudinal
quantitative qualitative
high fidelity artificial
high volume high quality
This data is about WHAT This data is about WHY
10

18. The data that
drives our decisions
Web Analytics User Experience
behavioral attitudinal
quantitative qualitative
high fidelity artificial
high volume high quality
This data is about WHAT This data is about WHY
10

19. The data that
drives our decisions
Web Analytics User Experience
behavioral attitudinal
quantitative qualitative
high fidelity artificial
high volume high quality
This data is about WHAT This data is about WHY
10

20. Not much use to know
what is happening if you
don’t know why
11

21. Not much use to know
what is happening if you
don’t know why
Hard to know why things
are happening if you don’t
know what is happening
11

22. The ways we analyze
our data
12

23. The ways we analyze
our data
12

24. The ways we analyze
our data
12

25. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200
971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX
X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=regional+transportation+governance
+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
13

26. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200
971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX
X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=regional+transportation+governance
+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
14

27. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200
971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX
X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=regional+transportation+governance
+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
Q “What were the most
common searches?”
14

28. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200
971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX
X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=regional+transportation+governance
+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
Q “What were the most
common searches?”
14

29. Analyzing data
the UX way:
play with the data,
look for patterns, trends,
and outliers

30. Analyzing data
the UX way:
play with the data,
look for patterns, trends,
and outliers
So what’s being measured?

31. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200
971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX
X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=regional+transportation+governance
+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
16

32. XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200
971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX
X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty
lesheet=www&q=regional+transportation+governance
+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
Q “Are we converting
license plate renewals?”
16

33. Before data analysis:
why are we here?
★ Commerce
★ Lead Generation
★ Content/Media
★ Support/Self-Service
17

34. Before data analysis:
why are we here?
★ Commerce
★ Lead Generation
★ Content/Media
★ Support/Self-Service
Data supports metrics
17

35. Analyzing data
the WA way:
start with metrics,
benchmark and
measure performance

36. Analyzing data
the WA way:
start with metrics,
benchmark and
measure performance
But you can’t measure
what you don’t know

37. WA: Top-down analysis
UX: Bottom-up analysis
19

38. what
WA: Top-down analysis
UX: Bottom-up analysis
19

39. what
WA: Top-down analysis
UX: Bottom-up analysis
why
19

40. INTEGRATING WEB ANALYTICS AND USER EXPERIENCE
20

41. Integrating methodologies:
What, then why
21

42. Common queries
can drive task analysis
22

43. Common queries
can drive task analysis
“Can you find a map of
the campus?”
“What study abroad
options are available to
students?”
“When is the last home
football game of the
season?”
22

44. Query data
can augment
personas
23

45. Query data
can augment
personas
“What Steven Searches”
added to existing persona
(from Adaptive Path)
23

46. Looking ahead
★ How do we improve other
qualitative methods with data?
★ How do qualitative data
impact quantitative analyses?
24

47. Methodology takeaways:
★ Qualitative research is
expensive
★ Start with quantitative
research to identify where/when
to use qualitative methods
25

48. Changing how we analyze:
Moving away from
the middle
26

49. 27

50. 28

51. What’s in
the middle?
28

52. What’s in
the middle?
Your analytics app’s
canned reports
28

53. Netflix moved away
from the middle
29

54. Netflix moved away
from the middle
29

55. Netflix moved away
from the middle
29

56. Netflix moved away
from the middle
29

57. Netflix moved away
from the middle
29

58. Analysis takeaways
★ Canned reports are only a
starting point
★ Move up, move down
★ Be prepared to “roll your own”
★ Demand better ad hoc
reporting from analytics apps
30

59. Changing our thinking:
Getting comfortable with
the other
31

60. UX people need to get
comfortable with
measuring the
unmeasurable
32

61. Can you measure
your content’s
quality?
Systems can help
us objectify the
subjective
33

62. Subjective
evaluations...
Can you measure
your content’s
quality?
Systems can help
us objectify the
subjective
33

63. Subjective
evaluations...
...lead to
Can you measure objective
decisions
your content’s
quality?
Systems can help
us objectify the
subjective
33

64. UX people need to get
comfortable with numbers
(but just a little)
34

65. This is not statistics
35

66. This is not statistics
This is not difficult
35

67. This is not statistics
This is not difficult
This is very useful
35

68. This is not statistics
This is not difficult
This is very useful
(and this is in MS Excel)
35

69. WA people need to get
comfortable with stories
36

72. WA people need to
understand the value of
intuition and mistakes
38

73. C# '&DE#F
<=>>?@A=B
!""#$%&'()*+*(,%+-'.()/-0%'(12*3(4+'(5"6()%'-#7%/ !
!"#$%&'#()*+,-.#/0/1#23#*'#456#3,5#17#1778.#17.19:#;6&%-#
!"#$%#&%'(&)%*#+,")#$%%-.#/(01#2($3,4.#5((1+%6"#)('#*%"714%&.#8,"#9(4:7&3%*#)8,)#8%6"#+%,;741#
<$%#8%,&#)(#=$7))%&>?#2($3,46"#&%,"(4"#:(&#@07))741#,&%#:,"974,)741#A#,4*#)8%B#"8($#$8B#
5((1+%6"#+("741#7)"#9((+?
2($3,4#C(74%*#5((1+%#)8&%%#B%,&"#,1(#A#)((#+,)%.#8%#4($#",B"?#=8%#9(3',4B6"#%4174%%&"D:7&")#
90+)0&%#$,"#:7&3+B#74#'+,9%.#3%,4741#%;%&B#*%97"7(4#8,*#)(#E%#'&(;%4#)8&(018#%F8,0")7;%#)%")741.#
&,)8%&#)8,4#,#&%+7,49%#(4#,#9+%,&#;7"7(4#(:#5((1+%6"#*%"714?#!4*#74#,#E,9-8,4*%*#"+,3#,)#5((1+%#GH#
I,&7"",#I,B%&.#)8%#8%,*#(:#J0"%&#%F'%&7%49%.J#8%#4()%"#)8,)#)('#3,4,1%3%4)#74#98,&1%#(:#*%"714#
8,*#4()#E,9-1&(04*#74#)8%#:7%+*K
L8%4#,#9(3',4B#7"#:7++%*#$7)8#%4174%%&".#7)#)0&4"#)(#%4174%%&741#)(#"(+;%#'&(E+%3"?#M%*09%#%,98#*%97"7(4#)(#,#"73'+%#+(179#
'&(E+%3?#M%3(;%#,++#"0EC%9)7;7)B#,4*#C0")#+((-#,)#)8%#*,),?#/,),#74#B(0&#:,;(&N#O-.#+,0498#7)?#/,),#"8($"#4%1,)7;%#%::%9)"N#2,9-#)(#
)8%#*&,$741#E(,&*?#!4*#)8,)#*,),#%;%4)0,++B#E%9(3%"#,#9&0)98#:(&#%;%&B#*%97"7(4.#',&,+BP741#)8%#9(3',4B#,4*#'&%;%4)741#7)#
:&(3#3,-741#,4B#*,&741#*%"714#*%97"7(4"? #
#
#
Q%".#7)6"#)&0%#)8,)#,#)%,3#,)#5((1+%#9(0+*46)#*%97*%#E%)$%%4#)$(#E+0%".#"(#)8%B6&%#)%")741#RS#"8,*%"#E%)$%%4#%,98#E+0%#)(#"%%#$8798#
(4%#'%&:(&3"#E%))%&?#T#8,*#,#&%9%4)#*%E,)%#(;%&#$8%)8%&#,#E(&*%&#"8(0+*#E%#U.#R#(&#V#'7F%+"#$7*%.#,4*#$,"#,"-%*#)(#'&(;%#3B#
9,"%?#T#9,46)#('%&,)%#74#,4#%4;7&(43%4)#+7-%#)8,)?#T6;%#1&($4#)7&%*#(:#*%E,)741#"098#3747"90+%#*%"714#*%97"7(4"?#=8%&%#,&%#3(&%#
%F97)741#*%"714#'&(E+%3"#74#)87"#$(&+*#)(#),9-+%?
WF97)741#*%"714#'&(E+%3".#+7-%#)8("%#,)#=$7))%&N#!#"(0&9%#)%++"#0"#)8,)6"#$8%&%#8%6"#1(741.#E0)#2($3,4#8,"46)#9(4:7&3%*#)8,)#
B%)?#<X%#'&(37"%"#)(#*7"9+("%#87"#4%$#%3'+(B%&#74#,#:(++($0'#E+(1#'(")?>

74. C# '&DE#F
<=>>?@A=B
!""#$%&'()*+*(,%+-'.()/-0%'(12*3(4+'(5"6()%'-#7%/ !
!"#$%&'#()*+,-.#/0/1#23#*'#456#3,5#17#1778.#17.19:#;6&%-#
!"#$%#&%'(&)%*#+,")#$%%-.#/(01#2($3,4.#5((1+%6"#)('#*%"714%&.#8,"#9(4:7&3%*#)8,)#8%6"#+%,;741#
<$%#8%,&#)(#=$7))%&>?#2($3,46"#&%,"(4"#:(&#@07))741#,&%#:,"974,)741#A#,4*#)8%B#"8($#$8B#
5((1+%6"#+("741#7)"#9((+?
2($3,4#C(74%*#5((1+%#)8&%%#B%,&"#,1(#A#)((#+,)%.#8%#4($#",B"?#=8%#9(3',4B6"#%4174%%&"D:7&")#
90+)0&%#$,"#:7&3+B#74#'+,9%.#3%,4741#%;%&B#*%97"7(4#8,*#)(#E%#'&(;%4#)8&(018#%F8,0")7;%#)%")741.#
&,)8%&#)8,4#,#&%+7,49%#(4#,#9+%,&#;7"7(4#(:#5((1+%6"#*%"714?#!4*#74#,#E,9-8,4*%*#"+,3#,)#5((1+%#GH#
I,&7"",#I,B%&.#)8%#8%,*#(:#J0"%&#%F'%&7%49%.J#8%#4()%"#)8,)#)('#3,4,1%3%4)#74#98,&1%#(:#*%"714#
8,*#4()#E,9-1&(04*#74#)8%#:7%+*K
L8%4#,#9(3',4B#7"#:7++%*#$7)8#%4174%%&".#7)#)0&4"#)(#%4174%%&741#)(#"(+;%#'&(E+%3"?#M%*09%#%,98#*%97"7(4#)(#,#"73'+%#+(179#
'&(E+%3?#M%3(;%#,++#"0EC%9)7;7)B#,4*#C0")#+((-#,)#)8%#*,),?#/,),#74#B(0&#:,;(&N#O-.#+,0498#7)?#/,),#"8($"#4%1,)7;%#%::%9)"N#2,9-#)(#
)8%#*&,$741#E(,&*?#!4*#)8,)#*,),#%;%4)0,++B#E%9(3%"#,#9&0)98#:(&#%;%&B#*%97"7(4.#',&,+BP741#)8%#9(3',4B#,4*#'&%;%4)741#7)#
:&(3#3,-741#,4B#*,&741#*%"714#*%97"7(4"? #
#
#
Q%".#7)6"#)&0%#)8,)#,#)%,3#,)#5((1+%#9(0+*46)#*%97*%#E%)$%%4#)$(#E+0%".#"(#)8%B6&%#)%")741#RS#"8,*%"#E%)$%%4#%,98#E+0%#)(#"%%#$8798#
(4%#'%&:(&3"#E%))%&?#T#8,*#,#&%9%4)#*%E,)%#(;%&#$8%)8%&#,#E(&*%&#"8(0+*#E%#U.#R#(&#V#'7F%+"#$7*%.#,4*#$,"#,"-%*#)(#'&(;%#3B#
9,"%?#T#9,46)#('%&,)%#74#,4#%4;7&(43%4)#+7-%#)8,)?#T6;%#1&($4#)7&%*#(:#*%E,)741#"098#3747"90+%#*%"714#*%97"7(4"?#=8%&%#,&%#3(&%#
%F97)741#*%"714#'&(E+%3"#74#)87"#$(&+*#)(#),9-+%?
WF97)741#*%"714#'&(E+%3".#+7-%#)8("%#,)#=$7))%&N#!#"(0&9%#)%++"#0"#)8,)6"#$8%&%#8%6"#1(741.#E0)#2($3,4#8,"46)#9(4:7&3%*#)8,)#
B%)?#<X%#'&(37"%"#)(#*7"9+("%#87"#4%$#%3'+(B%&#74#,#:(++($0'#E+(1#'(")?>

75. C# '&DE#F
<=>>?@A=B
!""#$%&'()*+*(,%+-'.()/-0%'(12*3(4+'(5"6()%'-#7%/ !
!"#$%&'#()*+,-.#/0/1#23#*'#456#3,5#17#1778.#17.19:#;6&%-#
!"#$%#&%'(&)%*#+,")#$%%-.#/(01#2($3,4.#5((1+%6"#)('#*%"714%&.#8,"#9(4:7&3%*#)8,)#8%6"#+%,;741#
<$%#8%,&#)(#=$7))%&>?#2($3,46"#&%,"(4"#:(&#@07))741#,&%#:,"974,)741#A#,4*#)8%B#"8($#$8B#
5((1+%6"#+("741#7)"#9((+?
2($3,4#C(74%*#5((1+%#)8&%%#B%,&"#,1(#A#)((#+,)%.#8%#4($#",B"?#=8%#9(3',4B6"#%4174%%&"D:7&")#
90+)0&%#$,"#:7&3+B#74#'+,9%.#3%,4741#%;%&B#*%97"7(4#8,*#)(#E%#'&(;%4#)8&(018#%F8,0")7;%#)%")741.#
&,)8%&#)8,4#,#&%+7,49%#(4#,#9+%,&#;7"7(4#(:#5((1+%6"#*%"714?#!4*#74#,#E,9-8,4*%*#"+,3#,)#5((1+%#GH#
I,&7"",#I,B%&.#)8%#8%,*#(:#J0"%&#%F'%&7%49%.J#8%#4()%"#)8,)#)('#3,4,1%3%4)#74#98,&1%#(:#*%"714#
8,*#4()#E,9-1&(04*#74#)8%#:7%+*K
L8%4#,#9(3',4B#7"#:7++%*#$7)8#%4174%%&".#7)#)0&4"#)(#%4174%%&741#)(#"(+;%#'&(E+%3"?#M%*09%#%,98#*%97"7(4#)(#,#"73'+%#+(179#
'&(E+%3?#M%3(;%#,++#"0EC%9)7;7)B#,4*#C0")#+((-#,)#)8%#*,),?#/,),#74#B(0&#:,;(&N#O-.#+,0498#7)?#/,),#"8($"#4%1,)7;%#%::%9)"N#2,9-#)(#
)8%#*&,$741#E(,&*?#!4*#)8,)#*,),#%;%4)0,++B#E%9(3%"#,#9&0)98#:(&#%;%&B#*%97"7(4.#',&,+BP741#)8%#9(3',4B#,4*#'&%;%4)741#7)#
:&(3#3,-741#,4B#*,&741#*%"714#*%97"7(4"? #
#
#
Q%".#7)6"#)&0%#)8,)#,#)%,3#,)#5((1+%#9(0+*46)#*%97*%#E%)$%%4#)$(#E+0%".#"(#)8%B6&%#)%")741#RS#"8,*%"#E%)$%%4#%,98#E+0%#)(#"%%#$8798#
(4%#'%&:(&3"#E%))%&?#T#8,*#,#&%9%4)#*%E,)%#(;%&#$8%)8%&#,#E(&*%&#"8(0+*#E%#U.#R#(&#V#'7F%+"#$7*%.#,4*#$,"#,"-%*#)(#'&(;%#3B#
9,"%?#T#9,46)#('%&,)%#74#,4#%4;7&(43%4)#+7-%#)8,)?#T6;%#1&($4#)7&%*#(:#*%E,)741#"098#3747"90+%#*%"714#*%97"7(4"?#=8%&%#,&%#3(&%#
%F97)741#*%"714#'&(E+%3"#74#)87"#$(&+*#)(#),9-+%?
WF97)741#*%"714#'&(E+%3".#+7-%#)8("%#,)#=$7))%&N#!#"(0&9%#)%++"#0"#)8,)6"#$8%&%#8%6"#1(741.#E0)#2($3,4#8,"46)#9(4:7&3%*#)8,)#
B%)?#<X%#'&(37"%"#)(#*7"9+("%#87"#4%$#%3'+(B%&#74#,#:(++($0'#E+(1#'(")?>

76. Tom Chi:
“Think of your designer as a guide in this
multi-variate optimization process. A good
designer has been all over parts of the territory
a dozen times on various projects and has
studied the design patterns and techniques
that help in different problems/situations.
Because of this, he or she has intuition on how
to approach a problem, just as an experienced
software architect has intuition on software
design approaches that provide different
benefits/drawbacks.”
40

77. UX and WA people need
to talk together about
project goals
41

78. 42

79. Vanguard and the
quantification of search
Target Oct 3 Oct 10 Oct 16
Mean distance from 1st 3 13 7 5
Median distance from 1st 2 7 3 1
Count: Below 1st 47% 84% 62% 58%
Count: Below 5th 12% 58% 38% 14%
Count: Below 10th 7% 38% 10% 7%
Precision – Strict 42% 15% 36% 39%
Precision – Loose 71% 38% 53% 65%
Precision – Permissive 96% 55% 72% 92%
Note: quantification, not monetization

80. Changing thinking
takeaways
★ Most things can be quantified
★ Stories and emotions can
make stronger cases than data,
and for data
★ We need more talking, and
more listening
44

81. Challenges: how do we...
★ Bridge cultural gaps?
★ Get different groups to speak
the same language?
★ Design and manage integrated
teams?
★ Find better, more open tools?
★ Develop a unified methodology?
45

82. Do we have a choice?
An individual often uses
only half their brain
Effective teams and
organizations use both
halves
46

84. Some day my book
will come...
Search Analytics for Your Site:
Conversations with Your Customers
Louis Rosenfeld & Marko Hurst
Rosenfeld Media, 2009.
rosenfeldmedia.com/books/searchanalytics
48

85. Until then...
Louis Rosenfeld
457 Third Street, #4R
Brooklyn, NY 11215 USA
lou@louisrosenfeld.com
www.louisrosenfeld.com
www.rosenfeldmedia.com
Twitter:
@louisrosenfeld
@rosenfeldmedia
This presentation @ http://www.slideshare.net/lrosenfeld