Slides from Velocity Conference Europe 2013 from my joint session with @bbrewer (Soasta). The talk covers implementation, analysis techniques and how you as the performance team can use RUM data to persuade and justify investments in changes that lead to improved performance....

1. Real World RUM

2. @jeroentjepkema
Founder & CEO

3. Eliminate the
noise, create
insights...

4. What type of performance data am I?

5. Right now?
Everything working?
Is it me or the internet?
What’s the impact?
How to fix it?
24h

6. Right now?
Service Levels
Everything working?
Yesterday
Is it me or the internet?
How did we perform?
What’s the impact?
Versus targets
How to fix it?
Historical context
24h
7days

7. Right now?
Service Levels
Trends &
Optimization
Everything working?
Yesterday
Future
Is it me or the internet?
How did we perform?
Big Picture
What’s the impact?
Versus targets
Actionable
How to fix it?
Historical context
24h
Optimization/Conversion
7days
??

8. Mapping Synthetic versus RUM

9. Keywords:
Error focused
Root Cause
Object driven information
Real
time
Reporting
life cycle
Keywords:
Alerting
Incident driven
MTTR
Keywords:
User experience
Conversion metrics
Volume/Trends
Data required
Day
Week
Month
Relevance for Application Life Cycle
Quarter
Keywords:
Business impact
Front End Optimization
Capacity Management

10. Keywords:
Error focused
Root Cause
Object driven information
Synthetic
Real
time
Reporting
life cycle
Keywords:
Alerting
Incident driven
MTTR
Keywords:
User experience
Conversion metrics
Volume/Trends
Data required
Day
Week
Month
Quarter
RUM
Relevance for Application Life Cycle
Keywords:
Business impact
Front End Optimization
Capacity Management

11. Real World RUM
Unsampled, Actionable

12. RUM
for Ops

13. Season Readiness Dashboard
(example 2: Real Time Performance Dashboard)
Real Time
Visibility

14. Example.com
Desktop

15. Example.com
Performance
Culture

16. Unknown
Unknowns

17. Newsletter
Marketing page
Case presented at Webperfdays Amsterdam

18. Email campaign
10Mb webpage!!
Slow traffic?
RUM
Triage issue
Case presented at Webperfdays Amsterdam

19. RUM for
Analytics

20. #1.
Group data

21. Put monitoring into your release cycle:
‣
Align to the development team(s)
‣
If possible, match with conversion targets

22. Home
Product
List
Budget
Segmented
Matched per page type
Matched per dev. team
Search
Matched per page template
Payment

23. #2.
Performance
SLA

24. Wait a week or 2
before setting
targets

25. Establish a baseline:

26. Establish a baseline:
A pre-defined set of metrics

27. Establish a baseline:
A pre-defined set of metrics
that describes normal behavior

28. Establish a baseline:
A pre-defined set of metrics
that describes normal behavior
in order to detect variancies

29. Establish a baseline:
A pre-defined set of metrics
that describes normal behavior
in order to detect variancies
and to be comparable within historic context

30. A practical (and easy DIY) example:

31. Purchasing a book,
must be completed (speed),
where every page loads under 3 sec.,
Customer journey
Metric: Speed
Target: Sec
using IE9 and higher,
User scenario
from any location in the Netherlands,
User locations
for 90% of all users,
every day between 6am and 12pm,
measured with Real User Monitoring.
Percentile
Window
Collection type
Read more: Metrics 101, Velocityconf 2010

32. Performance SLA

33. 100%
150%
125%
100%
75%
50%
50%
25%
25%
0%
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Sessions
Bounce rate
75%

34. LD50: 5.2 sec.
100%
150%
100%
125%
100%
75%
50%
50%
25%
25%
0%
0%
1
2
3
4
5
6
7
8
9
10
11
12
Baseline: 2,7 sec.
Poverty Line: 10,6 sec.
13
14
Sessions
Bounce rate
75%

35. #3.
Understand
User Behavior

36. Optimization begins with understanding
how users actually use your product...

37. 0:0
0:0
1:0 0"
0:0
2:0 0"
0:0
3:0 0"
0:0
4:0 0"
0:0
5:0 0"
0:0
6:0 0"
0:0
7:0 0"
0:0
8:0 0"
0:0
9:0 0"
0:0
10
:00 0"
11 :00"
:00
12 :00"
:00
13 :00"
:00
14 :00"
:00
15 :00"
:00
16 :00"
:00
17 :00"
:00
18 :00"
:00
19 :00"
:00
20 :00"
:00
21 :00"
:00
22 :00"
:00
23 :00"
:00
:00
"
Patterns
Pageviews (%)
Daily pattern
Totaal&
100%
450000"
400000"
75%
350000"
300000"
250000"
50%
200000"
150000"
25%
100000"
50000"
0%
0"

38. page%view%trend%(week)%
Weekly
Daily pattern
Totaal& pattern
20000"
Pageviews (%)
2000"
0:0
0:0
1:0 0"
0:0
2:0 0"
0:0
3:0 0"
0:0
4:0 0"
0:0
5:0 0"
0:0
6:0 0"
0:0
7:0 0"
0:0
8:0 0"
0:0
9:0 0"
0:0
10
:00 0"
11 :00"
:00
12 :00"
:00
13 :00"
:00
14 :00"
:00
15 :00"
:00
16 :00"
:00
17 :00"
:00
18 :00"
:00
19 :00"
:00
20 :00"
:00
21 :00"
:00
22 :00"
:00
23 :00"
:00
:00
"
Patterns
100%
450000"
18000"
400000"
16000"
75%
350000"
14000"
300000"
12000"
250000"
50%
10000"
200000"
8000"
150000"
25%
100000"
6000"
50000"
4000"
0%
0"
0"
03Jun2013"
04Jun2013"
05Jun2013"
06Jun2013"
07Jun2013"
08Jun2013"
09Jun2013"
10Jun2013"

39. 01
0:0 Jun2
020: 01
100 Ju00" 3"
: 30 n201
:n
Ju00 3"
200 2"0
: 40
Ju0 13"
: n0
300 2"01
: 50
Ju0 3"
: n0
400 2"01
: 60u
J : n 3"
2
500 J 00"013
7u
: 0
n
:00 0 "
08 2" 1
6:0 Ju
n
0:0 2 3"
09 0 01
7:0 Ju " 3"
n2
100:00 01
8:0 Jun " 3"
110:0 201
9:0 Jun0" 3"
120:0 201
10 Jun0" 3"
:
100: 201
11 3Ju00" 3"
:4 n
100 201
:n
12 Ju00" 3"
:5
100 201
:n
131 Ju00"0 3"
:00 2 13
6Ju
n2
141 :00"0 "
7J
:00u 13
n2
1518 :00"01 "
:00u
J n 3"
2
1619 :00"01
J
:00un 3"
2
1720J :00"013
:00un
2 "
21 :00 01
18 Ju " 3
:00 n2 "
:
1922Ju00"013"
: n
200:0 201
20 3Jun0" 3"
:
200: 201
21 4Ju00" 3"
:5 n
200 201
:n
22 Ju00" 3"
200 201
:6
J: n
232 u00"0 3"
:00 2 13
7Ju
:00 "
n
28 2"01
Jun 3"
29 201
Jun 3"
30 201
Jun 3"
20
13
"
Patterns
Pageviews (%)
100%
450000"
18000"
250000"
400000"
16000"
75%
200000"
350000"
14000"
300000"
150000"
12000"
250000"
50%
10000"
100000"
200000"
8000"
150000"
25%
50000"
100000"
6000"
50000"
0"
4000"
0%
0"
Pageviews
20000"
0"
03Jun2013"
page%view%trend%(week)%
Weekly pattern
Daily Totaal&
pattern
Totaal& pattern
Monthly
2000"
04Jun2013"
05Jun2013"
06Jun2013"
07Jun2013"
08Jun2013"
09Jun2013"
10Jun2013"

40. Top 5 DESKTOP
Top Browser type
Bouncerate
avg. Back-End Time
avg. Front-End time
Total avg. load time
1
Chrome30
49%
0.26
1.31
1.61
2
IE10
51%
0.27
0.81
1.18
3
Firefox25
48%
0.35
1.42
1.88
4
IE9
51%
0.27
1.26
1.61
5
Mobile Safari7
48%
0.52
1.86
2.79
Browsers, usage, performance...
Top 5 MOBILE
Browser type
Bouncerate
Backend Time
FrontEnd time
Loadtime
1
Android 4
72%
0.27
2.38
2.64
2
Mobile Safari 7
53%
0.93
1.04
2.58
3
Chrome Mobile 30
75%
0.40
1.19
1.99
4
Chrome Mobile 18
82%
0.41
1.53
2.25
5
Android 2
59%
0.72
1.88
3.05

41. Homepage_MHP"10th"&"90th"percen0les"
Mac"vs."Windows"
Mac"P_10"
Mac"P_90"
Windows"P_10""
Windows"P_90"
35"
30"
Load"0me"(sec)"
25"
20"
15"
10"
5"
0"
06(22(2013"0:00"
06(24(2013"0:00"
06(26(2013"0:00"
06(28(2013"0:00"

42. #4.
What affects
performance?

43. Understand the DNA of your website and what affects
your performance...

44. Speed vs. Engagement per month (Non-Bounce)
90
78,75
56,25
Pageviews (#)
Non-Bounce rate (%)
67,5
45
33,75
22,5
11,25
0
0-0,5
0,5-1
1-1,5
1,5-2
2-2,5
2,5-3
3-3,5
3,5-4
4-5
5-6
Non-Bounce CategorienaamPageviews
#
6-7
7-8
8-9
9-10
>10

45. First view vs. Repeat view
7
7000
6
0%
Patterns
Load Time (Sec.)
6000
5
5000
4
4000
3
3000
2
2000
1
1000
0
0%
0
Monday
Tuesday
Wednesday
Thursday
First View IE9 Windows
Friday
Saturday
Repeat View IE9 Windows
Sunday
Monday
Tuesday
First View Safari MacOS
Wednesday
Thursday
Repeat View Safari MacOS
Friday
Saturday
Sunday

46. Page views vs. Load time breakdown
0%
7
6
100%
5
2.5
Load (%)
Page views Time (Sec.)
Pageviews
4
75%
3
50%
1.5
Usage impacts Speed?
2
25%
1.0
0
0
0%
1
0%
Time (days)
TCP connection
SSL Handshake
DNS Resolution
Front-End time
DOM Ready
Page Load
Load Time (Sec.)
2.0

47. #5.
Analyze &
Optimize

48. Group & Graph:
Pageviews
5,00
4,50
4,00
3,50
3,00
2,50
2,00
1,50
1,00
Load Time (sec.)
Group all: Payment
100%
5,00
4,50
4,00
3,50
3,00
2,50
2,00
1,50
1,00
Load Time (sec.)
Pageviews
Payment page: Confirmation

49. Boxplots Outliers Incidents?
Group: Home
Avg. Load Time
Load Time (sec.)
Hours (#): 17
Avg. Loadtime (sec.): 7.34
Sum of Page Views: 11.327
Time (Hour)
Time

50. Train your audience:
‣ Spread the word to key stakeholders
‣ Report within 24 hours after new release period
‣ Always compared to previous period

51. Getting fancy

52. Service levels
Service Level Report
KPI’s
082
084
Performance (avg)
Page 1
www.homepage.nl
Loadtime.
Loadtime
Availability (Document.
(.fully.loaded.)
complete)
Competition
You
5
97,0%
2,20
3,50
10%
85%
Page 2
ge.nl/autoverzekeren
99,0%
2,50
3,30
20%
90%
Page 3
overzekeren/afsluiten
95,0%
2,30
3,60
15%
70%
95,0%
2,43
3,57
20%
67%
1
Page 5
homepage.nl/zoeken
94,0%
2,48
3,62
23%
59%
Branche
2
Page 4
homepage.nl/contact
Competitors
0
Page 6
e.nl/productoverzicht
93,0%
2,53
3,67
25%
92,0%
2,58
3,72
28%
44%
3
33
34
35
ge.nl/ab8test/variant2
Page 8
91,0%
2,63
3,77
30%
37%
ge.nl/facebookplugins
Page 9
90,0%
2,68
3,82
33%
89,0%
2,73
3,87
35%
22%
ww.homepage.nl/faq
Page 11
88,0%
2,78
3,92
38%
14%
37
38
39
40
# 6
3,4 sec
Is your ranking
against
competitors
is the average
performance of your
competitors
40%
40% of your
compeitors are
faster than you
29%
page.nl/muziekpagina
Page 10
36
Week
52%
Page 7
ge.nl/ab8test/variant1
4
Source:
Availability: Gomez
Loadtime: Soasta
Competition: Webpagetest.org

53. Performance Trend report
75
4
45
3
2
1
0
Objects (#)
60
15
3d Party
5
30
URL’s
6
0
33
34
35
36
37
38
39
Load time (sec.)
90
Speed
40
Week (#)
33
34
35
36
37
38
39
40
Speed (sec.)
2.9
2.4
2.2
2.8
3.9
2.9
3.1
2.9
URL’s (#)
67
68
65
69
80
72
69
73
3d Party (#)
7
8
6
9
8
9
7
7
Page Size (Kb)
2000
1800
1850
2100
2700
2200
2100
2150
Source:
Speed: Soasta
Object data: Webpagetest.org

54. Revenue risked
100.000 visits
(Performance/Time Period/Percentiles)
Optimal Flow (P_25)
Bouncerate (%)
Potential Conversion
2.1
100
49%
51.000
Page 2
4.0
100
51%
24.990
Page 3
2.0
100
60%
9996
Page 4
3.5
85
25%
6372
Actual Flow (P_85)
Speed (sec.)
Availability (%)
Bouncerate (%)
Actual Conversion
Page 1
4.1
100
59%
41.000
Page 2
6.3
100
54%
18.860
Page 3
Speed: Soasta
Availability: Webpagetest.org
Availability (%)
Page 1
Source:
Speed (sec.)
2.0
100
60%
7544
Page 4
4.7
85
53%
3013
0-2,7 sec.
2,7-3,6 sec.
>3,6 sec.

55. From noise to
insights...

56. 1
Datamodel
vs.
Development

57. 1
2
Datamodel
vs.
Development
Collect data & set
Performance KPI

58. 1
2
Datamodel
vs.
Development
3
Collect data & set
Performance KPI
Understand
User Behavior

59. 1
2
Datamodel
vs.
Development
3
Collect data & set
Performance KPI
4
Understand
User Behavior
Analyze, Correlate
& Get Faster!

60. 1
2
Datamodel
vs.
Development
RUM, FTW!!
3
Collect data & set
Performance KPI
4
Understand
User Behavior
Analyze, Correlate
& Optimize

61. Thanks! More questions?
M: jtjepkema@measureworks.nl
T: @jeroentjepkema
W: www.measureworks.nl