1. Metrics 101
What to watch

3. What we’ll cover
Why collect metrics
Understanding web latency
How to target your findings
Concrete steps to get started

4. Part one
Why collect metrics?

5. http://www.flickr.com/photos/chidorian/12411641/

6. Downtime costs

7. Downtime costs
eBay offline ($90K/h)
22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)

8. Downtime costs
eBay offline ($90K/h)
22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)
Financial company down ($100K/h)
53.2% of finance companies lose over $100,000/hour (nextslm.org)

9. Downtime costs
Amazon offline ($1M/h)
Amazon loses nearly $1M/hour if down (NYT, 2008)
eBay offline ($90K/h)
22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)
Financial company down ($100K/h)
53.2% of finance companies lose over $100,000/hour (nextslm.org)

10. Downtime costs
Amazon offline ($1M/h)
Amazon loses nearly $1M/hour if down (NYT, 2008)
Network downtime ($42K/h)
1 hour of network downtime costs $42,000 (Gartner, 2003)
eBay offline ($90K/h)
22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)
Financial company down ($100K/h)
53.2% of finance companies lose over $100,000/hour (nextslm.org)

11. Downtime costs
Amazon offline ($1M/h)
Amazon loses nearly $1M/hour if down (NYT, 2008)
Network downtime ($42K/h)
1 hour of network downtime costs $42,000 (Gartner, 2003)
eBay offline ($90K/h)
22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)
Financial company down ($100K/h)
53.2% of finance companies lose over $100,000/hour (nextslm.org)
Let’s say $50K/h if you’re serious.

12. Availability Downtime/year Loss @$50K/h
90% % 36.5 days Can$43,800,000
95% 18.25 days Can$21,900,000
98% 7.30 days Can$8,760,000
99% 3.65 days Can$4,380,000
99.5% 1.83 days Can$2,196,000
99.8% 17.52 hours Can$876,000
99.9% 8.76 hours Can$438,000
99.95% 4.38 hours Can$219,000
99.99% 52.6 minutes Can$43,833
99.999% 5.26 minutes Can$4,383
99.9999% 31.5 seconds Can$438

13. Availability Downtime/year Loss @$50K/h
90% % 36.5 days Can$43,800,000
95% 18.25 days Can$21,900,000
98% 7.30 days Can$8,760,000
99% 3.65 days Can$4,380,000
99.5% 1.83 days Can$2,196,000
99.8% 17.52 hours Can$876,000
99.9% 8.76 hours Can$438,000 Less than
99.95% 4.38 hours Can$219,000 an hour a
99.99% 52.6 minutes Can$43,833 year
99.999% 5.26 minutes Can$4,383
99.9999% 31.5 seconds Can$438

14. Availability Downtime/year Loss @$50K/h
90% % 36.5 days Can$43,800,000
95% 18.25 days Can$21,900,000
98% 7.30 days Can$8,760,000
99% 3.65 days Can$4,380,000
99.5% 1.83 days Can$2,196,000
99.8% 17.52 hours Can$876,000
99.9% 8.76 hours Can$438,000 Less than
99.95% 4.38 hours Can$219,000 an hour a
99.99% 52.6 minutes Can$43,833 year
99.999% 5.26 minutes Can$4,383 Less than
99.9999% 31.5 seconds Can$438 a minute a
year

15. Harris poll conducted by Tealeaf in 2008

16. You really don’t want web
users to call you.
$15
$12
$9
$6
$3
$0
Web self-service IVR Email Live phone
Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service” Low Average High

17. You really don’t want web
users to call you.
$15
$12
$9
$6
$3
Can$0.24
$0
Web self-service IVR Email Live phone
Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service” Low Average High

18. You really don’t want web
users to call you.
$15
$12
$9
$6
$3
Can$0.24 Can$0.45
$0
Web self-service IVR Email Live phone
Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service” Low Average High

19. You really don’t want web
users to call you.
$15
$12
$9
$6
Can$3.00
$3
Can$0.24 Can$0.45
$0
Web self-service IVR Email Live phone
Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service” Low Average High

20. You really don’t want web
users to call you.
$15
$12
$9
$6 Can$5.50
Can$3.00
$3
Can$0.24 Can$0.45
$0
Web self-service IVR Email Live phone
Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service” Low Average High

21. http://www.flickr.com/photos/pagedooley/2811157950/

22. If you don’t know the past
you can’t know the future.
If you don’t know the future,
you can’t budget for it.
Photo by Alan Cleaver from his Flicker Freestock set. Thanks, Alan!
http://www.flickr.com/photos/alancleaver/2638883650/

23. “A plan so crazy, it just might work.”

24. http://www.flickr.com/photos/genewolf/147722350

25. http://www.flickr.com/photos/billselak/366692332/

26. Everything starts with a baseline.

27. Everything starts with a baseline.
Know what’s
worst.

28. Everything starts with a baseline.
Know what’s Prove you
worst. made it better.

29. The cycle of optimization
Metrics & strategy

30. The cycle of optimization
Metrics & strategy
Collection

31. The cycle of optimization
Metrics & strategy
Collection
Reporting

32. The cycle of optimization
Metrics & strategy
Collection
Reporting
Institutionalizing
the results

33. The cycle of optimization
Metrics & strategy
Collection
Link to KPI/
Reporting
ROI
Institutionalizing
the results

34. The cycle of optimization
Metrics & strategy
Optimization
Collection
& change
Link to KPI/
Reporting
ROI
Institutionalizing
the results

35. The cycle of optimization
Metrics & strategy
Optimization
Collection
& change
Link to KPI/
Reporting
ROI
Institutionalizing
the results

36. http://www.flickr.com/photos/elsie/8229790/

37. Understanding your goals.
http://www.flickr.com/photos/itsgreg/446061432/

40. Organic Ad
Campaigns
search network $
1 1 1
Advertiser site
Visitor 2 O er 3 $
8 Upselling 4
Abandonment
Reach
5 Purchase step $
Mailing,
alerts, Purchase step $
9 promotions
$
Conversion $
Disengagement 7
Enrolment 6
Impact on site
$ Positive $ Negative

43. Bad
$
4 content
Social Search
Invitation
network link results
4 Good
content
1 $
1 1
Collaboration site
2
Visitor Content creation Moderation
$
3 Spam & trolls
$
Engagement 5
Viral
6 Social graph
spread
7
Disengagement $
Impact on site
$ Positive $ Negative

45. Enterprise subscriber $
1
End user (employee) $
Refund $
2
Renewal, upsell, SLA
reference SaaS site violation
Performance
Good Bad 3
Helpdesk Support
5 $
Usability escalation costs
7
4
Good Bad
Productivity
Good Bad
6
Churn $
Impact on site
$ Positive $ Negative

47. $
Media site
Enrolment Targeted
2 embedded ad 5
$
6 1
Ad
Visitor
network
4
3 5
Advertiser $
Departure $ site
Impact on site
$ Positive $ Negative

48. Why measure
Tactical, to find and fix
Strategic, to plan/trend
Part two
The elements of web latency

49. Slow sites suck

50. Slow sites suck
Lower conversion rates

51. Slow sites suck
Lower conversion rates
Less likely to attract a loyal following

52. Slow sites suck
Lower conversion rates
Less likely to attract a loyal following
Liable for damages

53. Slow sites suck
Lower conversion rates
Less likely to attract a loyal following
Liable for damages
Liable for refunds or service credits

54. Slow sites suck
Lower conversion rates
Less likely to attract a loyal following
Liable for damages
Liable for refunds or service credits
Customers find other channels that cost more

55. Why the web is slow
A crash course in performance & availability.

56. Load Web App
Internet balancer server server DB
Client
www.example.com

57. Your website
Load Web App
Internet balancer server server DB
Client
www.example.com

58. DNS
Load Web App
Internet balancer server server DB
Client
DNS “www.example.com”

59. DNS DNS
lookup
Load Web App
Internet balancer server server DB
Client
DNS “www.example.com”

60. DNS DNS
lookup
Load Web App
Internet balancer server server DB
Client
DNS “www.example.com”

61. IP IP
Load Web App
Internet balancer server server DB
Client

62. IP IP
Load Web App
Internet balancer server server DB
Client
Internet routing

63. IP R IP
R
Load Web App
Internet R balancer server server DB
Client R
R
Internet routing

64. IP R IP
R
Load Web App
Internet R balancer server server DB
Client R
R
TCP session

65. IP R IP
R
Load Web App
Internet R balancer server server DB
Client R
R
TCP session

69. Letter writing
Postal service

70. You Them
(sender) (receiver)

71. This
is
a
sentence
You Them
(sender) (receiver)

72. This
is
a
sentence
You Them
(sender) (receiver)

73. You Them
(sender) (receiver)

74. You Them
(sender) (receiver)

75. sentence
This is
a
You Them
(sender) (receiver)

76. You Them
(sender) (receiver)

77. This
is
a
sentence
You Them
(sender) (receiver)

78. This
is
a
sentence
3
2
1
4
You Them
(sender) (receiver)

79. You Them
(sender) (receiver)

80. This
is
a
sentence
You Them
(sender) (receiver)

81. This
is
sentence
2
1
You Them
(sender) (receiver)

82. This
is
sentence
2
1
4
You Them
(sender) (receiver)

83. This
WTF? is
sentence
2
1
4
You Them
(sender) (receiver)

84. sentence a This
4 3 1

85. sentence a This
4 3 1
“Can you send #2 again?”

86. sentence a This
4 3 1
“Can you send #2 again?”
is
“Sure. Here you go.”
2

87. How computers “connect”

88. IP IP
Load Web App
Internet balancer server server DB
Client

89. The HTTP “stack”
IP IP
Load Web App
Internet balancer server server DB
Client

90. The HTTP “stack”
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client

91. The HTTP “stack”
SSL SSL
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client

92. The HTTP “stack”
HTTP HTTP
SSL SSL
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client

93. Getting a page by hand

94. Getting a page by hand
Trying 67.205.65.12...
Connected to bitcurrent.com.
Escape character is '^]'.

95. Getting a page by hand
Trying 67.205.65.12...
Connected to bitcurrent.com.
Escape character is '^]'.
GET /
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/
xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head profile="http://gmpg.org/xfn/11">
<script type="text/javascript" src="http://
www.bitcurrent.com/wp-content/themes/
grid_focus_public/js/perftracker.js"></script>
<script>
</body>
</html>
Connection closed by foreign host.

96. Static
content
HTTP HTTP HTTP
SSL SSL
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client
image.gif
GET www.example.com/image.gif

97. Static
content
HTTP HTTP HTTP
SSL SSL
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client
image.gif
GET www.example.com/image.gif

98. Static
content Dynamic
content
HTTP HTTP HTTP
SSL SSL
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client
dynamic.jsp
GET www.example.com/dynamic.jsp

99. Static
content Dynamic
content
HTTP HTTP HTTP
SSL SSL
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client
dynamic.jsp
GET www.example.com/dynamic.jsp

100. Static
content Dynamic Stored
content data
HTTP HTTP HTTP
SSL SSL
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client
(Database)
POST www.example.com/data.cgi

101. Static
content Dynamic Stored
content data
HTTP HTTP HTTP
SSL SSL
TCP TCP
IP IP
Load Web App
Internet balancer server server DB
Client
(Database)
POST www.example.com/data.cgi

102. Browser Data center
Server

103. Browser Data center
Server

104. Browser Data center
Server
TCP SYN (“let’s talk”)
TCP SYN ACK (“Agreed: let’s talk”)
TCP ACK (“OK, we’re talking)

105. Browser Data center
Server
TCP SYN (“let’s talk”)
TCP SYN ACK (“Agreed: let’s talk”)
TCP ACK (“OK, we’re talking)
SSL (“Someone might be listening!”)
SSL (“Here’s a decoder ring”)

106. Browser Data center
Server
TCP SYN (“let’s talk”)
TCP SYN ACK (“Agreed: let’s talk”)
TCP ACK (“OK, we’re talking)
SSL (“Someone might be listening!”)
SSL (“Here’s a decoder ring”)
HTTP GET / (“Can I have your home page?”)
HTTP 200 OK (“Sure!”)
(thinks
[index.html] (“Here it is!”) a bit)
(Renders furiously) Bump, bump. [img js css] (“Have this too!”)

107. Browser Data center
Server
TCP SYN (“let’s talk”)
TCP SYN ACK (“Agreed: let’s talk”)
TCP ACK (“OK, we’re talking)
SSL (“Someone might be listening!”)
SSL (“Here’s a decoder ring”)
HTTP GET / (“Can I have your home page?”)
HTTP 200 OK (“Sure!”)
(thinks
[index.html] (“Here it is!”) a bit)
(Renders furiously) Bump, bump. [img js css] (“Have this too!”)
TCP FIN (“Thanks! I’m done now.”)
TCP FIN ACK (“You’re welcome. Have a nice day.”)

109. “Page load time” isn’t simple
Documents versus event models
AJAX
Mobility
CDNs
Third-party content
Embedded objects and plug-ins

110. Connections to load
Connection 0 - www.bitcurrent.com (67.205.65.12)
Connection 1 - www.bitcurrent.com (67.205.65.12)
Connection 2 - 4qinvite.4q.iperceptions.com (64.18.71.70)
Connection 3 - static.slideshare.net (66.114.49.24)
Connection 4 - static.slideshare.net (66.114.49.24)
Connection 5 - www.feedburner.com (66.150.96.123)
Connection 6 - static.getclicky.com (204.13.8.18)
Connection 7 - cetrk.com (208.67.183.100)
Connection 8 - in.getclicky.com (204.13.8.18)
Connection 9 - crazyegg.com (208.67.180.236)
Connection 10 - www.google-analytics.com (72.14.223.147)
Connection 11 - www.apture.com (67.192.46.19)
Connection 12 - static.apture.com (67.192.46.25)
Connection 13 - s.clicktale.net (66.114.49.24)
Connection 14 - www.clicktale.net (75.125.82.70)

111. Analytics site
Server
Data center
Browser
Server
Mashup Server
site

112. Analytics site
Server
Data center
Browser
Server
Snore.
Mashup Server
site

113. What ultimately matters:
When can the user start using the application as
its designers intended?

114. Part of the problem
You control You’re blamed for
Server latency Page rendering
Network latency for Total network latency
known content and
User environment
network parameters

115. Part of the problem
You control You’re blamed for
Server latency Page rendering
Network latency for Total network latency
known content and
User environment
network parameters
You need
diagnostic metrics
so you can fix it.

116. Part of the problem
You control You’re blamed for
Server latency Page rendering
Network latency for Total network latency
known content and
User environment
network parameters
You need escalation
You need
metrics so you can prove
diagnostic metrics
it and make it someone
so you can fix it.
else’s problem.

117. Why measure
Tactical, to find and fix
Strategic, to plan/trend
What to measure:
How long until a user can use
the app as you intended?
Part three
Where to measure

118. Three tiers of data
WAN accessibility: One test from many locations
Can everybody get here?
App functionality: Several tests of key processes
Is my business model working correctly?
Tiered tests: Frequent metrics of each tier
Is network, service, CPU, data I/O to blame?

119. WAN accessibility
Place A
Task B
Client
Goal C
...
Load Web App
Internet balancer server server DB
Client

120. Analytics can tell you a lot.

125. App functionality
Page A
Page B
Client
Event C
Load Web App
Internet balancer server server DB
Client

126. http://www.flickr.com/photos/tinfoilraccoon/197640807/

127. Places and Tasks.

129. Landing page:
View one story

130. Landing page:
View one story
Task: Log in
Enter credentials
Verify
Recovery

131. Landing page:
View one story
Task: Log in
Enter credentials
Verify
Recovery
Task:
Forward a story
Enter recipients
Enter message
Send

132. Landing page:
Task: View one story
Create account
Task: Log in
Pick name
Check if free Enter credentials
Set Password Verify
CAPTCHA Recovery
Send mail
Get confirm
Task:
Forward a story
Enter recipients
Enter message
Send

133. Landing page:
Task: View one story
Create account
Task: Log in
Pick name
Check if free Enter credentials
Set Password Verify
CAPTCHA Recovery
Send mail
Get confirm
Task:
Forward a story
Task: Submit Enter recipients
a new story Enter message
Send
Enter URL
Describe
Deduplicate
Post it

134. Landing page:
Task: View one story
Create account
Task: Log in
Pick name Place: View stories
Check if free Enter credentials
Vote up Next 25
Set Password Verify
Vote down Last 25
CAPTCHA Recovery
Send mail
Get confirm
Task:
Forward a story
Task: Submit Enter recipients
a new story Enter message
Send
Enter URL
Describe
Deduplicate
Post it

135. Landing page:
Task: View one story
Create account
Task: Log in
Pick name Place: View stories
Check if free Enter credentials
Vote up Next 25
Set Password Verify
Vote down Last 25
CAPTCHA Recovery
Send mail
Place: Read
Get confirm
poster comments
Vote up Next 25
Task:
Vote down Last 25
Forward a story
Task: Submit Enter recipients
a new story Enter message
Send
Enter URL
Describe
Deduplicate
Post it

136. Landing page:
Task: View one story
Create account
Task: Log in
Pick name Place: View stories
Check if free Enter credentials
Vote up Next 25
Set Password Verify
Vote down Last 25
CAPTCHA Recovery
Send mail
Place: Read
Get confirm
poster comments
Vote up Next 25
Task:
Vote down Last 25
Forward a story
Task: Submit Enter recipients
a new story Place: My Enter message
Enter URL account Send
Describe Change My
address comments
Deduplicate
Change PW See karma
Post it

137. Landing page:
Create acct. View one story
Task: Log in
Place: View stories
Place: Read
poster comments
Task:
Forward a story
Task: Submit
a new story Place: My
account

138. Landing page:
Create acct.
Create acct. View one story
Form uptime Place: View stories
Task: Log in
# started
Bad form
Place: Read
# CAPTCHA poster comments
Mail uptime Task:
Forward a story
Mail bounced
Task: Submit
a new story Place: My
Confirm & return account
Return 3x

139. Landing page:
Create acct. View one story
Task: Log in
Place: View stories
Place: View stories
Stories/visit
Place: Read
# up/down
poster comments
Time/story Top stories
Task:
Forward a story
Task: Submit Refresh time Views/page
a new story Place: My
account

140. Landing page:
Create acct. View one story
Task: Log in
Place: View stories
Place: Read
poster comments
Task:
Forward a story
Task: Submit
a new story Place: My
account

141. Places
Efficiency matters
How quickly, how many,
productivity
Learning curve OK
Leave when they’re bored
Collect “aha” feedback
A/B test content for
pages/session, exits

142. Tasks
Effectiveness matters
Completion, abandonment
Intuitiveness rules
Leave when they change their
mind or it breaks
Collect “motivation” feedback
A/B test layouts for conversion

143. 2 sides of the same coin
End user
Web analytics
monitoring
What did Could they
visitors do? do it?

144. For e-commerce sites
Can people buy things?

145. For media sites
Are ads loading quickly and successfully clicked
through?
Is content loading fast enough for visitors?

146. For collaboration sites
Can visitors contribute (posting content, voting?)
Is bad content being mitigated (trolling, spam)?

147. For SaaS sites
Are your end users productive?
Are they making fewer mistakes?
Is the site working during customers’ business hours?

148. Tiered tests
Place A
Task B
Client
Goal C
Load Web App
Internet balancer server server DB
Client

149. Testing the tiers
Load Web App
Internet balancer server server DB
Client
Request Do some Search a
Request a
uncached heavy dataset for
big object
object computing a string
(Or watch (Or track
CPU) query time)

150. ,)-$(&./01+2(3/04(#$+#+(
&)$
%,$
%+$
%*$
!"#$%&'()%(*+(
%&$
%)$
,$ '""#$($
+$ '""#$&$
*$
'""#$%$
&$
)$ !""#$&$
-./01$2341$ !""#$%$
53"67$2341$
8!9$2341$
':$2341$

151. Why measure
Tactical, to find and fix
Strategic, to plan/trend
What to measure:
How long until a user can use
the app as you intended?
Part four Where to measure:
How to measure WAN, from everywhere
Core app functionality
performance data Tiers of components

152. Synthetic testing.

153. Load Web App
Internet balancer server server DB
Client

154. Management tool
Load Web App
Internet balancer server server DB
Client

155. Load Web App
Internet balancer server server DB
Client

156. Load Web App
Internet balancer server server DB
Client

157. Test Testing
config node
Data center
Testing
node Website
Testing
node

158. Test Testing
config node
Data center
Testing
node Website
Testing
node

159. Test Testing
config node
Data center
Testing
node Website
Testing
node

160. Test Testing
config node
Data center
Testing
node Website
Reporting
service
Testing
node

168. Three things to watch for
Cached vs. uncached
Scripts vs. puppetry
Simultaneous vs. sequential

169. 0
10
Load time (seconds)
Cached
Uncached

170. 10
Load time (seconds)
3.157s
0
Cached Uncached

171. 13.349s
10
Load time (seconds)
3.157s
0
Cached Uncached

172. Testing script
Script interpreter

173. Testing script
Site: test.com
Page: index.html
Script interpreter

174. Testing script
Site: test.com
Page: index.html
Script interpreter
HTTP GET
www.test.com/index.html

175. Testing script
Site: test.com
Page: index.html
Script interpreter
200 OK
index.html
image.gif
stylesheet.css
etc...

176. Testing script
Site: test.com Test complete
Page: index.html
Script interpreter

177. Browser
controller
Actual browser

178. Browser
controller
DOM actions
(“click on button 4”)
Actual browser

179. Browser
controller
DOM actions
(“click on button 4”)
Actual browser
HTTP GET
www.test.com/index.html

180. Browser
controller
DOM actions
(“click on button 4”)
Actual browser
200 OK
index.html
image.gif
stylesheet.css
etc...

181. Browser
controller
DOM actions DOM contents
(“click on button 4”) (“DIV contains ‘error’”)
Actual browser

183. Simultaneous
5 tests at 15:00

184. Simultaneous Sequential
5 tests from
5 tests at 15:00
15:00 to 15:05

185. Synthetic pros & cons
Pros Cons
Easy to set up Brittle
Only way to test without Detects macro outages, not
actual visitor traffic user events
Can compare to Good geographic & network
competitors coverage costs money,
Easy baseline establishment generates load
Detects a problem before No measurement of traffic
visitors sees it volume
Consistent data over time Places load on the site
under test

186. Ultimately,
Synthetic testing shows you if the site’s working.

187. Real User Monitoring.

188. Synthetic isn’t enough

189. Synthetic isn’t enough

193. Browser Web
server

194. Browser Load Web
balancer server

195. Browser Load Web
Network balancer server
tap

196. Browser Load Web
Network balancer server
tap

197. Browser Load Web
Network balancer server
tap

198. Browser Load Web
Network balancer server
tap

199. Browser Load Web
Network balancer server
tap

200. Browser Load Web
Network balancer server
tap

201. Browser Load Web
Network balancer server
tap
User A

202. Browser Load Web
Network balancer server
tap
User A
User B
User C

203. Browser Load Web
Network balancer server
tap
User A
User B
User C
Visit
history
P1
P2
P3

204. Browser Load Web
Network balancer server
tap
User A
User B
User C
Visit Aggregate
history reports
P1
P2
P3

205. Browser Load Web
Network balancer server
tap
User A
User B
User C
Visit Aggregate Alerts
history reports
!
P1
P2
P3

212. TopN, worstN
RUM tools are excellent for more qualitative data
What’s most broken?
What’s biggest?
What’s slowest?
What’s most inconsistent?

213. RUM pros & cons
Pros Cons
Directly correlated with May require physical
clickstream, analytics installation
Watches everything, not just Can be a privacy risk
the things you know about Doesn’t work if there’s no
Can be used to reproduce traffic
problems Need to filter out your own
Measures traffic as well as visits, crawlers, etc.
performance

214. Ultimately
RUM shows you if the site’s working.

215. Why measure
Tactical, to find and fix
Strategic, to plan/trend
What to measure:
How long until a user can use
the app as you intended?
Part five Where to measure:
Getting the math right WAN, from everywhere
Core app functionality
Tiers of components
How to measure it:
Synth, to ensure it’s working
RUM, to see where it’s broken

216. http://upload.wikimedia.org/wikipedia/commons/0/0e/Count-von-count.jpg

217. 0 10 20 30 40 50 60 70 80 90
Age

218. Average age = 10
0 10 20 30 40 50 60 70 80 90
Age

219. 20
Average age = 10
Count
0
0 10 20 30 40 50 60 70 80 90
Age

220. 20
Average age = 10
Count
0
0 10 20 30 40 50 60 70 80 90
Age

221. 20
Average age = 10
Count
0
0 10 20 30 40 50 60 70 80 90
Age

224. Average varies wildly,
making it hard to
threshold properly or
see a real slow-down.

225. 80th percentile
only spikes once
for a legitimate
slow-down (20%
of users affected)

226. Setting a useful
threshold on
percentiles gives
less false positives
and more real alerts

227. 200
# of requests
0
0 2 4 6 8 10 12 14 16 18 20
Page load time (in seconds)

228. 200
Average latency = 5s
# of requests
0
0 2 4 6 8 10 12 14 16 18 20
Page load time (in seconds)

229. 0
# of requests
200
0 2
4
Average latency = 5s
6
Page load time (in seconds)
8 10 12 14 16
18
95th percentile latency = 19s
20

231. KISS

232. “It can scarcely be
denied that the supreme goal of
all theory is to make the irreducible
basic elements as simple and as few
as possible without having to
surrender the adequate
representation of a single
datum of experience.”
http://media.photobucket.com/image/einstein/derekabril/einstein_010.png

233. “As simple as possible,
but no simpler.”
(FYI, this is irony.)

234. http://www.flickr.com/photos/evilerin/3540381299/
http://www.flickr.com/photos/golf_pictures/2538894627/

235. Login
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Checkout
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Invite
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

236. Login Average 4s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Checkout Average 6s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Invite Average 9s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

237. Login Average 4s
95% 8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Checkout Average 6s
95% 10s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Invite Average 9s
95% 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

238. Login Average 4s
95% 8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Mode 2s
Checkout Average 6s
95% 10s
Mode 5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Invite Average 9s
95% 12s
Mode 1s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

239. Login Average 4s
95% 8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Mode 2s
Checkout Average 6s
95% 10s
Mode 5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Invite Average 9s
95% 12s
Mode 1s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Aggregate? Average 6s
95% 12s
Mode 5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

240. 740 260
Login:
<=4s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

241. 740 260
Total samples 1000
Login:
Below threshold 740
<=4s
Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
target threshold 74%

242. 740 260
Total samples 1000
Login:
Below threshold 740
<=4s
Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
target threshold 74%
370 630
Checkout: Total samples 1000
Below threshold 370
<=5s
Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
target threshold 37%
610 390
Invite: Total samples 1000
<=8s Below threshold 610
Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
target threshold 61%

243. 740 260
Total samples 1000
Login: Below threshold 740
<=4s Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 74%
target threshold
370 630
Checkout: Total samples 1000
Below threshold 370
<=5s Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 37%
target threshold
610 390
Invite: Total samples 1000
Below threshold 610
<=8s Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 61%
target threshold
Aggregate? Total samples 3000
Below threshold 1720
Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
57%
target threshold

244. 740 260
Total samples 1000
Login: Below threshold 740
<=4s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
148 252
Checkout: Total samples 400
Below threshold 148
<=5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
366 366
Invite: Total samples 600
<=8s Below threshold 366
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Total samples 2000
Below threshold 1254
Percent below
63%
target threshold

245. 740 260
Total samples 1000
Login: Below threshold 740
<=4s Weight 1
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
148 252
Checkout: Total samples 400
Below threshold 148
<=5s
Weight 5
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
366 366
Invite: Total samples 600
<=8s Below threshold 366
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Weight 2

246. Login:
<=4s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Checkout:
<=5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Invite:
<=8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Total requests
inside target
Login page 740/1000
Checkout page 148/400
Invite process 366/600

247. Login:
<=4s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Checkout:
<=5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Invite:
<=8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Total requests
inside target Weight Weighted
Login page 740/1000 1 740/1000
Checkout page 148/400 5 740/2000
Invite process 366/600 2 732/1200

248. Login:
<=4s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Checkout:
<=5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Invite:
<=8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Total requests
inside target Weight Weighted
Login page 740/1000 1 740/1000
Checkout page 148/400 5 740/2000
Invite process 366/600 2 732/1200
Total score 2212/4200 53%

249. (Snore?)

250. )" &!!!"
!"#$%&'()*+(,&-*.*/0".1&
!*2,&'.,%0)3.1&
("
%#!!"
#"
%!!!"
'"
$#!!"
&"
$!!!"
%"
#!!"
$"
!" !"
$" %" &" '" #" (" )" *" +" $!"$$"$%"$&"$'"$#"$("$)"$*"$+"%!"%$"%%"%&"%'"%#"%("%)"%*"%+"&!"&$"&%"&&"&'"&#"&("&)"&*"
4#5&
,-./0" 12-34-5.602" 14789:12-34-5.602;"

251. )" &!!!"
!"#$%&'()*+(,&-*.*/0".1&
!*2,&'.,%0)3.1&
("
%#!!"
#"
71% correlation
%!!!"
'"
$#!!"
between traffic
&"
$!!!"
%"
$"
!"
#!!"
!"
and latency.
$" %" &" '" #" (" )" *" +" $!"$$"$%"$&"$'"$#"$("$)"$*"$+"%!"%$"%%"%&"%'"%#"%("%)"%*"%+"&!"&$"&%"&&"&'"&#"&("&)"&*"
4#5&
,-./0" 12-34-5.602" 14789:12-34-5.602;"
If you have traffic predictions, and latency
is correlated with performance, you may
be able to estimate performance in the
future from the business plan.*
*It’s seldom this simple.

254. Baselines
Establish an agreed-upon set of metrics, and always
compare to these baselines.
What does “normal” look like?
Weekly variance? Seasonality?

255. Why measure
Tactical, to find and fix
Strategic, to plan/trend
What to measure:
How long until a user can use
the app as you intended?
Part six Where to measure:
Targeting metrics WAN, from everywhere
Core app functionality
to your audience Tiers of components
How to measure it:
Synth, to ensure it’s working
RUM, to see where it’s broken
Get the math right

256. Your goal is
to be clearly
understood.

257. How technical
are they?
Your goal is
to be clearly
understood.

258. How technical
are they?
Your goal is
How will they
to be clearly use it?
understood.

259. How technical
are they?
To fix
Your goal is something
How will they
to be clearly use it?
understood.

260. How technical
are they?
To fix
Your goal is something
How will they To escalate
to be clearly use it? to others
understood.

261. How technical
are they?
To fix
Your goal is something
How will they To escalate
to be clearly use it? to others
understood. To plan the
future

262. How technical
are they?
To fix
Your goal is something
How will they To escalate
to be clearly use it? to others
understood. To plan the
future
Translate to
their jargon

263. How technical
are they?
To fix
Your goal is something
How will they To escalate
to be clearly use it? to others
understood. To plan the
future
What words do Translate to
they use? their jargon

264. By timeframe
Type of
metric Timeframe Delivery Detail

265. By timeframe
Type of
metric Timeframe Delivery Detail
Break/fix
monitoring

266. By timeframe
Type of
metric Timeframe Delivery Detail
Break/fix
monitoring
Daily
reports

267. By timeframe
Type of
metric Timeframe Delivery Detail
Break/fix
monitoring
Daily
reports
Quarterly
planning

268. By timeframe
Type of
metric Timeframe Delivery Detail
Break/fix Push alerts Simple
Urgent
monitoring to PDA messages
Daily
reports
Quarterly
planning

269. By timeframe
Type of
metric Timeframe Delivery Detail
Break/fix Push alerts Simple
Urgent
monitoring to PDA messages
Daily Historical
Automated Mail PDF
reports context
Quarterly
planning

270. By timeframe
Type of
metric Timeframe Delivery Detail
Break/fix Push alerts Simple
Urgent
monitoring to PDA messages
Daily Historical
Automated Mail PDF
reports context
Quarterly Part of big
Prepared Slide deck
planning picture

271. By medium
Where will this wind up?
Dashboard
NOC screen
Log file
Someone’s
spreadsheet
Inbox
http://www.flickr.com/photos/warrenski/4190341621/

272. Why measure
Tactical, to find and fix
Strategic, to plan/trend
What to measure:
How long until a user can use
the app as you intended?
Part seven Where to measure:
Marching orders WAN, from everywhere
Core app functionality
Tiers of components
How to measure it:
Synth, to ensure it’s working
RUM, to see where it’s broken
Get the math right

273. So
what
should
you Some homework.
do?

274. First
Meet your analytics team
Find out
What are the key goals they’re monitoring?
Where are visitors coming from?
What are the most common entrance and exit
pages?

275. Second
Pick the three processes, pages, or functions that
matter most to you
Landing pages, or part of a conversion funnel

276. Third
Set up monitoring of:
Your site from many places (synthetic testing)
Your top 3 core business processes (synthetic or
RUM)
Your important infrastructure tiers (from agents +
synthetic, or RUM)

277. Fourth
Wait a week or two
To establish a baseline
To detect seasonal variance
To show others and get buy-in

278. Fifth
While you’re waiting, understand the elements latency
and how they affect your performance
DNS
SSL
Network latency
Host (server) latency
Client page load time

279. Set a target threshold
Now that you have an idea of what “normal” is, set a
threshold
... but not just any threshold.

280. The login page Function
will have a total latency Metric
of under 4 seconds Target
with a cached browser copy User situation
from any US branch office Testing point
95% of the time Percentile
weekdays, 8AM ET to 6M PST Time window
by synth test at 5m intervals Collection type

281. Apdex
score
= ( ) (
Satisfied
requests
+
Tolerating
requests
/2 )
All requests

282. How Apdex works
Frustrated: over 8 seconds
Tolerating: 2-8 seconds
Satisfied: 0-2 seconds

283. How Apdex works
Frustrated: 5 hits
Total requests: 100
Tolerating: 30 hits
(65) + (30/2)
= 0.80
100
Satisfied: 65 hits

284. Train your audience
Visit key stakeholders and walk them through the
report
Get them used to the information
In the same format
At the same time
From the same place

285. Put monitoring into your
release cycle
Talk to the development team
Adding instrumentation
Identifying new code functions that need testing
Verifying whether optimization worked

286. Part eight
Some tools to check out

287. Paid

288. Synthetic
Keynote Systems
Gomez
Webmetrics
Alertsite
Dotcom Monitor
Pingdom
...and many others

289. RUM
Client-side
AJAX (Gomez, Coradiant
TrueSight Edge) Full
Agent-based (Aternity) disclosure: We
both worked at
Inline (sniffer/tap) Coradiant.
Coradiant, Tealeaf, Beatbox(HP),
Atomic Labs, Compuware
Apdex
Server-side (logfile, agent)

290. Analytics
Omniture
Webtrends
Coremetrics
Woopra
etc. (lots of specialization)

291. Open Source

292. Firebug
getfirebug.com

293. Firebug
Also: Webkit inspector,
getfirebug.com Google Page Speed

294. Google Analytics
analytics.google.com

295. webpagetest.org

296. Monitor.us
(Free ain’t
pretty, and
pretty ain’t
free, but it
works.)
mon.itor.us

297. AJAX measurement libraries
Collecting from visitors:
Jiffy (http://code.google.com/p/jiffy-web/)
AJAX client sends measurements to Apache collector
Other resources
ZK-Gazer (http://code.google.com/p/zk-gazer/)
http://www.ajaxperformance.com/ (Ryan Breen)
http://www.opensourcetesting.org/performance.php

298. YSlow
http://justtalkaboutweb.com/wp-content/uploads/2008/06/yslow.gif
http://events.stanford.edu/events/196/19695/souders.jpg

299. Sites
Dashboard Juice analytics’ Dashboard
spy blog Insight’s gallery
Simple Complexity

300. Part nine
Planning for the future

301. AJAX

302. AJAX
As for your male and female
slaves whom you may have:
you may buy male and female
slaves from among the nations
that are around you.
- Leviticus 25:44

303. http://www.flickr.com/photos/farhannasir/4577508824/
Mobility

304. http://www.flickr.com/photos/andrewparnell/2738598951/

305. GET index.html HTTP/1.1
Host: www.stockprice.com
Cookie: sessionID=KDF74INED6
Accept: */*
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
Transitional//EN" "http://www.w3