For the Velocity Conference Europe 2012 workshop day this presentation is about the essentials for creation and building a Web Performance dashboard. This with ultimate goal of providing the audience a framework for designing and building a web performance dashboard. The session will cover the following 3 items: Design guidelines: What defines a web performance dashboard? How to make sure it’s actionable and for people to actually use it on day to day basis? Data collection: Why performance data? The various ways there are to collect data (e.g. synthetic versus RUM data, Webpagetest, Mobile) and how to correlate the different types of data and tools Building the dashboard: How to build the actual dashboard, providing an overview of the tools/techniques used At the end of the workshop you will be able to design and build your own dashboard based on the framework provided, or to optimize the current dashboards within your organization.

2. @jeroentjepkema
Performance :: Analytics :: Optimization

3. a Web Performance Dashboard,
up & running in 90 minutes
Key concepts for measuring and presenting performance data
Velocity Conference Europe
October 2, 2012

5. Dashboard design
principles

6. Dashboard design
principles
Collecting
Performance data

7. Dashboard design
principles
Collecting
Performance data
Building the
Web Performance
dashboard

8. Dashboard, FTW!

9. 500
0
1000
1500
2000
2500
3000
00:00:00
01:00:00
02:00:00
03:00:00
04:00:00
05:00:00
06:00:00
07:00:00
08:00:00
09:00:00
10:00:00
11:00:00
12:00:00
13:00:00
14:00:00
15:00:00
16:00:00
17:00:00
18:00:00
19:00:00
20:00:00
21:00:00
22:00:00
23:00:00
Visits
Users

10. 500
0
1000
1500
2000
2500
3000
00:00:00
01:00:00
02:00:00
03:00:00
04:00:00
05:00:00
06:00:00
07:00:00
08:00:00
09:00:00
10:00:00
Alert status
11:00:00
12:00:00
13:00:00
14:00:00
15:00:00
16:00:00
17:00:00
18:00:00
19:00:00
20:00:00
Be Careful
21:00:00
22:00:00
23:00:00
Visits
Users

12. # Payment orders Login process
Payment process Savings

13. January 25 2007, 9:05am
# Payment orders Login process
Payment process Savings

14. January 25 2007, 9:05am
# Payment orders Login process
Payment process Savings

23. http://www.nrc.nl/apps/bigboard/

24. Part 1: Five Design principles for a Web Performance Dashboard

25. #1 - Data for all, not for everything

26. Your goal is to be understood...

27. How technical is your audience?
Your goal is to be understood...

28. How technical is your audience?
Your goal is to be understood... How will they use it?

29. How technical is your audience?
Your goal is to be understood... How will they use it?
How fast can they execute?

30. 3 levels of performance data...

31. Trending

32. Trending
Future
Big picture
Actionable
Capacity
Optimization

33. Service Levels Trending
Future
Big picture
Actionable
Capacity
Optimization

34. Service Levels Trending
Yesterday Future
How did we perform? Big picture
Versus targets? Actionable
Historical context Capacity
Optimization

35. Real Time Service Levels Trending
Yesterday Future
How did we perform? Big picture
Versus targets? Actionable
Historical context Capacity
Optimization

36. Real Time Service Levels Trending
Now Yesterday Future
Everything working? How did we perform? Big picture
Is it me or the internet? Versus targets? Actionable
How to solve it? Historical context Capacity
Simple Optimization
0hours 24hours 7days 1Quarter

37. Real Time Service Levels Trending
Now Yesterday Future
Everything working? How did we perform? Big picture
Is it me or the internet? Versus targets? Actionable
How to solve it? Historical context Capacity
Simple
Data for dashboard Optimization
0hours 24hours 7days 1Quarter

38. #2 - Focus on End User Experience

39. Why? Customer loyalty!

40. Just a
1-second
delay

41. Just a
1-second 7%
loss in conversion
delay
@jeroentjepkema, MeasureWorks

42. Just a
7% 11%
1-second loss in conversion
fewer
delay page views
@jeroentjepkema, MeasureWorks

43. Just a
7% 11%
1-second fewer 16% decrease
loss in conversion in customer satisfaction
delay page views
@jeroentjepkema, MeasureWorks

44. Study conducted by MeasureWorks & idr1, Emerce, 2012
Dutch eRetail100 Dutch eTravel30 Dutch eFinance50
4% 6%
12%
Social Media mentions = Performance KPI

45. Speed
@jeroentjepkema, MeasureWorks

46. Globe Neckermann Sunweb
D-reizen Arke
Shopping for a holiday...
@jeroentjepkema, MeasureWorks

47. www.vrijuit.nl
Page size: 1201Kb
Experience fast? Try... http://bit.ly/SHYZpR
http://blog.dynatrace.com/2012/09/26/why-page-size-matters-even-more-for-mobile-web-apps/
@jeroentjepkema, MeasureWorks

48. Reliability
@jeroentjepkema, MeasureWorks

49. Functional errors
@jeroentjepkema, MeasureWorks

50. Technical errors
@jeroentjepkema, MeasureWorks

51. @jeroentjepkema, MeasureWorks

52. And there’s marketing...

54. Good
campaign

55. Good
gone
campaign

56. Good
gone BAD!
campaign

57. #3 - Provide context

59. What does it mean?

60. What does it mean?
Why is it important?

61. What does it mean?
Why is it important?
Next best action?

62. #4 - Functionality first

63. What do you sell to the end users?

64. Creates common language between
business and IT...

65. Metric: Availability
99% versus 99min
...with understandable targets

66. #5 - Make it visually attractive

67. Group items

68. Use colours to display status

69. Make it move...

70. Design with the end user in mind

71. And the dashboard is...

72. Real Time
And the dashboard is...

73. Real Time
Focused on End User
And the dashboard is...

74. Real Time
Focused on End User
Providing context
And the dashboard is...

75. Real Time
Focused on End User
Providing context
And the dashboard is... Functionality first

76. Real Time
Focused on End User
Providing context
And the dashboard is... Functionality first
Visually attractive

77. All that matters... Show them where it hurts

78. Part 2: Collecting Performance Data

79. Technology:
Responsiveness:
How End Users End User Experience How we build,
measure & optimize
perceive (front-end)
performance performance
@jeroentjepkema, MeasureWorks

80. Complexity of a typical web transaction
Source: Compuware Gomez @jeroentjepkema, MeasureWorks

81. (CLOUD) DATA CENTER INTERNAL USERS INTERNET CUSTOMERS
Third-party/
Cloud Services
Storage DB Servers Web Servers Major Local
This is what you control... Network What you’re blamed for..
ISP
Content
ISP
Delivery Mobile
Load Networks Carriers
Middleware App Balancers
Mainframe Servers Servers
Web Performance Delivery Chain
@jeroentjepkema, MeasureWorks

82. Measuring End User Experience is not
easy...
@jeroentjepkema, MeasureWorks

83. Measuring End User Experience is not
easy... You need diagnostic details for things you can
control and/or change
@jeroentjepkema, MeasureWorks

84. Measuring End User Experience is not
easy... You need diagnostic details for things you can
control and/or change
You need insights in the things you can control, but
do impact your bottom line
@jeroentjepkema, MeasureWorks

85. Measure End User Experience? Outside-in,
from the browser perspective...
@jeroentjepkema, MeasureWorks

86. Performance Measurement Toolkit

87. Lot’s of different user scenario’s
@jeroentjepkema, MeasureWorks

88. @jeroentjepkema, MeasureWorks

89. Changing bandwith: DSL, Mobile, WiFi, etc.

90. Real Devices

91. Real usage...

92. Consistency in measurements
@jeroentjepkema, MeasureWorks

93. Correlating data for root cause analysis
@jeroentjepkema, MeasureWorks

94. Performance Data collection @jeroentjepkema, MeasureWorks

95. Synthethic
Real User Benchmark
Monitoring
Performance Data collection @jeroentjepkema, MeasureWorks

96. Simulate business transactions

97. Via multiple devices & browsers

98. From where multiple locations...

99. Used for error detection & Root Cause Analysis

100. Used for error detection & Root Cause Analysis

101. Used for error detection & Root Cause Analysis

102. Used for error detection & Root Cause Analysis

103. Synthetic monitoring
Metrics collected Cons
Easy collection of data No real user data, unlimited bandwith
while testing
Heartbeat collection of data over time
Combination with CDN
Testing without actual visitor traffic
Detects macro outages, not user
Test from multiple locations
events
Object level details
Detects only what you script
Detailed alerting via Text, Mail or SNMP
No measurement of traffic volume
Root cause analysis data, with error codes,
Places load on the site under test
screenshots, source code and waterfall data
Maintenance of tests takes time

104. Synthetic monitoring
Vendors When selecting:
Alertsite Ease of use with recording scripts
Gomez
Check the API functionalities
Keynote
Level of detail with alert messages
Catchpoint
Global versus local coverage
Siteconfidence
IP Label Reliability of sending alert messages
Websitepulse
Pingdom
Etc...

105. Synthetic monitoring
Vendors When selecting:
Alertsite Ease of use with recording scripts
Gomez
Check the API functionalities
Keynote
Level of detail with alert messages
Catchpoint
Global versus local coverage
Siteconfidence
IP Label Reliability of sending alert messages
Websitepulse For Native Apps:
Pingdom Gomez
Keynote
Etc... ...

106. Ultimately, synthetic monitoring shows you
if your site’s working or not...

107. But, synthetic isn’t enough...

108. Synthetic heartbeat
Real Users
But, synthetic isn’t enough...

109. Real User
Real User Benchmark
Monitoring
Performance Data collection @jeroentjepkema, MeasureWorks

110. Navigation timing
2 ways of measuring...
Browser RUM
Disclaimer: There’s also third category Datacenter RUM, this will not be
covered in this out of the presentation. Contact me if your want details

111. h"p://www.w3.org/TR/naviga2on32ming/5
Some background info:
Navigation timing
http://66.7percentangel.com/2011/12/breaking-down-onload-event-performance-bookmarklet/
http://www.html5rocks.com/en/tutorials/webperformance/basics/
http://www.w3.org/TR/2011/CR-navigation-timing-20110315/#nt-dom-content-event-start

112. http://caniuse.com/nav-timing

113. 1 2 3 4
As pages execute, After onload tag send
Insert tag (.js file) into Pages are requested
tag collects detailed report for
(mobile) web pages from browser/device
performance metrics further analysis
tag.js
tag.js
tag.js
tag.js
Browser RUM

114. Gomez

115. Gomez

116. Lognormal
Soasta

117. Torbit

118. Relies on navigation timing API, custom variables can be added
Google Analytics

119. New Relic

120. Real User monitoring
Metrics collected Cons
Measure ALL pages from ALL users No traffic is no data
Measures traffic as well as performance Needs change to application code
Quantative performance data to analyse user May require physical installation of
satisfaction data storage or data reporter
Data can be directly correlated with Web Sample rate
Analytics
Great for trending and creating the big picture

121. Real User monitoring
Vendors When selecting:
Lognormal Do I need to build my own datastorage?
Gomez Check the API functionalities
Keynote How long is raw data stored
Torbit Interface
Google Analytics Mobile support
Boomerang.js
Oracle
Etc...

122. Real User monitoring
Vendors When selecting:
Lognormal Do I need to build my own datastorage?
Gomez Check the API functionalities
Keynote How long is raw data stored
Torbit Interface
Google Analytics Mobile support
Boomerang.js
Oracle For Native Apps:
Google Analytics
Etc... Gomez
Localytics

123. Ultimately, Real User Monitoring shows you how many
users are affected by bad performance...

124. Real User
benchmarking
Performance Data collection @jeroentjepkema, MeasureWorks

125. First view
11.349s
Real usage
Repeat view
4.357s

126. Average page load time per bandwith (seconds) for Dutch eTravel30
8,8 1,5 mbps
10 mbps
20 mbps
3,9
3,4
56%
13%
Bandwith

127. Average page load time per Browser (seconds) Dutch eTravel30
IE7
6,5
IE8
3,9 IE9
3,6
40%
8%
Devices used

128. Benchmark competitors
arke.nl 5,50 sec
sunweb.nl 3,91 sec
vakantiexperts.nl 2,15 sec
dreizen.nl 6,13 sec
thomascook.nl 1,79 sec
Source: Webpagetest.org, IE9 10Mb up/ 2 Mb down

129. Webpagetest.org

130. Object level
!
!
Optimization tips Webpagetest.org
!

131. Webpagetest.org

132. Mobitest / Akamai

133. Real User benchmarking
Metrics collected Cons
Variety of real browsers and real devices Can be difficult to setup
available for testing
Requires physical installation
Repetitive collection of real usage scenario’s
Scripting (is difficult)
Collect optimization metrics, waterfall and
page speed score
Great for trending and creating the big picture

134. Ultimately, Real User benchmarking gives you periodic
insight in real usage scenario’s...

135. Synthethic monitoring Real User Monitoring Real User benchmarking
Performance Data collection @jeroentjepkema, MeasureWorks

136. Synthethic monitoring Real User Monitoring Real User benchmarking
Used for...
Heartbeat, runs without traffic
Test specific customer journeys
Object level detail
Collect detailed alerts, including
root cause analysis
Desktop/Mobile Site
Gomez
Keynote
Watchmouse
Alertsite
Mobile Apps
Gomez
Keynote
Performance Data collection @jeroentjepkema, MeasureWorks

137. Synthethic monitoring Real User Monitoring Real User benchmarking
Used for... Used for...
Heartbeat, runs without traffic Real usage information from
Test specific customer journeys all users!!
Object level detail Trending/Optimization
Collect detailed alerts, including Business impact
root cause analysis
Desktop/Mobile Site Desktop/Mobile Site
Gomez Gomez
Keynote LogNormal
Watchmouse Torbit
Alertsite Google Analytics
Mobile Apps Mobile Apps
Gomez Gomez
Keynote Localytics
Google Analytics
Performance Data collection @jeroentjepkema, MeasureWorks

138. Synthethic monitoring Real User Monitoring Real User benchmarking
Used for... Used for... Used for...
Heartbeat, runs without traffic Real usage information from Periodic testing of user scenario’s
Test specific customer journeys all users!! with real devices and bandwith
Object level detail Trending/Optimization Optimization details
Collect detailed alerts, including Business impact Competitive scan
root cause analysis
Desktop/Mobile Site Desktop/Mobile Site Desktop/Mobile Site
Gomez Gomez Webpagetest
Keynote LogNormal
Watchmouse Torbit
Alertsite Google Analytics
Mobile Apps Mobile Apps Mobile Apps
Gomez Gomez Perfecto Mobile
Keynote Localytics Device Anywhere
Google Analytics
Performance Data collection @jeroentjepkema, MeasureWorks

139. Synthetic Browser RUM Competition

140. Synthetic vs. RUM
Synthetic vs. RUM
Synthetic Browser RUM Competition

141. Synthetic vs. RUM
Synthetic vs. RUM
You vs. competition
Synthetic Browser RUM Competition

142. Raw data

143. Part 3: Building the dashboard

144. Start with your web analytics tool...

145. Every website has goals
http://www.flickr.com/photos/itsgreg/446061432/

147. Organic Search
Campaigns
Ad Network
Transactional site
Visitor
Offer
€"
Abondenment)
Upselling
Reach
Purchase step 1
€"
Purchase step 2
€"
Mailing,
alerts,
€" promotions
Conversion
€"
Disengagement) Enrolment
Impact)on)site)
€" Negative €" Positive
Goals = Customer journeys

148. Select your top customer journeys

149. Map the customer journeys with your own
web application delivery chain...

150. Journey 1 Journey 2 Journey 3
Synthetic transactions
Tier1 Webserver Webserver Webserver Webserver
Application Application
Tier2 server server
Database
Tier3 Backend Backend Backend
Preferably one per backend used

151. Journey 1 Journey 2 Journey 3
Synthetic transactions
Tier1 Webserver Webserver Webserver Webserver
Application Application
Tier2 server server
Database
Tier3 Backend Backend Backend
Preferably one per backend used

152. Journey 1 Journey 2 Journey 3
Synthetic transactions
Tier1 Webserver Webserver Webserver Webserver
Application Application
Tier2 server server
Database
Tier3 Backend Backend Backend
Preferably one per backend used

153. Journey 1 Journey 2 Journey 3
Synthetic transactions
Tier1 Webserver Webserver Webserver Webserver
For proper alerting, eliminate doubles
Tier2 Application
server
Application
server
Database
Tier3 Backend Backend Backend
Preferably one per backend used

154. Don’t forget third parties

155. Defining service levels & thresholds...

156. First thing is to establishing a baseline:

157. First thing is to establishing a baseline:
A pre-defined set of metrics

158. First thing is to establishing a baseline:
A pre-defined set of metrics
that describes normal behavior

159. First thing is to establishing a baseline:
A pre-defined set of metrics
that describes normal behavior
in order to detect variancies

160. First thing is to establishing a baseline:
A pre-defined set of metrics
that describes normal behavior
in order to detect variancies
and to be comparable within historic context

161. Example...

162. Purchasing a book, Customer journey
must be completed (speed), Metric: Speed
where every page loads under 4 sec., Target: Sec
using IE8 and higher, User scenario
from any location in the Netherlands, User locations
for 95% of all users, Percentile
every day between 6am and 12pm, Window
measured with Real User Monitoring. Collection type
Source: Metrics 101, Velocityconf 2010

163. Repeat this for every customer journey defined...

164. Next, group the content...

165. KISS

167. Layered approach
Online brand Bol.com

168. Layered approach
Online brand Bol.com
Books
Music
Products Toys
etc.

169. Layered approach
Online brand Bol.com
Books
Music
Products Toys
etc.
Search
Services per product Reviews
Purchase
Discounts

170. Layered approach
Online brand Bol.com Overall performance
Books
Music
Products Toys
etc.
Search
Services per product Reviews
Purchase
Discounts

171. Layered approach
Online brand Bol.com Overall performance
Books
Music
Products Toys Detailed performance
etc.
Search
Services per product Reviews
Purchase
Discounts

172. Layered approach
Online brand Bol.com Overall performance
Books
Music
Products Toys Detailed performance
etc.
Search
Services per product Reviews Root cause
Purchase
Discounts

173. Layered approach
Big Picture
Online brand Bol.com Overall performance
Books
Music
Products Toys Detailed performance
etc.
Search
Services per product Reviews Root cause
Purchase
Discounts
Detailed level of performance

174. Mixing it all together...

175. Displayed in dashboard Metrics Group information

176. Displayed in dashboard Metrics Group information
Functionality

177. Displayed in dashboard Metrics Group information
Functionality Service Levels

178. Displayed in dashboard Metrics Group information
Threshholds
Functionality Service Levels
Real Users versus Synthethic

179. Displayed in dashboard Metrics Group information
Threshholds
Functionality Service Levels
Real Users versus Synthethic
Business context

180. Displayed in dashboard Metrics Group information
Threshholds
Functionality Service Levels
Real Users versus Synthethic
Business context Customer Journey

181. Displayed in dashboard Metrics Group information
Threshholds
Functionality Service Levels
Real Users versus Synthethic
Page views
Bouncerate
Business context Customer Journey
Affected users sessions
Competition

182. Displayed in dashboard Metrics Group information
Threshholds
Functionality Service Levels
Real Users versus Synthethic
Page views
Bouncerate
Business context Customer Journey
Affected users sessions
Competition
Real Time Performance

183. Displayed in dashboard Metrics Group information
Threshholds
Functionality Service Levels
Real Users versus Synthethic
Page views
Bouncerate
Business context Customer Journey
Affected users sessions
Competition
Real Time Performance Delivery chain

184. Displayed in dashboard Metrics Group information
Threshholds
Functionality Real Users versus Synthethic
Service Levels
Page views
Bouncerate
Business context Customer Journey
Affected users sessions
Competition
Speed
Application errors
Real Time Performance Content errors Delivery chain
Third party errors
Downtime

185. Designing the infrastructure...

186. Custom reports Mobile Performance
(SLA/Trending) application dashboard
4 4
5
Authentication layer
Vendor API used for both real time
1 information as raw data download
API for data upload to mobile app / 2
2 dashboard 3
3 Import HTML alert mail Alert
Datawarehouse 3
message
Authentication mechanism for both
4 login as selective data transfer
1 1 1 1
Remote access to Datawarehouse for SLA
5 or Trend reporting, based on same data as
real time dashboard
WPT Synthethic RUM Analytics

187. Synthethic RUM

188. Synthethic RUM

189. Synthethic RUM

190. Data model RUM

192. It all looks the same...

193. ...but still so different

194. ...but still so different
Synthethic monitoring

195. ...but still so different
http://bit.ly/MW-namingconvention

196. Authentication layer
http://github.com/symfony/symfony
http://www.symfony-project.org

197. Visuals
https://www.html5rocks.com/en/tutorials/canvas/performance
https://www.highcharts.com
http://zqi.me/vizd3

198. Demo
Demo time

199. Designing the interface...

201. Technical metrics
Navigation

202. Technical metrics
Navigation
Business metrics

203. Technical metrics
This is the part that moves
Navigation
Business metrics

205. Demo:
URL: https://app.measureworks.nl
UID: demo@measureworks.nl
Password: performance

206. Metric: Page speed
Datacollection: RUM
Treshold: Historically based or fixed
Timeframe: Average 5min

207. Metric: Uptime
Datacollection: Synthethic monitoring
Treshold: Page errors (4xx, 5xx)
Timeframe: Average 5min

208. Metric: Third Party uptime
Datacollection: Synthethic monitoring
Treshold: Error objects/domain (4xx, 5xx)
Timeframe: Average 5min

209. Metric: Application error
Datacollection: RUM & Synthethic monitoring
Treshold: # object errors per funnel (4xx, 5xx)
Timeframe: Average 5min

210. Metric: Content error
Datacollection: Synthethic monitoring
Treshold: Object size, Context match, User transaction failure
Timeframe: Average 5min

211. Metric: Pageviews impacted
Datacollection: RUM
Treshold: % of pageviews affected vs. total pageviews
Timeframe: Average 5min

212. Metric: Users affected
Datacollection: RUM & Navigation timing
Treshold: % of browser session vs. bouncerate
Timeframe: Average 5min

213. Metric: Competition ranking
Datacollection: Webpagetest
Treshold: % competitors faster based on
current average page speed
Timeframe: Average 60min

214. URL: https://app.measureworks.nl
UID: velocity@measureworks.nl
PW: performance

215. Become a dashboard rockstar...

216. 1. Look into your webanalytics
Become a dashboard rockstar...

217. 1. Look into your webanalytics
2. Define a performance baseline
Become a dashboard rockstar...

218. 1. Look into your webanalytics
2. Define a performance baseline
3. Start with synthethic monitoring
Become a dashboard rockstar...

219. 1. Look into your webanalytics
2. Define a performance baseline
3. Start with synthethic monitoring
4. Design your dashboard
Become a dashboard rockstar...

220. 1. Look into your webanalytics
2. Define a performance baseline
3. Start with synthethic monitoring
4. Design your dashboard
5. Build a report based on your design
and collect feedback
Become a dashboard rockstar...

221. 1. Look into your webanalytics
2. Define a performance baseline
3. Start with synthethic monitoring
4. Design your dashboard
5. Build a report based on your design
and collect feedback
6. Build a dashboard
Become a dashboard rockstar...

222. 1. Look into your webanalytics
2. Define a performance baseline
3. Start with synthethic monitoring
4. Design your dashboard
5. Build a report based on your design
and collect feedback
6. Build a dashboard
7. Add rum and other datasources
Become a dashboard rockstar...

223. One more things...
Couple

224. of

225. Start with a functional design

226. Organisatie
Create a good team
@jeroentjepkema, MeasureWorks

227. Design with your end users in mind

228. Collect feedback as early as possible

229. The art of deleting

230. Be careful with using API’s

231. Combining datasources... WTF?

232. Recommended...

233. Data Visualization:
Noah Iliinsky/Julie Steel: Desiging Data Visualizations - http://oreil.ly/SryQyV
Real User Monitoring:
Wednesday Oct 3, 11.20am - The 3.5s dash for attention and other stuff we found in RUM
WebPage test:
Thursday Oct 4, 15.30pm - WebPage test, beyond the basics
Join a Web Performance meetup:
Go to www.meetup.com and search for web performance.
For Netherlands go to http://www.meetup.com/Dutch-Web-Operations-Meetup/

234. Questions?

235. Thanks! More questions?
M: jtjepkema@measureworks.nl
T: @jeroentjepkema
W: www.measureworks.nl