Grafana/Graphite provides lightweight application performance monitoring of internal services and metrics. Rigor offers synthetic performance monitoring of selected pages from different locations. SOASTA mPulse enables real user monitoring to collect performance data from real browsers. SOASTA Data Science Workbench allows complex analysis of performance data through notebooks to answer questions about page groups, user paths, and conversion impact. Tools should be used together to gain internal and external views of performance from synthetic and real user perspectives.

1. Performance Monitoring
at Spreadshirt

2. Why is performance relevant?

3. Performance has an
impact on conversion rate ...

4. http://de.slideshare.net/devonauerswald/walmart-pagespeedslide

5. … and other business metrics
● bounce rate
● cart size
● revenue
● time on site
● page views
http://www.soasta.com/whitepapers/time-is-money-the-business-value-of-web-performance/
Time is Money, The Business Value of Web Performance, Tammy Everts
● user satisfaction
● user retention
● organic search traffic
● brand perception
● productivity

6. How can performance be measured?

7. Core Metrics Overview
time to first byte

8. Core Metrics Overview
time to first byte start render time

9. Core Metrics Overview
time to first byte start render time
above the fold time

10. Core Metrics Overview
time to first byte start render time
above the fold time page load time

11. Metric Details
time to first byte start render time page load time fully loaded
above the fold time

12. Metric Details
backend frontend

13. Metric Details
DNS lookup
Init TCP connection
Server processing
Content download

14. Metric Details
Number of resources and transfered bytes +
order of resources

15. Which performance tools can be useful in
software development lifecycle?

16. Software Development Lifecycle
Lifecycle image taken from SOASTA tool overview

17. Data Analysis &
Optimization Tools
Synthetic Performance Monitoring &
Real User Monitoring (RUM) &
Application Performance Monitoring (APM)
Load Testing &
Optimization Tools
Performance Tools

18. Performance Tools at Spreadshirt
Grafana/Graphite

19. Performance Tools at Spreadshirt
Data Analysis
Synthetic Performance
Monitoring
Real User Monitoring
(RUM)
Grafana/Graphite
Application Performance
Monitoring (APM)

20. Why do we have an
Application Performance Monitoring tool
and why isn’t it enough?

21. Machine
Application Performance Monitoring (APM)
Service
Monitoring
Server
Monitoring Data
Send Metrics
Write Monitoring Data Read Monitoring Data
and Aggregations
Browser
Developer
Request
Send Metrics
Service
Agent
Machine
Agent
Monitoring
Website/
Application

22. Pros
● Good overview on performance of monitored machines and services
● Machine level information on cpu, memory, disk usage
● Service level information like JVM heap usage, number of threads, etc.
● Resource level information on response times, throughput, error rates
● Analysis tools allow to quickly find performance problems
● End-to-end drill down on a per request basis allows to quickly find root cause of
problems

23. Cons
● Complex to set up in a microservice environment as each service and machine needs
to be monitored with an agent
● No information on how pages load in real browser environments
● No information on how my DNS provider or CDN works
● No information on how 3rd party libraries behave in browser

24. Application Performance Monitoring
provides view on internal service and
machine performance
synthetic and real user monitoring
and why do we need both?

25. What is the difference between
Synthetic and Real User Monitoring
and why do we need both?

26. Synthetic Performance Monitoring
Spreadshirt
Website
Monitoring
Server
Monitoring
Website
Monitoring Data
Poll
Page
Write Monitoring Data Read Monitoring Data
and Aggregations
Browser
Developer
Request

27. Pros
● Provides information about page load times in real browsers
● Controlled test environment
● Defined test locations
● Defined frequency
● Traffic independent
● Allows to check availability, speed, errors
● Allows to monitor 3rd party libraries, DNS provider and CDN

28. Cons
● Only information about configured pages
● No information about speed, errors on different customer environments like
○ browser
○ device
○ country
○ operating system
○ ...
● No information about traffic distribution
● No connection to business metrics (like conversion rate)

29. Real User Monitoring (RUM)
Spreadshirt
Website
Browser
Monitoring
Website
Monitoring Data
Request
Page
Write Monitoring Data
Read Monitoring Data
and Aggregations
Browser
Developer
Request
Monitoring
Javascript
Monitoring
Server
Send
Beacon

30. Pros
● Runs on end users browsers and collects information about speed and errors on
different
○ browsers
○ devices
○ countries
○ operating systems
○ …
● Allows to track real user traffic in real time
● Allows to track business metrics (conversion rate) in connection to performance
metrics
● Allows to track 3rd party libraries, DNS provider and CDN

31. Cons
● Provides data on pages with traffic only
● Uncontrolled environment
● Monitoring Javascript needs to be added to each page
● Single page apps (SPA) need to be monitored individually

32. Synthetic and Real User Monitoring
complement each other
and provide outside view on performance
synthetic and real user monitoring
and why do we need both?

33. Which features do the available
monitoring tools provide and how do
they work?

34. Application Performance Monitoring
light with Grafana/Graphite

35. Grafana/Graphite Features
● Available on https://grafana.sprd.net resp. https://graph.server.lej.sprd.net
● Light version of Application Performance Monitoring
● Allows to track all kinds of metrics (not only performance related ones)
● Allows to create custom dashboards
● Allows to visualize metrics in different ways (time lines, numbers, tables, ...)
● Allows to run more complex analysis with the available Graphite functions (like top
10 slowest resources, ...)
● Allows to define and visualize thresholds
● Does not allow to define alerts yet

36. Grafana Dashboard
Favorite dashboards Available dashboards
Search dashboard
Create new dashboard

37. Grafana Shopping Overview
Throughput metric as number
Response time metric with
exceeded threshold
Linked to dashboard
with details (on click)
Add panel

38. Grafana UMP Details
Response time metric p99 as timeline
Response time metric mean as timeline
Stacked throughput as timeline
Edit panel

39. Grafana UMP Details Configuration
Response time p99 metric
Response time mean metric

40. Synthetic Performance Monitoring
with Rigor

41. Rigor Features
● Available on https://monitoring.rigor.com
● Allows to configure uptime, real browser and even selenium-style checks
● Allows to measure from different locations with different frequencies
● Allows to define thresholds on measured metrics
● Provides dashboards to visualize performance data
● Provides http archive on measurement and visualization as waterfall diagram
● Collects all kinds of performance metrics
● Allows to create and distribute performance reports
● Provides tags to be used in dashboards and for conducting actions
● Allows to alert via e-mail, phone, sms, chat, web hook
● Provides status page
● Provides API

42. Rigor Dashboard
Real browser checks
Access check

43. Rigor Real Browser Checks
Add check
Configure checkAccess check

44. Rigor Real Browser Configuration
Configure URL
Configure frequency
Configure alert

45. Rigor Real Browser Check Configuration
Configure locations
Configure thresholds

46. Rigor Real Browser Check Dashboard
Switch view
Access measurement
Adjust timeDisplay events
System event

47. Rigor Real Browser Check Page Performance Dashboard
Select metric
Access measurement

48. Rigor Real Browser Check Domain Performance Dashboard
Select metric
Access measurement

49. Rigor Real Browser Check Waterfall Diagram
Open details

50. Rigor Executive Performance Dashboard
Access check

51. Rigor Operations Performance Dashboard
Access check

52. Rigor Custom Tag Performance Dashboard
Select tag

53. Rigor Custom Tag Performance Dashboard
Access check

54. Rigor Status Page - https://monitoring.rigor.com/status/218
Current load time
History

55. Rigor Custom Report Configuration
Select checks
Select date range
Select metric

56. Rigor Custom Report Visualization
Access check

57. Rigor E-Mail Alert
Access check
Current state

58. Rigor E-Mail Report
Access check

59. Rigor Rocket.Chat Integration
Access check

60. Real User Monitoring
with SOASTA mPulse

61. SOASTA mPulse Features
● Available on https://mpulse.soasta.com/concerto/Central
● Allows to configure applications per domain (e.g. for spreadshirt.de)
● Allows to configure page groups, metrics, timers and dimensions
● Collects performance metrics available to browsers
● Displays real-time performance data
● Provides dashboards to visualize performance data for different roles
● Allows to create custom dashboards
● Allows to filter data by all kinds of dimensions, e.g. browser, device, country
● Provides means to visualize complex correlations
● Allows to integrate data from other sources, like Rigor
● Allows to create and distribute reports
● Allows to alert via e-mail, chat, web hook …

62. SOASTA Applications
Create application
Access application

63. SOASTA Application Configuration
Access page groups

64. SOASTA Page Group Configuration
Configure page groups
Test page

65. SOASTA Metrics Configuration
Configure metric

66. SOASTA Timers Configuration
Configure timer

67. SOASTA Dimensions Configuration
Configure dimension

68. SOASTA Javascript Plugin
Add Boomerang Javascript
Add custom Javascript
code for handling SPA

69. SOASTA Javascript Beacon
Loaded Boomerang Javascript
Sent beacon

70. SOASTA Dashboards
Pre-built dashboards
Custom dashboards
Add dashboard
Access dashboard

71. SOASTA Summary Dashboard
Filters for drill down
Page load vs page views
Key metrics and timers
Filter statistics
Page load by country

72. SOASTA Summary Dashboard Configuration
Show available widgets
Show dashboard configuration
Drag & drop
to add to dashboard
Select to configure widget
Configure widget
Configure default filter

73. SOASTA Showcase Dashboard
Live beacons
Live page load and back-end time
Live statistics

74. SOASTA DevOps Dashboard

75. SOASTA Custom Revenue Dashboard
Business relevant timers and metrics
Revenue vs. bounce rate
Page views vs. page load
Orders

76. SOASTA Custom Page Load Times Dashboard
Page load mean
Page load percentiles

77. SOASTA Custom Shopping Team Dashboard
Backend time vs. frontend time vs. page views
Page load time by page type
Load times from Rigor (external data)

78. SOASTA What-If Dashboard
Move sliders to see impact of
improvements on revenue
Sessions with average load time vs.
conversion rate per average load time

79. SOASTA Waterfall Dashboard
Found beacons for selected filters
Resource timings for selected beacon
Access to beacon data

80. SOASTA Waterfall Dashboard Performance Data

81. SOASTA Waterfall Dashboard Request Data

82. SOASTA Waterfall Dashboard Page Construction Data

83. SOASTA Reports
Add report
Custom reports

84. SOASTA Report Configuration
Configure schedule
Configure recipient
Drag & drop dashboard

85. SOASTA Report E-Mail
Configured dashboard
as image or link

86. SOASTA Alerts
Add Alert
Custom Alerts

87. SOASTA Alert Configuration
Configure data event
Configure recipient
Configure message
Drag & drop elements

88. SOASTA Alert E-Mail
Link to dashboard that
provides more details

89. SOASTA Rocket.Chat Integration
Link to dashboard that
provides more details
Alert from SOASTA

90. What if there are more complex
questions to answer?

91. Questions like ...
● On which page groups do I have performance problems?
● What are the most used path through my application regarding the defined
conversion goal?
● Which page groups are relevant regarding my defined conversion goal (e.g. order)?
● On which environments do I have performance problems (devices, browsers,
countries, operating systems)?
● Which page resources on my site are slow?
● Which page groups should I optimize first?

92. Data Analysis
with SOASTA Data Science Workbench
(DSWB)

93. Data Science Workbench (DSWB)
Soasta
DSWB
Importer
Monitoring
Data
Browser
Analyst
Request
AWS
Redshift
Cluster
Page Data
Resource
Data
Soasta
DSWB
(Julia)Workbench
Request
Data
Import

94. Data Science Workbench Features
● Available on https://spreadshirt.soasta-dswb.com
● Soasta DSWB is an environment that allows to analyze and visualize
performance data
● Runs on AWS Redshift Cluster with Julia as programming language
● Packages for Julia available to access databases, work with data, etc.
● Soasta DSWB provides additional packages for data analysis, drawing and
charting

95. SOASTA DSWB Overview
Select running notebook

96. SOASTA DSWB Notebook
Cell with default connection configuration
for spreadshirt.de data
Execute selected cell
(or Shift + Enter)
Tables with page and resource data
for spreadshirt.de
Selected time frame
Documentation

97. SOASTA DSWB Page Load Co-occurrences
Histogram that shows page views vs.
page load for selected time frame

98. SOASTA DSWB Page Load Co-occurrences
Visualization of factors that
influence load times

99. SOASTA DSWB Page Load Treemap
Treemap that visualizes load times in relationship
to page views for selected filter hierarchy

100. SOASTA DSWB User Path Sunburst Chart
Visualization of all user click paths
towards checkout success page
(means measured conversion goal)

101. SOASTA DSWB Conversion Impact Chart
Visualization of load time relevance of page
group in order to reach conversion goal
(means reaching checkout success page)

102. SOASTA DSWB Conversion vs Load Time Chart
Visualization of average session load time for converted and unconverted
sessions (means sessions that reached checkout success page or not) vs
conversion rate for average session load time

103. SOASTA DSWB Data Access
Visualization of plain beacon data for spreadshirt.de based
on an SQL executed against AWS Redshift Cluster

104. So, which tool should I use
for what again?

105. Conclusion
● Use Grafana/Graphite (and Icinga) to get information about internal services like
response times, throughput and error rates
● Use tools like JProfiler to debug performance problems
● Use Rigor to get outside view on selected pages regarding page response times and
availability
● Use SOASTA mPulse to get extended view on page load time behavior in real user
environments
● Use SOASTA DSWB as planning tool to answer more complex questions regarding
performance and plan the next steps