Become a RUM Runner
Use Real User Monitoring
to learn what life is like
as one of your own customers
Vesna Vuynovich Kovach
@Vesna_V_K

 What it is
 What it’s not
 Why you need it
 How you can start doing it
Real-User Monitoring (RUM)

 PHP developer at OfficeSupply.com
 Freelance writer and consumer tech editor
 eHow Tech, Chron.com (Houston Chronicle), SFGate
(San Francisco Chron.), LIVESTRONG.com
 Isthmus, Wisconsin Trails,
Corporate Report Wisconsin
 Maintain a few sites:
 how-to-cook-with-vesna.com
 atheist-books.com
About me

 What is Application Performance Management?
 What do my end users experience?
 How to interpret charts and terminology
 What is Apdex?
 How to shop for a Web monitoring service
 Ideas and resources for rolling your own
What you’ll learn

 Network bandwidth
 Network latency
 Database access time
 CPU usage levels
 Memory usage
What do you measure?

The folks in IT
Who cares about these things?

The end user’s experience
What matters?

“In application performance, the end user’s experience
is king”
-- The Five Essential Elements of Application Performance Monitoring
Don Jones
Quest Software
www.realtimepublishers.com/book?id=168
What matters?

 Concerned with metrics of End User Experience (EUE)
 Also known as:
 Passive top-down monitoring
 Real-time application monitoring
 An element of application performance management
Real User Monitoring

 APM
 A.K.A. “application performance monitoring”
 Part of website monitoring
 APM asks:
 how is an instance of an application performing?
Application Performance
Management

Anatomy of APM

Top-Down Monitoring
Real User
 Humans (or wild users)
 Passive
 Find problems that have
occurred
 All events, all the time
 Only works if your site has
traffic
Synthetic
 Robots
 Active
 Find problems before they
occur
 Scheduled events
 No real visitor traffic needed

 Measure
 Performance
 Availability
 Diagnose
 Individual incidents
 Track
 Impact of change
What RUM can do

“Is it broken for anyone, anywhere?”
-- Complete Web Monitoring
Alistair Croll & Sean Power
O’Reilly
books.google.com/books?isbn=0596551355
The question for RUM

 Browser differences
 Unexpected navigation
 Untested site area
 Brief outage
 Different network connection
 Different data
 Intermittent problem
RUM Over Robots

 RUM and synthetic testing are complementary
 A cautionary tale from Website Monitoring
 The synthetic testing looked great
 But no one was buying the product
 Honchos: “Fire the bum!”
 RUM revealed the truth
 The product was SO popular, the purchase requests were
timing out
 Company beefed up infrastructure
RUM’s Robot Friend

Apdex
What the heck is

 Application Performance Index
 Metric for measuring user satisfaction
 An open-source standard
 Apdex Alliance
 Collaboration of several companies
 Developed Apdex spec by consensus
 Has become a standard measure since 2008
Apdex

 A metric of user satisfaction
 For transactional applications
 Unit-neutral
 Operates within a fixed range
 0 (worst) to 1 (best)
 Illuminates equivalence
Apdex

 Specific values of the index report the same user
experience across any application, user group, or
enterprise
 Example: 0.5
 Equivalent user experiences observed by different
measurement and reporting tools will report the
same value
- From the Apdex Specification V1.1, Apdex Alliance, Inc.
Apdex Objectives

Apdex term General Def. HTML/HTTP equiv
Session Continuous period of
interaction time
Ex: cookie assignment (not
really HTML or HTTP)
Process A series of user interactions +
User Think Times.
“Transaction” “App Use Case”
A series of Web pages for a
defined app function (ex: buy
a book)
Task Each user interaction with the
app during a Process. “User
Wait Time” “Application
Response Time.”
A container and 0 or more
component objects, and the
time required to download.
Task Chain A defined sequence of Tasks.
Sum of Task times only. (No
user think time.)
A defined sequence of Web
pages and their download
times.
Turn Each interaction needed to
generate a user response or
Task. A client-server request-
driven round trip.
“Chattiness”
HTTP GETs for parts of a Web
page. Each object will
generate a Turn.
Protocol DNS lookups, TCP ACKS, etc. TCP is the protocol supporting
HTTP
Packet Smallest unit of interaction --
Transactional App Taxonomy

 Three performance zones
 Satisfied
 Tolerating
 Frustrated
Apdex Zones

 Two thresholds (each one a measure of time, usually
in seconds)
 T
 Defines the boundary between satisfied and tolerating
 F
 Defines the boundary between tolerating and frustrated
 F is four times T, by definition
Apdex Thresholds

 Count the number of samples in each zone relative to
all samples
 If T = 4 seconds and all 100 users get transactions <4
sec, then A[T] = 1
 T = 4 sec means F = 16 sec
 If all 100 users get transactions >16 seconds, then A[T]
= 0
Apdex Index

- Source: New Relic
Computing Apdex

Compute Apdex

Example 1
newrelic.com

Example 2
newrelic.com

 New Relic
 newrelic.com
 SharePath RUM Express (free)
 real-user-monitoring.com
 Compuware APM Real-User Monitoring
 Formerly Gomez, DynaTrace
 compuware.com
 Pingdom
 pingdom.com
 SOASTA mPULSE
 soasta.com/products/mpulse
Products that offer RUM

RUM API

 Vesna_V_K
 Slides will be on joind.in
Q & A