More Related Content

Become a RUM Runner

  1. Become a RUM Runner Use Real User Monitoring to learn what life is like as one of your own customers Vesna Vuynovich Kovach @Vesna_V_K
  2.  What it is  What it’s not  Why you need it  How you can start doing it Real-User Monitoring (RUM)
  3.  PHP developer at  Freelance writer and consumer tech editor  eHow Tech, (Houston Chronicle), SFGate (San Francisco Chron.),  Isthmus, Wisconsin Trails, Corporate Report Wisconsin  Maintain a few sites:   About me
  4.  What is Application Performance Management?  What do my end users experience?  How to interpret charts and terminology  What is Apdex?  How to shop for a Web monitoring service  Ideas and resources for rolling your own What you’ll learn
  5.  Network bandwidth  Network latency  Database access time  CPU usage levels  Memory usage What do you measure?
  6. The folks in IT Who cares about these things?
  7. The end user’s experience What matters?
  8. “In application performance, the end user’s experience is king” -- The Five Essential Elements of Application Performance Monitoring Don Jones Quest Software What matters?
  9.  Concerned with metrics of End User Experience (EUE)  Also known as:  Passive top-down monitoring  Real-time application monitoring  An element of application performance management Real User Monitoring
  10.  APM  A.K.A. “application performance monitoring”  Part of website monitoring  APM asks:  how is an instance of an application performing? Application Performance Management
  11. Anatomy of APM
  12. Top-Down Monitoring Real User  Humans (or wild users)  Passive  Find problems that have occurred  All events, all the time  Only works if your site has traffic Synthetic  Robots  Active  Find problems before they occur  Scheduled events  No real visitor traffic needed
  13.  Measure  Performance  Availability  Diagnose  Individual incidents  Track  Impact of change What RUM can do
  14. “Is it broken for anyone, anywhere?” -- Complete Web Monitoring Alistair Croll & Sean Power O’Reilly The question for RUM
  15.  Browser differences  Unexpected navigation  Untested site area  Brief outage  Different network connection  Different data  Intermittent problem RUM Over Robots
  16.  RUM and synthetic testing are complementary  A cautionary tale from Website Monitoring  The synthetic testing looked great  But no one was buying the product  Honchos: “Fire the bum!”  RUM revealed the truth  The product was SO popular, the purchase requests were timing out  Company beefed up infrastructure RUM’s Robot Friend
  17. Apdex What the heck is
  18.  Application Performance Index  Metric for measuring user satisfaction  An open-source standard  Apdex Alliance  Collaboration of several companies  Developed Apdex spec by consensus  Has become a standard measure since 2008 Apdex
  19.  A metric of user satisfaction  For transactional applications  Unit-neutral  Operates within a fixed range  0 (worst) to 1 (best)  Illuminates equivalence Apdex
  20.  Specific values of the index report the same user experience across any application, user group, or enterprise  Example: 0.5  Equivalent user experiences observed by different measurement and reporting tools will report the same value - From the Apdex Specification V1.1, Apdex Alliance, Inc. Apdex Objectives
  21. Apdex term General Def. HTML/HTTP equiv Session Continuous period of interaction time Ex: cookie assignment (not really HTML or HTTP) Process A series of user interactions + User Think Times. “Transaction” “App Use Case” A series of Web pages for a defined app function (ex: buy a book) Task Each user interaction with the app during a Process. “User Wait Time” “Application Response Time.” A container and 0 or more component objects, and the time required to download. Task Chain A defined sequence of Tasks. Sum of Task times only. (No user think time.) A defined sequence of Web pages and their download times. Turn Each interaction needed to generate a user response or Task. A client-server request- driven round trip. “Chattiness” HTTP GETs for parts of a Web page. Each object will generate a Turn. Protocol DNS lookups, TCP ACKS, etc. TCP is the protocol supporting HTTP Packet Smallest unit of interaction -- Transactional App Taxonomy
  22.  Three performance zones  Satisfied  Tolerating  Frustrated Apdex Zones
  23.  Two thresholds (each one a measure of time, usually in seconds)  T  Defines the boundary between satisfied and tolerating  F  Defines the boundary between tolerating and frustrated  F is four times T, by definition Apdex Thresholds
  24.  Count the number of samples in each zone relative to all samples  If T = 4 seconds and all 100 users get transactions <4 sec, then A[T] = 1  T = 4 sec means F = 16 sec  If all 100 users get transactions >16 seconds, then A[T] = 0 Apdex Index
  25. - Source: New Relic Computing Apdex
  26. Compute Apdex
  27. Example 1
  28. Example 2
  29.  New Relic   SharePath RUM Express (free)   Compuware APM Real-User Monitoring  Formerly Gomez, DynaTrace   Pingdom   SOASTA mPULSE  Products that offer RUM
  30. RUM API
  31.  Vesna_V_K  Slides will be on Q & A

Editor's Notes

  1. Now let’s dive down into what that means.
  2. Why RUM, rather than just lots of synthetic testing Different net conn: firewall, proxy use, geo. distance Different data: hitting the back button three times, cookie length, cart contents, long name Intermittent: With one of three bad servers, robot has 2/3 chance of good hit, cust has 1/3 chance of bad
  3. And everyone lived happily ever after
  4. There are 7 Apdex objectives in total. These two explain what I mean by “illuminating equivalence.”