SlideShare a Scribd company logo
1 of 340
Metrics 101
What to watch
What we’ll cover

 Why collect metrics
 Understanding web latency
 How to target your findings
 Concrete steps to get started
Part one
Why collect metrics?
http://www.flickr.com/photos/chidorian/12411641/
Downtime costs
Downtime costs


 eBay offline ($90K/h)
                        22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)
Downtime costs


 eBay offline ($90K/h)
                           22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)

  Financial company down ($100K/h)
                        53.2% of finance companies lose over $100,000/hour (nextslm.org)
Downtime costs
                                          Amazon offline ($1M/h)
                                      Amazon loses nearly $1M/hour if down (NYT, 2008)




 eBay offline ($90K/h)
                           22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)

  Financial company down ($100K/h)
                        53.2% of finance companies lose over $100,000/hour (nextslm.org)
Downtime costs
                                           Amazon offline ($1M/h)
                                       Amazon loses nearly $1M/hour if down (NYT, 2008)
Network downtime ($42K/h)
                                1 hour of network downtime costs $42,000 (Gartner, 2003)

  eBay offline ($90K/h)
                            22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)

   Financial company down ($100K/h)
                         53.2% of finance companies lose over $100,000/hour (nextslm.org)
Downtime costs
                                           Amazon offline ($1M/h)
                                       Amazon loses nearly $1M/hour if down (NYT, 2008)
Network downtime ($42K/h)
                                1 hour of network downtime costs $42,000 (Gartner, 2003)

  eBay offline ($90K/h)
                            22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)

   Financial company down ($100K/h)
                         53.2% of finance companies lose over $100,000/hour (nextslm.org)




Let’s say $50K/h if you’re serious.
Availability Downtime/year Loss @$50K/h
      90% %    36.5 days   Can$43,800,000
     95%     18.25 days    Can$21,900,000
     98%     7.30 days     Can$8,760,000
     99%     3.65 days     Can$4,380,000
   99.5%     1.83 days     Can$2,196,000
   99.8%    17.52 hours     Can$876,000
   99.9%     8.76 hours     Can$438,000
  99.95%     4.38 hours     Can$219,000
  99.99%    52.6 minutes     Can$43,833
 99.999%    5.26 minutes      Can$4,383
99.9999%   31.5 seconds         Can$438
Availability Downtime/year Loss @$50K/h
      90% %    36.5 days   Can$43,800,000
     95%     18.25 days    Can$21,900,000
     98%     7.30 days     Can$8,760,000
     99%     3.65 days     Can$4,380,000
   99.5%     1.83 days     Can$2,196,000
   99.8%    17.52 hours     Can$876,000
   99.9%     8.76 hours     Can$438,000     Less than
  99.95%     4.38 hours     Can$219,000     an hour a
  99.99%    52.6 minutes     Can$43,833       year
 99.999%    5.26 minutes      Can$4,383
99.9999%   31.5 seconds         Can$438
Availability Downtime/year Loss @$50K/h
      90% %    36.5 days   Can$43,800,000
     95%     18.25 days    Can$21,900,000
     98%     7.30 days     Can$8,760,000
     99%     3.65 days     Can$4,380,000
   99.5%     1.83 days     Can$2,196,000
   99.8%    17.52 hours     Can$876,000
   99.9%     8.76 hours     Can$438,000     Less than
  99.95%     4.38 hours     Can$219,000     an hour a
  99.99%    52.6 minutes     Can$43,833       year
 99.999%    5.26 minutes      Can$4,383     Less than
99.9999%   31.5 seconds         Can$438     a minute a
                                               year
Harris poll conducted by Tealeaf in 2008
You really don’t want web
        users to call you.
         $15


         $12


           $9


           $6


           $3


           $0
                      Web self-service             IVR   Email            Live phone

                                                         Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service”            Low           Average      High
You really don’t want web
        users to call you.
         $15


         $12


           $9


           $6


           $3

                        Can$0.24
           $0
                      Web self-service             IVR   Email            Live phone

                                                         Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service”            Low           Average      High
You really don’t want web
        users to call you.
         $15


         $12


           $9


           $6


           $3

                        Can$0.24                   Can$0.45
           $0
                      Web self-service                IVR     Email            Live phone

                                                              Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service”                 Low           Average      High
You really don’t want web
        users to call you.
         $15


         $12


           $9


           $6

                                                              Can$3.00
           $3

                        Can$0.24                   Can$0.45
           $0
                      Web self-service                IVR       Email            Live phone

                                                                Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service”                   Low           Average      High
You really don’t want web
        users to call you.
         $15


         $12


           $9


           $6                                                                    Can$5.50
                                                              Can$3.00
           $3

                        Can$0.24                   Can$0.45
           $0
                      Web self-service                IVR       Email            Live phone

                                                                Cost estimates
BiT Group White Paper: “Web Self-Service Lowers
Call Center Costs and Improves Customer Service”                   Low           Average      High
http://www.flickr.com/photos/pagedooley/2811157950/
If you don’t know the past
                                                                       you can’t know the future.


                                                                      If you don’t know the future,
                                                                          you can’t budget for it.




Photo by Alan Cleaver from his Flicker Freestock set. Thanks, Alan!
     http://www.flickr.com/photos/alancleaver/2638883650/
“A plan so crazy, it just might work.”
http://www.flickr.com/photos/genewolf/147722350
http://www.flickr.com/photos/billselak/366692332/
Everything starts with a baseline.
Everything starts with a baseline.



 Know what’s
   worst.
Everything starts with a baseline.



 Know what’s          Prove you
   worst.            made it better.
The cycle of optimization
         Metrics & strategy
The cycle of optimization
         Metrics & strategy


                              Collection
The cycle of optimization
         Metrics & strategy


                              Collection




                              Reporting
The cycle of optimization
         Metrics & strategy


                               Collection




                               Reporting


          Institutionalizing
             the results
The cycle of optimization
               Metrics & strategy


                                     Collection



Link to KPI/
                                     Reporting
    ROI

                Institutionalizing
                   the results
The cycle of optimization
               Metrics & strategy

Optimization
                                     Collection
 & change


Link to KPI/
                                     Reporting
    ROI

                Institutionalizing
                   the results
The cycle of optimization
               Metrics & strategy

Optimization
                                     Collection
 & change


Link to KPI/
                                     Reporting
    ROI

                Institutionalizing
                   the results
http://www.flickr.com/photos/elsie/8229790/
Understanding your goals.




              http://www.flickr.com/photos/itsgreg/446061432/
Organic                                  Ad
                       Campaigns
     search                                 network       $

               1           1            1
                                                      Advertiser site

                         Visitor        2                  O er        3       $


                           8                             Upselling 4




                                                                                   Abandonment
                         Reach
                                                  5    Purchase step           $

                         Mailing,
                         alerts,                       Purchase step           $
               9       promotions
         $
                                                      Conversion $

Disengagement                       7
                                                        Enrolment          6


Impact on site
 $      Positive   $     Negative
Bad
                                                                                   $
                                                                  4        content
                     Social              Search
 Invitation
                  network link           results
                                                                  4           Good
                                                                             content
                        1                                                                 $
              1                      1
                                                               Collaboration site
                                                   2
                      Visitor                      Content creation          Moderation

 $
                                                                       3 Spam & trolls

                                 $
                                                       Engagement 5

      Viral
                                6                      Social graph
     spread

                                                                       7

                                                             Disengagement $
Impact on site
$      Positive   $   Negative
Enterprise subscriber $

                                         1

                              End user (employee) $
                                                            Refund $
                                         2

Renewal, upsell,                                                SLA
   reference                        SaaS site                violation
                                   Performance
                                  Good       Bad        3
                                                             Helpdesk         Support
                                                                          5           $
                                     Usability               escalation        costs
       7
                                                        4
                                  Good       Bad


                                   Productivity
                                  Good       Bad


                                                 6

                                         Churn $
Impact on site
 $    Positive     $   Negative
$



                                     Media site
     Enrolment                         Targeted
                                 2   embedded ad       5
                                                               $
           6                                       1
                                                                 Ad
                      Visitor
                                                               network
           4
                                 3                         5
                                      Advertiser   $
Departure $                              site


Impact on site
 $     Positive   $   Negative
Why measure
                          Tactical, to find and fix
                          Strategic, to plan/trend




Part two
The elements of web latency
Slow sites suck
Slow sites suck

Lower conversion rates
Slow sites suck

Lower conversion rates
Less likely to attract a loyal following
Slow sites suck

Lower conversion rates
Less likely to attract a loyal following
Liable for damages
Slow sites suck

Lower conversion rates
Less likely to attract a loyal following
Liable for damages
Liable for refunds or service credits
Slow sites suck

Lower conversion rates
Less likely to attract a loyal following
Liable for damages
Liable for refunds or service credits
Customers find other channels that cost more
Why the web is slow
A crash course in performance & availability.
Load      Web      App
         Internet   balancer   server   server   DB
Client
                                  www.example.com
Your website
                      Load      Web      App
         Internet   balancer   server   server   DB
Client
                                  www.example.com
DNS




                          Load      Web      App
            Internet    balancer   server   server   DB
Client

DNS “www.example.com”
DNS     DNS
                lookup



                           Load      Web      App
            Internet     balancer   server   server   DB
Client

DNS “www.example.com”
DNS     DNS
                lookup



                           Load      Web      App
            Internet     balancer   server   server   DB
Client

DNS “www.example.com”
IP                   IP

                      Load      Web      App
         Internet   balancer   server   server   DB
Client
IP                    IP

                       Load      Web      App
         Internet    balancer   server   server   DB
Client

          Internet routing
IP             R             IP
         R
                              Load      Web      App
             Internet   R   balancer   server   server   DB
Client   R
                  R

              Internet routing
IP             R             IP
         R
                              Load      Web      App
             Internet   R   balancer   server   server   DB
Client   R
                  R

                      TCP session
IP             R             IP
         R
                              Load      Web      App
             Internet   R   balancer   server   server   DB
Client   R
                  R

                      TCP session
Letter writing
                 Postal service
You        Them
(sender)   (receiver)
This
     is
      a
    sentence




  You            Them
(sender)       (receiver)
This
     is
      a
    sentence




  You            Them
(sender)       (receiver)
You        Them
(sender)   (receiver)
You        Them
(sender)   (receiver)
sentence
           This       is
                     a




  You                 Them
(sender)            (receiver)
You        Them
(sender)   (receiver)
This
     is
      a
    sentence




  You            Them
(sender)       (receiver)
This
                is
               a
           sentence


                3
                2
                1
                4
  You           Them
(sender)      (receiver)
You        Them
(sender)   (receiver)
This
     is
      a
    sentence




  You            Them
(sender)       (receiver)
This
                 is


    sentence


                 2
                 1
  You            Them
(sender)       (receiver)
This
                is


           sentence


                2
                1
                4
  You           Them
(sender)      (receiver)
This
           WTF?        is


                  sentence


                       2
                       1
                       4
  You                  Them
(sender)             (receiver)
sentence   a       This


     4         3      1
sentence   a                         This


     4         3                         1
                   “Can you send #2 again?”
sentence      a                         This


     4            3                           1
                      “Can you send #2 again?”




                                         is

           “Sure. Here you go.”
                                              2
How computers “connect”
IP                   IP

                      Load      Web      App
         Internet   balancer   server   server   DB
Client
The HTTP “stack”



  IP                     IP

                        Load      Web      App
           Internet   balancer   server   server   DB
Client
The HTTP “stack”


 TCP                    TCP

  IP                     IP

                        Load      Web      App
           Internet   balancer   server   server   DB
Client
The HTTP “stack”

 SSL                    SSL

 TCP                    TCP

  IP                     IP

                        Load      Web      App
           Internet   balancer   server   server   DB
Client
The HTTP “stack”
 HTTP                  HTTP

 SSL                    SSL

 TCP                    TCP

  IP                     IP

                        Load      Web      App
           Internet   balancer   server   server   DB
Client
Getting a page by hand
Getting a page by hand
Trying 67.205.65.12...
Connected to bitcurrent.com.
Escape character is '^]'.
Getting a page by hand
Trying 67.205.65.12...
Connected to bitcurrent.com.
Escape character is '^]'.
GET /
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/
xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head profile="http://gmpg.org/xfn/11">
<script type="text/javascript" src="http://
www.bitcurrent.com/wp-content/themes/
grid_focus_public/js/perftracker.js"></script>
<script>
</body>
</html>
Connection closed by foreign host.
Static
                                     content

 HTTP                      HTTP       HTTP

 SSL                        SSL

 TCP                        TCP

  IP                         IP

                            Load      Web       App
             Internet     balancer   server    server       DB
Client
                                                        image.gif
GET www.example.com/image.gif
Static
                                     content

 HTTP                      HTTP       HTTP

 SSL                        SSL

 TCP                        TCP

  IP                         IP

                            Load      Web       App
             Internet     balancer   server    server       DB
Client
                                                        image.gif
GET www.example.com/image.gif
Static
                                     content   Dynamic
                                               content
 HTTP                      HTTP       HTTP

 SSL                        SSL

 TCP                        TCP

  IP                         IP

                            Load      Web       App
             Internet     balancer   server    server    DB
Client
                                                 dynamic.jsp
GET www.example.com/dynamic.jsp
Static
                                     content   Dynamic
                                               content
 HTTP                      HTTP       HTTP

 SSL                        SSL

 TCP                        TCP

  IP                         IP

                            Load      Web       App
             Internet     balancer   server    server    DB
Client
                                                 dynamic.jsp
GET www.example.com/dynamic.jsp
Static
                                     content   Dynamic   Stored
                                               content    data
 HTTP                      HTTP       HTTP

  SSL                       SSL

 TCP                        TCP

  IP                         IP

                            Load      Web       App
             Internet     balancer   server    server    DB
Client
                                                   (Database)
POST www.example.com/data.cgi
Static
                                     content   Dynamic   Stored
                                               content    data
 HTTP                      HTTP       HTTP

  SSL                       SSL

 TCP                        TCP

  IP                         IP

                            Load      Web       App
             Internet     balancer   server    server    DB
Client
                                                   (Database)
POST www.example.com/data.cgi
Browser   Data center

            Server
Browser   Data center

            Server
Browser                                               Data center

                                                           Server
TCP SYN (“let’s talk”)
                               TCP SYN ACK (“Agreed: let’s talk”)
TCP ACK (“OK, we’re talking)
Browser                                                      Data center

                                                                  Server
TCP SYN (“let’s talk”)
                                      TCP SYN ACK (“Agreed: let’s talk”)
TCP ACK (“OK, we’re talking)

SSL (“Someone might be listening!”)
                                           SSL (“Here’s a decoder ring”)
Browser                                                                 Data center

                                                                              Server
TCP SYN (“let’s talk”)
                                                TCP SYN ACK (“Agreed: let’s talk”)
TCP ACK (“OK, we’re talking)

SSL (“Someone might be listening!”)
                                                     SSL (“Here’s a decoder ring”)
HTTP GET / (“Can I have your home page?”)
                                                              HTTP 200 OK (“Sure!”)
                                                                                         (thinks
                                                          [index.html] (“Here it is!”)    a bit)
           (Renders furiously)        Bump, bump.    [img js css] (“Have this too!”)
Browser                                                                     Data center

                                                                                  Server
TCP SYN (“let’s talk”)
                                                    TCP SYN ACK (“Agreed: let’s talk”)
TCP ACK (“OK, we’re talking)

SSL (“Someone might be listening!”)
                                                         SSL (“Here’s a decoder ring”)
HTTP GET / (“Can I have your home page?”)
                                                                  HTTP 200 OK (“Sure!”)
                                                                                             (thinks
                                                              [index.html] (“Here it is!”)    a bit)
           (Renders furiously)          Bump, bump.      [img js css] (“Have this too!”)
TCP FIN (“Thanks! I’m done now.”)
                                      TCP FIN ACK (“You’re welcome. Have a nice day.”)
“Page load time” isn’t simple
 Documents versus event models
 AJAX
 Mobility
 CDNs
 Third-party content
 Embedded objects and plug-ins
Connections to load
Connection   0 - www.bitcurrent.com (67.205.65.12)
Connection   1 - www.bitcurrent.com (67.205.65.12)
Connection   2 - 4qinvite.4q.iperceptions.com (64.18.71.70)
Connection   3 - static.slideshare.net (66.114.49.24)
Connection   4 - static.slideshare.net (66.114.49.24)
Connection   5 - www.feedburner.com (66.150.96.123)
Connection   6 - static.getclicky.com (204.13.8.18)
Connection   7 - cetrk.com (208.67.183.100)
Connection   8 - in.getclicky.com (204.13.8.18)
Connection   9 - crazyegg.com (208.67.180.236)
Connection   10 - www.google-analytics.com (72.14.223.147)
Connection   11 - www.apture.com (67.192.46.19)
Connection   12 - static.apture.com (67.192.46.25)
Connection   13 - s.clicktale.net (66.114.49.24)
Connection   14 - www.clicktale.net (75.125.82.70)
Analytics site

                        Server


                          Data center
Browser
                               Server



          Mashup                 Server
           site
Analytics site

                        Server


                          Data center
Browser
                               Server



Snore.
          Mashup                 Server
           site
What ultimately matters:
When can the user start using the application as
its designers intended?
Part of the problem
 You control           You’re blamed for
 Server latency        Page rendering
 Network latency for   Total network latency
 known content and
                       User environment
 network parameters
Part of the problem
 You control           You’re blamed for
 Server latency        Page rendering
 Network latency for   Total network latency
 known content and
                       User environment
 network parameters


      You need
 diagnostic metrics
  so you can fix it.
Part of the problem
 You control           You’re blamed for
 Server latency        Page rendering
 Network latency for   Total network latency
 known content and
                       User environment
 network parameters

                          You need escalation
      You need
                       metrics so you can prove
 diagnostic metrics
                       it and make it someone
  so you can fix it.
                            else’s problem.
Why measure
                     Tactical, to find and fix
                     Strategic, to plan/trend

                   What to measure:
                     How long until a user can use
                     the app as you intended?

Part three
Where to measure
Three tiers of data
 WAN accessibility: One test from many locations
   Can everybody get here?
 App functionality: Several tests of key processes
   Is my business model working correctly?
 Tiered tests: Frequent metrics of each tier
   Is network, service, CPU, data I/O to blame?
WAN accessibility
                               Place A


                               Task B
Client


                               Goal C

 ...
                      Load      Web       App
         Internet   balancer   server    server   DB
Client
Analytics can tell you a lot.
App functionality
                               Page A


                               Page B
Client


                               Event C



                      Load      Web       App
         Internet   balancer   server    server   DB
Client
http://www.flickr.com/photos/tinfoilraccoon/197640807/
Places and Tasks.
Landing page:
View one story
Landing page:
View one story
                 Task: Log in
                  Enter credentials

                       Verify

                     Recovery
Landing page:
View one story
                  Task: Log in
                   Enter credentials

                         Verify

                       Recovery




                       Task:
                 Forward a story
                    Enter recipients

                    Enter message

                         Send
Landing page:
    Task:          View one story
Create account
                                     Task: Log in
    Pick name

   Check if free                      Enter credentials

   Set Password                             Verify

    CAPTCHA                               Recovery

    Send mail

    Get confirm


                                          Task:
                                    Forward a story
                                       Enter recipients

                                       Enter message

                                            Send
Landing page:
    Task:          View one story
Create account
                                     Task: Log in
    Pick name

   Check if free                      Enter credentials

   Set Password                             Verify

    CAPTCHA                               Recovery

    Send mail

    Get confirm


                                          Task:
                                    Forward a story
 Task: Submit                          Enter recipients

  a new story                          Enter message

                                            Send
     Enter URL

      Describe

    Deduplicate

       Post it
Landing page:
    Task:            View one story
Create account
                                           Task: Log in
    Pick name      Place: View stories
   Check if free                            Enter credentials
                     Vote up    Next 25
   Set Password                                   Verify
                    Vote down   Last 25
    CAPTCHA                                     Recovery

    Send mail

    Get confirm


                                                Task:
                                          Forward a story
 Task: Submit                                Enter recipients

  a new story                                Enter message

                                                  Send
     Enter URL

      Describe

    Deduplicate

       Post it
Landing page:
    Task:            View one story
Create account
                                           Task: Log in
    Pick name      Place: View stories
   Check if free                            Enter credentials
                     Vote up    Next 25
   Set Password                                   Verify
                    Vote down   Last 25
    CAPTCHA                                     Recovery

    Send mail
                      Place: Read
    Get confirm
                   poster comments
                     Vote up    Next 25
                                                Task:
                    Vote down   Last 25
                                          Forward a story
 Task: Submit                                Enter recipients

  a new story                                Enter message

                                                  Send
     Enter URL

      Describe

    Deduplicate

       Post it
Landing page:
    Task:            View one story
Create account
                                             Task: Log in
    Pick name      Place: View stories
   Check if free                              Enter credentials
                     Vote up     Next 25
   Set Password                                     Verify
                    Vote down    Last 25
    CAPTCHA                                       Recovery

    Send mail
                      Place: Read
    Get confirm
                   poster comments
                     Vote up     Next 25
                                                  Task:
                    Vote down    Last 25
                                            Forward a story
 Task: Submit                                  Enter recipients

  a new story           Place: My              Enter message

     Enter URL            account                   Send

      Describe       Change        My
                     address    comments
    Deduplicate
                    Change PW   See karma
       Post it
Landing page:
Create acct.     View one story
                                      Task: Log in
               Place: View stories




                  Place: Read
               poster comments
                                         Task:
                                     Forward a story
Task: Submit
 a new story       Place: My
                    account
Landing page:
  Create acct.
Create acct.        View one story
   Form uptime    Place: View stories
                                         Task: Log in

      # started
      Bad form
                     Place: Read
    # CAPTCHA     poster comments

    Mail uptime                             Task:
                                        Forward a story
  Mail bounced
Task: Submit
 a new story          Place: My
Confirm & return        account

     Return 3x
Landing page:
Create acct.          View one story
                                              Task: Log in
                    Place: View stories

                    Place: View stories
               Stories/visit
                     Place: Read
                                          # up/down
                     poster comments
               Time/story                 Top stories
                                                Task:
                                             Forward a story
Task: Submit   Refresh time               Views/page
 a new story            Place: My
                         account
Landing page:
Create acct.     View one story
                                      Task: Log in
               Place: View stories




                  Place: Read
               poster comments
                                         Task:
                                     Forward a story
Task: Submit
 a new story       Place: My
                    account
Places
Efficiency matters
  How quickly, how many,
  productivity
  Learning curve OK
Leave when they’re bored
Collect “aha” feedback
A/B test content for
pages/session, exits
Tasks
Effectiveness matters
  Completion, abandonment
  Intuitiveness rules
Leave when they change their
mind or it breaks
Collect “motivation” feedback
A/B test layouts for conversion
2 sides of the same coin

                  End user
 Web analytics
                  monitoring

  What did       Could they
 visitors do?      do it?
For e-commerce sites




Can people buy things?
For media sites




 Are ads loading quickly and successfully clicked
 through?
 Is content loading fast enough for visitors?
For collaboration sites




 Can visitors contribute (posting content, voting?)
 Is bad content being mitigated (trolling, spam)?
For SaaS sites




Are your end users productive?
Are they making fewer mistakes?
Is the site working during customers’ business hours?
Tiered tests
                               Place A


                               Task B
Client


                               Goal C



                      Load      Web       App
         Internet   balancer   server    server   DB
Client
Testing the tiers

                           Load      Web      App
         Internet        balancer   server   server        DB
Client




                         Request      Do some          Search a
           Request a
                        uncached       heavy          dataset for
           big object
                          object     computing          a string

                                      (Or watch        (Or track
                                        CPU)          query time)
,)-$(&./01+2(3/04(#$+#+(




                 &)$
                 %,$
                 %+$
                 %*$
!"#$%&'()%(*+(




                  %&$
                  %)$
                   ,$                                                                       '""#$($
                   +$                                                                     '""#$&$
                    *$
                                                                                      '""#$%$
                    &$
                    )$                                                             !""#$&$

                         -./01$2341$                                            !""#$%$
                                       53"67$2341$
                                                      8!9$2341$
                                                                     ':$2341$
Why measure
                     Tactical, to find and fix
                     Strategic, to plan/trend

                   What to measure:
                     How long until a user can use
                     the app as you intended?

Part four          Where to measure:
How to measure       WAN, from everywhere
                     Core app functionality
performance data     Tiers of components
Synthetic testing.
Load      Web      App
         Internet   balancer   server   server   DB
Client
Management tool




                      Load      Web        App
         Internet   balancer   server     server   DB
Client
Load      Web      App
         Internet   balancer   server   server   DB
Client
Load      Web      App
         Internet   balancer   server   server   DB
Client
Test   Testing
config    node

                  Data center
        Testing
         node      Website


        Testing
         node
Test   Testing
config    node

                  Data center
        Testing
         node      Website


        Testing
         node
Test   Testing
config    node

                  Data center
        Testing
         node      Website


        Testing
         node
Test     Testing
  config      node

                      Data center
            Testing
             node      Website
Reporting
 service
            Testing
             node
Three things to watch for


 Cached vs. uncached
 Scripts vs. puppetry
 Simultaneous vs. sequential
0
                                     10
               Load time (seconds)




Cached
Uncached
10
Load time (seconds)




                      3.157s



         0
                      Cached   Uncached
13.349s



10
Load time (seconds)




                      3.157s



         0
                      Cached   Uncached
Testing script



Script interpreter
Testing script
  Site: test.com
Page: index.html

             Script interpreter
Testing script
         Site: test.com
       Page: index.html

                    Script interpreter

            HTTP GET
www.test.com/index.html
Testing script
  Site: test.com
Page: index.html

             Script interpreter

                            200 OK
                            index.html
                            image.gif
                            stylesheet.css
                            etc...
Testing script
  Site: test.com            Test complete
Page: index.html

             Script interpreter
Browser
  controller



Actual browser
Browser
                        controller
        DOM actions
(“click on button 4”)

                   Actual browser
Browser
                             controller
             DOM actions
     (“click on button 4”)

                        Actual browser

            HTTP GET
www.test.com/index.html
Browser
                        controller
        DOM actions
(“click on button 4”)

                   Actual browser

                                     200 OK
                                     index.html
                                     image.gif
                                     stylesheet.css
                                     etc...
Browser
                        controller
        DOM actions                  DOM contents
(“click on button 4”)                (“DIV contains ‘error’”)

                   Actual browser
Simultaneous

5 tests at 15:00
Simultaneous        Sequential
                    5 tests from
5 tests at 15:00
                   15:00 to 15:05
Synthetic pros & cons
Pros                          Cons
Easy to set up                Brittle
Only way to test without      Detects macro outages, not
actual visitor traffic         user events
Can compare to                Good geographic & network
competitors                   coverage costs money,
Easy baseline establishment   generates load

Detects a problem before      No measurement of traffic
visitors sees it              volume

Consistent data over time     Places load on the site
                              under test
Ultimately,
Synthetic testing shows you if the site’s working.
Real User Monitoring.
Synthetic isn’t enough
Synthetic isn’t enough
Browser    Web
          server
Browser     Load      Web
          balancer   server
Browser               Load      Web
          Network   balancer   server
            tap
Browser               Load      Web
          Network   balancer   server
            tap
Browser               Load      Web
          Network   balancer   server
            tap
Browser               Load      Web
          Network   balancer   server
            tap
Browser               Load      Web
          Network   balancer   server
            tap
Browser               Load      Web
          Network   balancer   server
            tap
Browser               Load      Web
          Network   balancer   server
            tap

                      User A
Browser               Load      Web
          Network   balancer   server
            tap

                      User A


                      User B


                      User C
Browser                  Load      Web
          Network      balancer   server
            tap

                         User A


                         User B


                         User C



             Visit
             history
             P1

             P2

             P3
Browser                  Load      Web
          Network      balancer   server
            tap

                         User A


                         User B


                         User C



             Visit           Aggregate
             history         reports
             P1

             P2

             P3
Browser                  Load      Web
          Network      balancer   server
            tap

                         User A


                         User B


                         User C



             Visit           Aggregate     Alerts
             history         reports

                                               !
             P1

             P2

             P3
TopN, worstN

RUM tools are excellent for more qualitative data
  What’s most broken?
  What’s biggest?
  What’s slowest?
  What’s most inconsistent?
RUM pros & cons
Pros                           Cons
Directly correlated with       May require physical
clickstream, analytics         installation
Watches everything, not just   Can be a privacy risk
the things you know about      Doesn’t work if there’s no
Can be used to reproduce       traffic
problems                       Need to filter out your own
Measures traffic as well as     visits, crawlers, etc.
performance
Ultimately
RUM shows you if the site’s working.
Why measure
                           Tactical, to find and fix
                           Strategic, to plan/trend

                         What to measure:
                           How long until a user can use
                           the app as you intended?

Part five                 Where to measure:
Getting the math right     WAN, from everywhere
                           Core app functionality
                           Tiers of components

                         How to measure it:
                           Synth, to ensure it’s working
                           RUM, to see where it’s broken
http://upload.wikimedia.org/wikipedia/commons/0/0e/Count-von-count.jpg
0   10   20   30   40 50   60   70   80   90
                    Age
Average age = 10




0   10                 20   30   40 50   60   70   80   90
                                  Age
20


            Average age = 10
Count




  0
        0   10                 20   30   40 50   60   70   80   90
                                          Age
20


            Average age = 10
Count




  0
        0   10                 20   30   40 50   60   70   80   90
                                          Age
20


            Average age = 10
Count




  0
        0   10                 20   30   40 50   60   70   80   90
                                          Age
Average varies wildly,
making it hard to
threshold properly or
see a real slow-down.
80th percentile
only spikes once
for a legitimate
slow-down (20%
of users affected)
Setting a useful
threshold on
percentiles gives
less false positives
and more real alerts
200
# of requests




      0
                0 2   4     6     8 10 12 14 16         18   20
                          Page load time (in seconds)
200




                          Average latency = 5s
# of requests




      0
                0 2   4               6     8 10 12 14 16         18   20
                                    Page load time (in seconds)
0
                                        # of requests
                                                                  200




               0 2
               4
                                     Average latency = 5s




  6
Page load time (in seconds)
        8 10 12 14 16
               18




                                  95th percentile latency = 19s
               20
KISS
“It can scarcely be
                            denied that the supreme goal of
                          all theory is to make the irreducible
                         basic elements as simple and as few
                              as possible without having to
                                 surrender the adequate
                                representation of a single
                                  datum of experience.”




http://media.photobucket.com/image/einstein/derekabril/einstein_010.png
“As simple as possible,
   but no simpler.”




  (FYI, this is irony.)
http://www.flickr.com/photos/evilerin/3540381299/
http://www.flickr.com/photos/golf_pictures/2538894627/
Login

1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Checkout

1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Invite

1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login      Average 4s


1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Checkout   Average 6s


1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Invite     Average 9s


1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login      Average 4s
                                                       95% 8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Checkout   Average 6s
                                                       95% 10s

1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Invite     Average 9s
                                                       95% 12s

1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login      Average 4s
                                                       95% 8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s                Mode 2s


                                         Checkout   Average 6s
                                                       95% 10s
                                                      Mode 5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Invite     Average 9s
                                                       95% 12s
                                                      Mode 1s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login        Average 4s
                                                         95% 8s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s                  Mode 2s


                                         Checkout     Average 6s
                                                         95% 10s
                                                        Mode 5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Invite       Average 9s
                                                         95% 12s
                                                        Mode 1s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s




                                         Aggregate?   Average 6s
                                                         95% 12s
                                                        Mode 5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
740                     260
                                         Login:
                                         <=4s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
740                     260
                                                    Total samples 1000
                                         Login:
                                                  Below threshold 740
                                         <=4s
                                                    Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
                                                  target threshold   74%
740                     260
                                                       Total samples 1000
                                         Login:
                                                     Below threshold 740
                                         <=4s
                                                       Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
                                                     target threshold   74%
   370                     630
                                         Checkout:     Total samples 1000
                                                     Below threshold 370
                                         <=5s
                                                       Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
                                                     target threshold   37%
   610                     390
                                         Invite:       Total samples 1000
                                         <=8s        Below threshold 610
                                                       Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
                                                     target threshold   61%
740                     260
                                                       Total samples 1000
                                         Login:      Below threshold 740
                                         <=4s          Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s                              74%
                                                     target threshold

   370                     630
                                         Checkout:      Total samples 1000
                                                     Below threshold 370
                                         <=5s          Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s                              37%
                                                     target threshold

   610                     390
                                         Invite:       Total samples 1000
                                                     Below threshold 610
                                         <=8s          Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s                              61%
                                                     target threshold

                                         Aggregate?    Total samples 3000
                                                     Below threshold 1720
                                                       Percent below
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
                                                                    57%
                                                     target threshold
740                     260
                                                       Total samples 1000
                                         Login:      Below threshold 740
                                         <=4s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s



   148                     252
                                         Checkout:     Total samples 400
                                                     Below threshold 148
                                         <=5s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s



   366                     366
                                         Invite:       Total samples 600
                                         <=8s        Below threshold 366
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s



                                                       Total samples 2000
                                                     Below threshold 1254
                                                       Percent below
                                                                    63%
                                                     target threshold
740                     260
                                                       Total samples 1000
                                         Login:      Below threshold 740
                                         <=4s               Weight 1
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s



   148                     252
                                         Checkout:     Total samples 400
                                                     Below threshold 148
                                         <=5s
                                                            Weight 5
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s



   366                     366
                                         Invite:       Total samples 600
                                         <=8s        Below threshold 366
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s                      Weight 2
Login:
                                           <=4s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                                           Checkout:
                                           <=5s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                                           Invite:
                                           <=8s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                   Total requests
                    inside target
    Login page          740/1000
Checkout page           148/400
 Invite process         366/600
Login:
                                           <=4s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                                           Checkout:
                                           <=5s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                                           Invite:
                                           <=8s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                   Total requests
                    inside target Weight             Weighted
    Login page          740/1000             1       740/1000
Checkout page           148/400              5       740/2000
 Invite process         366/600              2       732/1200
Login:
                                           <=4s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                                           Checkout:
                                           <=5s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                                           Invite:
                                           <=8s
  1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s


                   Total requests
                    inside target Weight             Weighted
    Login page          740/1000             1       740/1000
Checkout page           148/400              5       740/2000
 Invite process         366/600              2       732/1200
    Total score                                      2212/4200   53%
(Snore?)
)"                                                                                                                        &!!!"




                                                                                                                                               !"#$%&'()*+(,&-*.*/0".1&
!*2,&'.,%0)3.1&


             ("
                                                                                                                                       %#!!"


             #"
                                                                                                                                       %!!!"

             '"

                                                                                                                                       $#!!"

             &"

                                                                                                                                       $!!!"
             %"


                                                                                                                                       #!!"
             $"



             !"                                                                                                                        !"
                  $" %" &" '" #" (" )" *" +" $!"$$"$%"$&"$'"$#"$("$)"$*"$+"%!"%$"%%"%&"%'"%#"%("%)"%*"%+"&!"&$"&%"&&"&'"&#"&("&)"&*"

                                                                       4#5&

                                        ,-./0"               12-34-5.602"                   14789:12-34-5.602;"
)"                                                                                                                        &!!!"




                                                                                                                                               !"#$%&'()*+(,&-*.*/0".1&
!*2,&'.,%0)3.1&



             ("
                                                                                                                                       %#!!"


             #"




                                                                                                                                                                          71% correlation
                                                                                                                                       %!!!"

             '"

                                                                                                                                       $#!!"




                                                                                                                                                                          between traffic
             &"

                                                                                                                                       $!!!"
             %"



             $"



             !"
                                                                                                                                       #!!"



                                                                                                                                       !"
                                                                                                                                                                           and latency.
                  $" %" &" '" #" (" )" *" +" $!"$$"$%"$&"$'"$#"$("$)"$*"$+"%!"%$"%%"%&"%'"%#"%("%)"%*"%+"&!"&$"&%"&&"&'"&#"&("&)"&*"

                                                                       4#5&

                                        ,-./0"               12-34-5.602"                   14789:12-34-5.602;"




                                     If you have traffic predictions, and latency
                                     is correlated with performance, you may
                                     be able to estimate performance in the
                                     future from the business plan.*


                                                                                                                                                                                *It’s seldom this simple.
Baselines


Establish an agreed-upon set of metrics, and always
compare to these baselines.
What does “normal” look like?
Weekly variance? Seasonality?
Why measure
                      Tactical, to find and fix
                      Strategic, to plan/trend

                    What to measure:
                      How long until a user can use
                      the app as you intended?

Part six            Where to measure:
Targeting metrics     WAN, from everywhere
                      Core app functionality
to your audience      Tiers of components

                    How to measure it:
                      Synth, to ensure it’s working
                      RUM, to see where it’s broken

                    Get the math right
Your goal is
to be clearly
understood.
How technical
                  are they?




Your goal is
to be clearly
understood.
How technical
                  are they?




Your goal is
                How will they
to be clearly     use it?

understood.
How technical
                  are they?


                                  To fix
Your goal is                    something
                How will they
to be clearly     use it?

understood.
How technical
                  are they?


                                  To fix
Your goal is                    something
                How will they   To escalate
to be clearly     use it?        to others

understood.
How technical
                  are they?


                                  To fix
Your goal is                    something
                How will they   To escalate
to be clearly     use it?        to others

understood.                     To plan the
                                  future
How technical
                  are they?


                                  To fix
Your goal is                    something
                How will they   To escalate
to be clearly     use it?        to others

understood.                     To plan the
                                  future


                                Translate to
                                their jargon
How technical
                  are they?


                                  To fix
Your goal is                    something
                How will they   To escalate
to be clearly     use it?        to others

understood.                     To plan the
                                  future


                What words do   Translate to
                  they use?     their jargon
By timeframe
Type of
metric    Timeframe   Delivery   Detail
By timeframe
 Type of
 metric      Timeframe   Delivery   Detail

Break/fix
monitoring
By timeframe
 Type of
 metric      Timeframe   Delivery   Detail

Break/fix
monitoring

  Daily
 reports
By timeframe
 Type of
 metric      Timeframe   Delivery   Detail

Break/fix
monitoring

  Daily
 reports

Quarterly
planning
By timeframe
 Type of
 metric      Timeframe    Delivery      Detail

Break/fix                 Push alerts    Simple
              Urgent
monitoring                to PDA       messages

  Daily
 reports

Quarterly
planning
By timeframe
 Type of
 metric      Timeframe    Delivery      Detail

Break/fix                 Push alerts    Simple
              Urgent
monitoring                to PDA       messages

  Daily                                Historical
             Automated    Mail PDF
 reports                                context

Quarterly
planning
By timeframe
 Type of
 metric      Timeframe    Delivery       Detail

Break/fix                 Push alerts    Simple
              Urgent
monitoring                to PDA       messages

  Daily                                Historical
             Automated    Mail PDF
 reports                                context

Quarterly                              Part of big
             Prepared    Slide deck
planning                                picture
By medium
Where will this wind up?
  Dashboard
  NOC screen
  Log file
  Someone’s
  spreadsheet
  Inbox


                      http://www.flickr.com/photos/warrenski/4190341621/
Why measure
                    Tactical, to find and fix
                    Strategic, to plan/trend

                  What to measure:
                    How long until a user can use
                    the app as you intended?

Part seven        Where to measure:
Marching orders     WAN, from everywhere
                    Core app functionality
                    Tiers of components

                  How to measure it:
                    Synth, to ensure it’s working
                    RUM, to see where it’s broken

                  Get the math right
So
what
should
you      Some homework.
do?
First

 Meet your analytics team
 Find out
   What are the key goals they’re monitoring?
   Where are visitors coming from?
   What are the most common entrance and exit
   pages?
Second


Pick the three processes, pages, or functions that
matter most to you
  Landing pages, or part of a conversion funnel
Third

 Set up monitoring of:
   Your site from many places (synthetic testing)
   Your top 3 core business processes (synthetic or
   RUM)
   Your important infrastructure tiers (from agents +
   synthetic, or RUM)
Fourth

Wait a week or two
  To establish a baseline
  To detect seasonal variance
  To show others and get buy-in
Fifth
 While you’re waiting, understand the elements latency
 and how they affect your performance
   DNS
   SSL
   Network latency
   Host (server) latency
   Client page load time
Set a target threshold


 Now that you have an idea of what “normal” is, set a
 threshold
   ... but not just any threshold.
The login page                     Function

will have a total latency           Metric

of under 4 seconds                  Target

with a cached browser copy      User situation

from any US branch office        Testing point

95% of the time                   Percentile

weekdays, 8AM ET to 6M PST      Time window

by synth test at 5m intervals   Collection type
Apdex
score
      =   (          ) (
              Satisfied
              requests
                       +
                           Tolerating
                            requests
                                     /2   )
                   All requests
How Apdex works
 Frustrated: over 8 seconds




  Tolerating: 2-8 seconds




   Satisfied: 0-2 seconds
How Apdex works
                        Frustrated: 5 hits
  Total requests: 100



                        Tolerating: 30 hits

                                              (65) + (30/2)
                                                              = 0.80
                                                  100
                        Satisfied: 65 hits
Train your audience

 Visit key stakeholders and walk them through the
 report
 Get them used to the information
   In the same format
   At the same time
   From the same place
Put monitoring into your
release cycle

 Talk to the development team
   Adding instrumentation
   Identifying new code functions that need testing
   Verifying whether optimization worked
Part eight
Some tools to check out
Paid
Synthetic
 Keynote Systems
 Gomez
 Webmetrics
 Alertsite
 Dotcom Monitor
 Pingdom
 ...and many others
RUM
Client-side
  AJAX (Gomez, Coradiant
  TrueSight Edge)                          Full
  Agent-based (Aternity)             disclosure: We
                                     both worked at
Inline (sniffer/tap)                   Coradiant.
  Coradiant, Tealeaf, Beatbox(HP),
  Atomic Labs, Compuware
  Apdex
Server-side (logfile, agent)
Analytics

 Omniture
 Webtrends
 Coremetrics
 Woopra
 etc. (lots of specialization)
Open Source
Firebug




getfirebug.com
Firebug




                Also: Webkit inspector,
getfirebug.com    Google Page Speed
Google Analytics




analytics.google.com
webpagetest.org
Monitor.us

   (Free ain’t
   pretty, and
   pretty ain’t
   free, but it
     works.)



mon.itor.us
AJAX measurement libraries
Collecting from visitors:
  Jiffy (http://code.google.com/p/jiffy-web/)
    AJAX client sends measurements to Apache collector
Other resources
  ZK-Gazer (http://code.google.com/p/zk-gazer/)
  http://www.ajaxperformance.com/ (Ryan Breen)
  http://www.opensourcetesting.org/performance.php
YSlow




        http://justtalkaboutweb.com/wp-content/uploads/2008/06/yslow.gif
                    http://events.stanford.edu/events/196/19695/souders.jpg
Sites

Dashboard        Juice analytics’     Dashboard
   spy                blog          Insight’s gallery




  Simple Complexity
Part nine
Planning for the future
AJAX
AJAX




        As for your male and female
        slaves whom you may have:
       you may buy male and female
       slaves from among the nations
            that are around you.
                            - Leviticus 25:44
http://www.flickr.com/photos/farhannasir/4577508824/



                                      Mobility
http://www.flickr.com/photos/andrewparnell/2738598951/
GET index.html HTTP/1.1
Host: www.stockprice.com
Cookie: sessionID=KDF74INED6
Accept: */*

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/
xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" />
<head profile="http://gmpg.org/xfn/11">
<title>Stock price for Apple</title>
<script type="text/javascript" src="http://
www.bitcurrent.com/wp-content/themes/
grid_focus_public/js/perftracker.js"></script>
<script>
<body id=gsr topmargin=3 marginheight=3>
AAPL:243.20
</body>
</html>
GET index.html
Host: www.stockprice.com
Cookie: sessionID=KDF74INED6

AAPL:243.20
Web of                          Web of
documents                           events
(circa 1999)                    (circa 2008)

               http://www.flickr.com/photos/dnorman/2781572080/
                http://www.flickr.com/photos/adamkr/4650637393/
Recap
What you need to go and do now.
Metrics must be

Relevant: related to a core business assumption
Actionable: the basis for a decision or improvement
Reproducible: documented and generated cleanly
Understandable: easy for stakeholders to grok
Accurate: providing the correct view of what happened
Visit your analytics team & read your business model
Pick three core business functions to watch
Start monitoring
  One page from many places
  Key business functions
  Infrastructure tiers
Take a deep breath and establish a baseline
  Analyze elements of latency while you wait
Set target thresholds using a meaningful SLA
  Calculate a consistent score & train your audience
Make it part of the release cycle
Metric   Source   Target
Metric   Source   Target
Onload
 time
Metric    Source     Target
Onload   From many
 time      places
Metric    Source       Target
Onload   From many     To the top
 time      places    landing page
Metric    Source       Target
Onload   From many     To the top   Uncached
 time      places    landing page   Cached
Metric    Source       Target
Onload   From many     To the top   Uncached
 time      places    landing page   Cached

Server
 time
Metric    Source          Target
Onload   From many        To the top   Uncached
 time      places       landing page   Cached

Server    From one
 time    place, often
Metric    Source            Target
Onload   From many          To the top     Uncached
 time      places         landing page     Cached

Server    From one        To your core
 time    place, often   business process
Metric    Source            Target
Onload   From many          To the top     Uncached
 time      places         landing page     Cached

Server    From one        To your core
 time    place, often   business process

Server
 time
Metric    Source            Target
Onload   From many          To the top     Uncached
 time      places         landing page     Cached

Server    From one        To your core
 time    place, often   business process

Server    From one
 time    place, often
Metric    Source            Target
Onload   From many          To the top     Uncached
 time      places         landing page     Cached

Server    From one        To your core
 time    place, often   business process

Server    From one        To each tier
 time    place, often   (web, I/O, CPU)
Metric    Source            Target
Onload   From many          To the top     Uncached
 time      places         landing page     Cached

Server    From one        To your core
 time    place, often   business process

Server    From one        To each tier
 time    place, often   (web, I/O, CPU)


 List    Criteria             Segmentation
Metric    Source            Target
Onload   From many          To the top     Uncached
 time      places         landing page     Cached

Server    From one        To your core
 time    place, often   business process

Server    From one        To each tier
 time    place, often   (web, I/O, CPU)


 List    Criteria             Segmentation

TopN
Metric    Source            Target
Onload   From many          To the top     Uncached
 time      places         landing page     Cached

Server    From one        To your core
 time    place, often   business process

Server    From one        To each tier
 time    place, often   (web, I/O, CPU)


 List    Criteria             Segmentation
          Worst
TopN
          pages
Metric    Source             Target
Onload   From many           To the top     Uncached
 time      places          landing page     Cached

Server    From one         To your core
 time    place, often    business process

Server    From one         To each tier
 time    place, often    (web, I/O, CPU)


 List    Criteria              Segmentation
          Worst
TopN
          pages
                    by
Metric    Source             Target
Onload   From many           To the top     Uncached
 time      places          landing page     Cached

Server    From one         To your core
 time    place, often    business process

Server    From one         To each tier
 time    place, often    (web, I/O, CPU)


 List    Criteria              Segmentation
          Worst
TopN
          pages
                    by Error rate
Metric    Source             Target
Onload   From many           To the top       Uncached
 time      places          landing page       Cached

Server    From one         To your core
 time    place, often    business process

Server    From one         To each tier
 time    place, often    (web, I/O, CPU)


 List    Criteria              Segmentation
          Worst                      Server
TopN
          pages
                    by Error rate   latency
Metric    Source             Target
Onload   From many           To the top       Uncached
 time      places          landing page       Cached

Server    From one         To your core
 time    place, often    business process

Server    From one         To each tier
 time    place, often    (web, I/O, CPU)


 List    Criteria              Segmentation
          Worst                      Server    Network
TopN
          pages
                    by Error rate   latency    latency
Got one report?




1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000
  Unique page views




             0
                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000
  Unique page views




                                                                                 >4s or error

                                                                                 2-4s

             0                                                                   <2s

                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000
  Unique page views




                                                                                 >4s or error

                                                                                 2-4s

             0                                                                   <2s

                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000
  Unique page views




                                                                                 >4s or error

                                                                                 2-4s

             0                                                                   <2s

                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000
  Unique page views




                                                                                 >4s or error

                                                                                 2-4s

             0                                                                   <2s

                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000
  Unique page views




                                                                                 >4s or error

                                                                                 2-4s

             0                                                                   <2s

                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000                                                                            $10,000




                                                                                 Revenue (total sales)
  Unique page views




                                                                                                         >4s or error

                                                                                                         2-4s

             0                                                                   $0                      <2s

                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000                                                                            $10,000




                                                                                 Revenue (total sales)
  Unique page views




                                                                                                         Conversions


                                                                                                         >4s or error

                                                                                                         2-4s

             0                                                                   $0                      <2s

                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Got one report?
5,000                                                                            $10,000




                                                                                 Revenue (total sales)
  Unique page views




                                                                                                         Conversions


                                                                                                         >4s or error

                                                                                                         2-4s

             0                                                                   $0                      <2s

                      1   2   3   4   5   6   7   8   9   10 11 12 13 14 15 16
Thanks!
@seanpower
sean@httpd.org
@acroll
alistair@bitcurrent.com

www.watchingwebsites.com




                   (and go buy this.)

More Related Content

What's hot

Big Data for the CMO
Big Data for the CMOBig Data for the CMO
Big Data for the CMO
Bruno Aziza
 

What's hot (20)

Slides from Growthcon 2014 Lean Analytics masterclass
Slides from Growthcon 2014 Lean Analytics masterclassSlides from Growthcon 2014 Lean Analytics masterclass
Slides from Growthcon 2014 Lean Analytics masterclass
 
Lean Analytics for Nikkei BP
Lean Analytics for Nikkei BPLean Analytics for Nikkei BP
Lean Analytics for Nikkei BP
 
7 Myths of Lean and How Analytics Can Help
7 Myths of Lean and How Analytics Can Help7 Myths of Lean and How Analytics Can Help
7 Myths of Lean and How Analytics Can Help
 
Startup metrics toronto March 19
Startup metrics toronto March 19Startup metrics toronto March 19
Startup metrics toronto March 19
 
Lean analytics: Five lessons beyond the basics
Lean analytics: Five lessons beyond the basicsLean analytics: Five lessons beyond the basics
Lean analytics: Five lessons beyond the basics
 
Croll lean analytics workshop (3h) - lean ux nyc april 2014
Croll   lean analytics workshop (3h) - lean ux nyc april 2014Croll   lean analytics workshop (3h) - lean ux nyc april 2014
Croll lean analytics workshop (3h) - lean ux nyc april 2014
 
Melbourne Business School - mba talk october 14 - croll - 40m - lean analytics
Melbourne Business School - mba talk october 14 - croll - 40m - lean analyticsMelbourne Business School - mba talk october 14 - croll - 40m - lean analytics
Melbourne Business School - mba talk october 14 - croll - 40m - lean analytics
 
Lean Analytics: A short summary
Lean Analytics: A short summaryLean Analytics: A short summary
Lean Analytics: A short summary
 
Lean Analytics workshop for Dublin City University, April 2014
Lean Analytics workshop for Dublin City University, April 2014Lean Analytics workshop for Dublin City University, April 2014
Lean Analytics workshop for Dublin City University, April 2014
 
Understanding Lean Analytics (and how analytics helps businesses win)
Understanding Lean Analytics (and how analytics helps businesses win)Understanding Lean Analytics (and how analytics helps businesses win)
Understanding Lean Analytics (and how analytics helps businesses win)
 
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
Lean Analytics - Ben Yoskovitz, Co-author, Lean Analytics & VP Product, Varag...
 
Big Data for the CMO
Big Data for the CMOBig Data for the CMO
Big Data for the CMO
 
Growth Hacking with Cassie Lancellotti-Young
Growth Hacking with Cassie Lancellotti-YoungGrowth Hacking with Cassie Lancellotti-Young
Growth Hacking with Cassie Lancellotti-Young
 
Startup Metrics for Pirates (Startonomics Beijing, June 2009)
Startup Metrics for Pirates (Startonomics Beijing, June 2009)Startup Metrics for Pirates (Startonomics Beijing, June 2009)
Startup Metrics for Pirates (Startonomics Beijing, June 2009)
 
Measuring the ROI of content marketing
Measuring the ROI of content marketingMeasuring the ROI of content marketing
Measuring the ROI of content marketing
 
Big data tokyo (extended version)
Big data tokyo  (extended version)Big data tokyo  (extended version)
Big data tokyo (extended version)
 
Lean startup metrics
Lean startup metricsLean startup metrics
Lean startup metrics
 
Startup Growth: Up and to the Right
Startup Growth: Up and to the RightStartup Growth: Up and to the Right
Startup Growth: Up and to the Right
 
Managing a Virtual Economy
Managing a Virtual EconomyManaging a Virtual Economy
Managing a Virtual Economy
 
Organisation Hacking
Organisation HackingOrganisation Hacking
Organisation Hacking
 

Viewers also liked

How Obama Won Using Digital and Social Media
How Obama Won Using Digital and Social MediaHow Obama Won Using Digital and Social Media
How Obama Won Using Digital and Social Media
James Burnes
 
Cracking the Facebook Coding Interview
Cracking the Facebook Coding InterviewCracking the Facebook Coding Interview
Cracking the Facebook Coding Interview
Gayle McDowell
 

Viewers also liked (20)

Internal Medicine Image Challenge MCQs
Internal Medicine Image Challenge MCQsInternal Medicine Image Challenge MCQs
Internal Medicine Image Challenge MCQs
 
Factors affecting learning PSYCHOLOGICAL, PHYSIOLOGICAL / BIOLOGICAL, ENVIRO...
Factors affecting learning PSYCHOLOGICAL,  PHYSIOLOGICAL / BIOLOGICAL, ENVIRO...Factors affecting learning PSYCHOLOGICAL,  PHYSIOLOGICAL / BIOLOGICAL, ENVIRO...
Factors affecting learning PSYCHOLOGICAL, PHYSIOLOGICAL / BIOLOGICAL, ENVIRO...
 
Tmj anatomy
Tmj anatomyTmj anatomy
Tmj anatomy
 
Basics of c++ Programming Language
Basics of c++ Programming LanguageBasics of c++ Programming Language
Basics of c++ Programming Language
 
Carbohydrate Chemistry
Carbohydrate ChemistryCarbohydrate Chemistry
Carbohydrate Chemistry
 
Micro Expressions
Micro ExpressionsMicro Expressions
Micro Expressions
 
Off the Page Into the Wild: Designing For the Internet of Things
Off the Page Into the Wild: Designing For the Internet of ThingsOff the Page Into the Wild: Designing For the Internet of Things
Off the Page Into the Wild: Designing For the Internet of Things
 
How Obama Won Using Digital and Social Media
How Obama Won Using Digital and Social MediaHow Obama Won Using Digital and Social Media
How Obama Won Using Digital and Social Media
 
How to write a good business letter
How to write a good business letter   How to write a good business letter
How to write a good business letter
 
Nathalie Nahai - The secret psychology of persuasive copy (Conversion Confere...
Nathalie Nahai - The secret psychology of persuasive copy (Conversion Confere...Nathalie Nahai - The secret psychology of persuasive copy (Conversion Confere...
Nathalie Nahai - The secret psychology of persuasive copy (Conversion Confere...
 
Engineering Geology
Engineering GeologyEngineering Geology
Engineering Geology
 
How to do lean planning
How to do lean planningHow to do lean planning
How to do lean planning
 
Basics of radiation and production of x rays
Basics of radiation and production of x raysBasics of radiation and production of x rays
Basics of radiation and production of x rays
 
Fintech and Transformation of the Financial Services Industry
Fintech and Transformation of the Financial Services IndustryFintech and Transformation of the Financial Services Industry
Fintech and Transformation of the Financial Services Industry
 
BUSINESS QUIZ -Round 1
 BUSINESS QUIZ -Round 1 BUSINESS QUIZ -Round 1
BUSINESS QUIZ -Round 1
 
Mango Training for NGOs - key financial concepts and jargon
Mango Training for NGOs - key financial concepts and jargonMango Training for NGOs - key financial concepts and jargon
Mango Training for NGOs - key financial concepts and jargon
 
Cracking the Facebook Coding Interview
Cracking the Facebook Coding InterviewCracking the Facebook Coding Interview
Cracking the Facebook Coding Interview
 
Environmental Impact Assessment (EIA) report on Rampal 1320MW coal-based powe...
Environmental Impact Assessment (EIA) report on Rampal 1320MW coal-based powe...Environmental Impact Assessment (EIA) report on Rampal 1320MW coal-based powe...
Environmental Impact Assessment (EIA) report on Rampal 1320MW coal-based powe...
 
Engaging Learners with Technology
Engaging Learners with TechnologyEngaging Learners with Technology
Engaging Learners with Technology
 
Deep Learning through Examples
Deep Learning through ExamplesDeep Learning through Examples
Deep Learning through Examples
 

Similar to Metrics 101

Monedize.com Business Plan
Monedize.com Business PlanMonedize.com Business Plan
Monedize.com Business Plan
noswald
 
Measuring Meaning In Social Media (Web Visions)
Measuring Meaning In Social Media (Web Visions)Measuring Meaning In Social Media (Web Visions)
Measuring Meaning In Social Media (Web Visions)
Ryan Turner
 

Similar to Metrics 101 (20)

Why care about communities?
Why care about communities?Why care about communities?
Why care about communities?
 
Monedize.com Business Plan
Monedize.com Business PlanMonedize.com Business Plan
Monedize.com Business Plan
 
Hubspot Case Presentation - First Place
Hubspot Case Presentation - First PlaceHubspot Case Presentation - First Place
Hubspot Case Presentation - First Place
 
How To Make An Excellent SMS Call To Action
How To Make An Excellent SMS Call To ActionHow To Make An Excellent SMS Call To Action
How To Make An Excellent SMS Call To Action
 
5 Epiphanies from 20 Years in Search Marketing
5 Epiphanies from 20 Years in Search Marketing5 Epiphanies from 20 Years in Search Marketing
5 Epiphanies from 20 Years in Search Marketing
 
Bitcoin, Transaction Fees and The Cost of Poor Quality
Bitcoin, Transaction Fees and The Cost of Poor QualityBitcoin, Transaction Fees and The Cost of Poor Quality
Bitcoin, Transaction Fees and The Cost of Poor Quality
 
Measuring Meaning In Social Media (Web Visions)
Measuring Meaning In Social Media (Web Visions)Measuring Meaning In Social Media (Web Visions)
Measuring Meaning In Social Media (Web Visions)
 
Mint.pdf
Mint.pdfMint.pdf
Mint.pdf
 
Airbnb: Investment Recommendation
Airbnb: Investment Recommendation Airbnb: Investment Recommendation
Airbnb: Investment Recommendation
 
Gift Of Time Ohio Corporate And Hospital Presentation
Gift Of Time Ohio Corporate And Hospital PresentationGift Of Time Ohio Corporate And Hospital Presentation
Gift Of Time Ohio Corporate And Hospital Presentation
 
Addressing the High Cost of Quality
Addressing the High Cost of QualityAddressing the High Cost of Quality
Addressing the High Cost of Quality
 
Gitex2010 ICT strategies moving to the cloud v11
Gitex2010 ICT strategies moving to the cloud v11Gitex2010 ICT strategies moving to the cloud v11
Gitex2010 ICT strategies moving to the cloud v11
 
McDougall Interactive-presentation - 2011
McDougall Interactive-presentation - 2011McDougall Interactive-presentation - 2011
McDougall Interactive-presentation - 2011
 
Mike Mcderment - Marketing Metrics and the Systems You Need to Measure Them
Mike Mcderment - Marketing Metrics and the Systems You Need to Measure ThemMike Mcderment - Marketing Metrics and the Systems You Need to Measure Them
Mike Mcderment - Marketing Metrics and the Systems You Need to Measure Them
 
Mint: $325K VC investment turned into $170M. Mint's initial pitch deck
Mint: $325K VC investment turned into $170M. Mint's initial pitch deckMint: $325K VC investment turned into $170M. Mint's initial pitch deck
Mint: $325K VC investment turned into $170M. Mint's initial pitch deck
 
Mint pitch deck
Mint pitch deckMint pitch deck
Mint pitch deck
 
dgm | Online Retailer Expo Sydney 2012 | Chris Garner
dgm | Online Retailer Expo Sydney 2012 | Chris Garnerdgm | Online Retailer Expo Sydney 2012 | Chris Garner
dgm | Online Retailer Expo Sydney 2012 | Chris Garner
 
Technology Economics: Economics of Computing - The Internal Combustion Mainframe
Technology Economics: Economics of Computing - The Internal Combustion MainframeTechnology Economics: Economics of Computing - The Internal Combustion Mainframe
Technology Economics: Economics of Computing - The Internal Combustion Mainframe
 
Internet Marketing for Home Performance
Internet Marketing for Home PerformanceInternet Marketing for Home Performance
Internet Marketing for Home Performance
 
Increase AUM with Marketing Automation Part 3: Marketing Analytics to Prove ROI
Increase AUM with Marketing Automation Part 3: Marketing Analytics to Prove ROIIncrease AUM with Marketing Automation Part 3: Marketing Analytics to Prove ROI
Increase AUM with Marketing Automation Part 3: Marketing Analytics to Prove ROI
 

More from Alistair Croll

Book as api hugh mc guire and alistair croll - toc nyc 2013
Book as api   hugh mc guire and alistair croll - toc nyc 2013Book as api   hugh mc guire and alistair croll - toc nyc 2013
Book as api hugh mc guire and alistair croll - toc nyc 2013
Alistair Croll
 
Infopresse montreal feb 6 big data
Infopresse montreal feb 6   big dataInfopresse montreal feb 6   big data
Infopresse montreal feb 6 big data
Alistair Croll
 
Form Fits Function: Choosing IaaS, Pa
Form Fits Function: Choosing IaaS, PaForm Fits Function: Choosing IaaS, Pa
Form Fits Function: Choosing IaaS, Pa
Alistair Croll
 

More from Alistair Croll (20)

Book as api hugh mc guire and alistair croll - toc nyc 2013
Book as api   hugh mc guire and alistair croll - toc nyc 2013Book as api   hugh mc guire and alistair croll - toc nyc 2013
Book as api hugh mc guire and alistair croll - toc nyc 2013
 
Infopresse montreal feb 6 big data
Infopresse montreal feb 6   big dataInfopresse montreal feb 6   big data
Infopresse montreal feb 6 big data
 
CloudOps evening presentation from Savvis
CloudOps evening presentation from SavvisCloudOps evening presentation from Savvis
CloudOps evening presentation from Savvis
 
CloudOps evening presentation from Amazon
CloudOps evening presentation from AmazonCloudOps evening presentation from Amazon
CloudOps evening presentation from Amazon
 
CloudOps evening presentation from IBM
CloudOps evening presentation from IBMCloudOps evening presentation from IBM
CloudOps evening presentation from IBM
 
CloudOps evening presentation from Google
CloudOps evening presentation from GoogleCloudOps evening presentation from Google
CloudOps evening presentation from Google
 
CloudOps evening presentation from Salesforce.com
CloudOps evening presentation from Salesforce.comCloudOps evening presentation from Salesforce.com
CloudOps evening presentation from Salesforce.com
 
The move to turnkey computing
The move to turnkey computingThe move to turnkey computing
The move to turnkey computing
 
Transforming the data center
Transforming the data centerTransforming the data center
Transforming the data center
 
The three themes of Strata
The three themes of StrataThe three themes of Strata
The three themes of Strata
 
Lean analytics for startups - Leweb2010
Lean analytics for startups - Leweb2010Lean analytics for startups - Leweb2010
Lean analytics for startups - Leweb2010
 
Productcamp montreal 2010
Productcamp montreal 2010Productcamp montreal 2010
Productcamp montreal 2010
 
Slides from a lecture at Concordia's Desautels school of management
Slides from a lecture at Concordia's Desautels school of managementSlides from a lecture at Concordia's Desautels school of management
Slides from a lecture at Concordia's Desautels school of management
 
Form Fits Function: Choosing IaaS, Pa
Form Fits Function: Choosing IaaS, PaForm Fits Function: Choosing IaaS, Pa
Form Fits Function: Choosing IaaS, Pa
 
Democratization of IT - october 18 - 20m
Democratization of IT - october 18 - 20mDemocratization of IT - october 18 - 20m
Democratization of IT - october 18 - 20m
 
Public clouds go mainstream - october 19 - 10m
Public clouds go mainstream - october 19 - 10mPublic clouds go mainstream - october 19 - 10m
Public clouds go mainstream - october 19 - 10m
 
Private clouds and enterprise it - october 18 - 30m
Private clouds and enterprise it  - october 18 - 30mPrivate clouds and enterprise it  - october 18 - 30m
Private clouds and enterprise it - october 18 - 30m
 
Delivery strategies: Apps don't deploy themselves
Delivery strategies: Apps don't deploy themselvesDelivery strategies: Apps don't deploy themselves
Delivery strategies: Apps don't deploy themselves
 
Cloud 101 - Workshop from Gov2.0 in DC, May 2010
Cloud 101 - Workshop from Gov2.0 in DC, May 2010Cloud 101 - Workshop from Gov2.0 in DC, May 2010
Cloud 101 - Workshop from Gov2.0 in DC, May 2010
 
Cloud 101
Cloud 101Cloud 101
Cloud 101
 

Recently uploaded

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
Wonjun Hwang
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 

Recently uploaded (20)

ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
Simplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxSimplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptx
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 

Metrics 101

  • 2.
  • 3. What we’ll cover Why collect metrics Understanding web latency How to target your findings Concrete steps to get started
  • 7. Downtime costs eBay offline ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)
  • 8. Downtime costs eBay offline ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999) Financial company down ($100K/h) 53.2% of finance companies lose over $100,000/hour (nextslm.org)
  • 9. Downtime costs Amazon offline ($1M/h) Amazon loses nearly $1M/hour if down (NYT, 2008) eBay offline ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999) Financial company down ($100K/h) 53.2% of finance companies lose over $100,000/hour (nextslm.org)
  • 10. Downtime costs Amazon offline ($1M/h) Amazon loses nearly $1M/hour if down (NYT, 2008) Network downtime ($42K/h) 1 hour of network downtime costs $42,000 (Gartner, 2003) eBay offline ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999) Financial company down ($100K/h) 53.2% of finance companies lose over $100,000/hour (nextslm.org)
  • 11. Downtime costs Amazon offline ($1M/h) Amazon loses nearly $1M/hour if down (NYT, 2008) Network downtime ($42K/h) 1 hour of network downtime costs $42,000 (Gartner, 2003) eBay offline ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999) Financial company down ($100K/h) 53.2% of finance companies lose over $100,000/hour (nextslm.org) Let’s say $50K/h if you’re serious.
  • 12. Availability Downtime/year Loss @$50K/h 90% % 36.5 days Can$43,800,000 95% 18.25 days Can$21,900,000 98% 7.30 days Can$8,760,000 99% 3.65 days Can$4,380,000 99.5% 1.83 days Can$2,196,000 99.8% 17.52 hours Can$876,000 99.9% 8.76 hours Can$438,000 99.95% 4.38 hours Can$219,000 99.99% 52.6 minutes Can$43,833 99.999% 5.26 minutes Can$4,383 99.9999% 31.5 seconds Can$438
  • 13. Availability Downtime/year Loss @$50K/h 90% % 36.5 days Can$43,800,000 95% 18.25 days Can$21,900,000 98% 7.30 days Can$8,760,000 99% 3.65 days Can$4,380,000 99.5% 1.83 days Can$2,196,000 99.8% 17.52 hours Can$876,000 99.9% 8.76 hours Can$438,000 Less than 99.95% 4.38 hours Can$219,000 an hour a 99.99% 52.6 minutes Can$43,833 year 99.999% 5.26 minutes Can$4,383 99.9999% 31.5 seconds Can$438
  • 14. Availability Downtime/year Loss @$50K/h 90% % 36.5 days Can$43,800,000 95% 18.25 days Can$21,900,000 98% 7.30 days Can$8,760,000 99% 3.65 days Can$4,380,000 99.5% 1.83 days Can$2,196,000 99.8% 17.52 hours Can$876,000 99.9% 8.76 hours Can$438,000 Less than 99.95% 4.38 hours Can$219,000 an hour a 99.99% 52.6 minutes Can$43,833 year 99.999% 5.26 minutes Can$4,383 Less than 99.9999% 31.5 seconds Can$438 a minute a year
  • 15. Harris poll conducted by Tealeaf in 2008
  • 16. You really don’t want web users to call you. $15 $12 $9 $6 $3 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High
  • 17. You really don’t want web users to call you. $15 $12 $9 $6 $3 Can$0.24 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High
  • 18. You really don’t want web users to call you. $15 $12 $9 $6 $3 Can$0.24 Can$0.45 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High
  • 19. You really don’t want web users to call you. $15 $12 $9 $6 Can$3.00 $3 Can$0.24 Can$0.45 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High
  • 20. You really don’t want web users to call you. $15 $12 $9 $6 Can$5.50 Can$3.00 $3 Can$0.24 Can$0.45 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High
  • 22. If you don’t know the past you can’t know the future. If you don’t know the future, you can’t budget for it. Photo by Alan Cleaver from his Flicker Freestock set. Thanks, Alan! http://www.flickr.com/photos/alancleaver/2638883650/
  • 23. “A plan so crazy, it just might work.”
  • 26. Everything starts with a baseline.
  • 27. Everything starts with a baseline. Know what’s worst.
  • 28. Everything starts with a baseline. Know what’s Prove you worst. made it better.
  • 29. The cycle of optimization Metrics & strategy
  • 30. The cycle of optimization Metrics & strategy Collection
  • 31. The cycle of optimization Metrics & strategy Collection Reporting
  • 32. The cycle of optimization Metrics & strategy Collection Reporting Institutionalizing the results
  • 33. The cycle of optimization Metrics & strategy Collection Link to KPI/ Reporting ROI Institutionalizing the results
  • 34. The cycle of optimization Metrics & strategy Optimization Collection & change Link to KPI/ Reporting ROI Institutionalizing the results
  • 35. The cycle of optimization Metrics & strategy Optimization Collection & change Link to KPI/ Reporting ROI Institutionalizing the results
  • 37. Understanding your goals. http://www.flickr.com/photos/itsgreg/446061432/
  • 38.
  • 39.
  • 40. Organic Ad Campaigns search network $ 1 1 1 Advertiser site Visitor 2 O er 3 $ 8 Upselling 4 Abandonment Reach 5 Purchase step $ Mailing, alerts, Purchase step $ 9 promotions $ Conversion $ Disengagement 7 Enrolment 6 Impact on site $ Positive $ Negative
  • 41.
  • 42.
  • 43. Bad $ 4 content Social Search Invitation network link results 4 Good content 1 $ 1 1 Collaboration site 2 Visitor Content creation Moderation $ 3 Spam & trolls $ Engagement 5 Viral 6 Social graph spread 7 Disengagement $ Impact on site $ Positive $ Negative
  • 44.
  • 45. Enterprise subscriber $ 1 End user (employee) $ Refund $ 2 Renewal, upsell, SLA reference SaaS site violation Performance Good Bad 3 Helpdesk Support 5 $ Usability escalation costs 7 4 Good Bad Productivity Good Bad 6 Churn $ Impact on site $ Positive $ Negative
  • 46.
  • 47. $ Media site Enrolment Targeted 2 embedded ad 5 $ 6 1 Ad Visitor network 4 3 5 Advertiser $ Departure $ site Impact on site $ Positive $ Negative
  • 48. Why measure Tactical, to find and fix Strategic, to plan/trend Part two The elements of web latency
  • 50. Slow sites suck Lower conversion rates
  • 51. Slow sites suck Lower conversion rates Less likely to attract a loyal following
  • 52. Slow sites suck Lower conversion rates Less likely to attract a loyal following Liable for damages
  • 53. Slow sites suck Lower conversion rates Less likely to attract a loyal following Liable for damages Liable for refunds or service credits
  • 54. Slow sites suck Lower conversion rates Less likely to attract a loyal following Liable for damages Liable for refunds or service credits Customers find other channels that cost more
  • 55. Why the web is slow A crash course in performance & availability.
  • 56. Load Web App Internet balancer server server DB Client www.example.com
  • 57. Your website Load Web App Internet balancer server server DB Client www.example.com
  • 58. DNS Load Web App Internet balancer server server DB Client DNS “www.example.com”
  • 59. DNS DNS lookup Load Web App Internet balancer server server DB Client DNS “www.example.com”
  • 60. DNS DNS lookup Load Web App Internet balancer server server DB Client DNS “www.example.com”
  • 61. IP IP Load Web App Internet balancer server server DB Client
  • 62. IP IP Load Web App Internet balancer server server DB Client Internet routing
  • 63. IP R IP R Load Web App Internet R balancer server server DB Client R R Internet routing
  • 64. IP R IP R Load Web App Internet R balancer server server DB Client R R TCP session
  • 65. IP R IP R Load Web App Internet R balancer server server DB Client R R TCP session
  • 66.
  • 67.
  • 68.
  • 69. Letter writing Postal service
  • 70. You Them (sender) (receiver)
  • 71. This is a sentence You Them (sender) (receiver)
  • 72. This is a sentence You Them (sender) (receiver)
  • 73. You Them (sender) (receiver)
  • 74. You Them (sender) (receiver)
  • 75. sentence This is a You Them (sender) (receiver)
  • 76. You Them (sender) (receiver)
  • 77. This is a sentence You Them (sender) (receiver)
  • 78. This is a sentence 3 2 1 4 You Them (sender) (receiver)
  • 79. You Them (sender) (receiver)
  • 80. This is a sentence You Them (sender) (receiver)
  • 81. This is sentence 2 1 You Them (sender) (receiver)
  • 82. This is sentence 2 1 4 You Them (sender) (receiver)
  • 83. This WTF? is sentence 2 1 4 You Them (sender) (receiver)
  • 84. sentence a This 4 3 1
  • 85. sentence a This 4 3 1 “Can you send #2 again?”
  • 86. sentence a This 4 3 1 “Can you send #2 again?” is “Sure. Here you go.” 2
  • 88. IP IP Load Web App Internet balancer server server DB Client
  • 89. The HTTP “stack” IP IP Load Web App Internet balancer server server DB Client
  • 90. The HTTP “stack” TCP TCP IP IP Load Web App Internet balancer server server DB Client
  • 91. The HTTP “stack” SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client
  • 92. The HTTP “stack” HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client
  • 93. Getting a page by hand
  • 94. Getting a page by hand Trying 67.205.65.12... Connected to bitcurrent.com. Escape character is '^]'.
  • 95. Getting a page by hand Trying 67.205.65.12... Connected to bitcurrent.com. Escape character is '^]'. GET / <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/ xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head profile="http://gmpg.org/xfn/11"> <script type="text/javascript" src="http:// www.bitcurrent.com/wp-content/themes/ grid_focus_public/js/perftracker.js"></script> <script> </body> </html> Connection closed by foreign host.
  • 96. Static content HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client image.gif GET www.example.com/image.gif
  • 97. Static content HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client image.gif GET www.example.com/image.gif
  • 98. Static content Dynamic content HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client dynamic.jsp GET www.example.com/dynamic.jsp
  • 99. Static content Dynamic content HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client dynamic.jsp GET www.example.com/dynamic.jsp
  • 100. Static content Dynamic Stored content data HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client (Database) POST www.example.com/data.cgi
  • 101. Static content Dynamic Stored content data HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client (Database) POST www.example.com/data.cgi
  • 102. Browser Data center Server
  • 103. Browser Data center Server
  • 104. Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking)
  • 105. Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking) SSL (“Someone might be listening!”) SSL (“Here’s a decoder ring”)
  • 106. Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking) SSL (“Someone might be listening!”) SSL (“Here’s a decoder ring”) HTTP GET / (“Can I have your home page?”) HTTP 200 OK (“Sure!”) (thinks [index.html] (“Here it is!”) a bit) (Renders furiously) Bump, bump. [img js css] (“Have this too!”)
  • 107. Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking) SSL (“Someone might be listening!”) SSL (“Here’s a decoder ring”) HTTP GET / (“Can I have your home page?”) HTTP 200 OK (“Sure!”) (thinks [index.html] (“Here it is!”) a bit) (Renders furiously) Bump, bump. [img js css] (“Have this too!”) TCP FIN (“Thanks! I’m done now.”) TCP FIN ACK (“You’re welcome. Have a nice day.”)
  • 108.
  • 109. “Page load time” isn’t simple Documents versus event models AJAX Mobility CDNs Third-party content Embedded objects and plug-ins
  • 110. Connections to load Connection 0 - www.bitcurrent.com (67.205.65.12) Connection 1 - www.bitcurrent.com (67.205.65.12) Connection 2 - 4qinvite.4q.iperceptions.com (64.18.71.70) Connection 3 - static.slideshare.net (66.114.49.24) Connection 4 - static.slideshare.net (66.114.49.24) Connection 5 - www.feedburner.com (66.150.96.123) Connection 6 - static.getclicky.com (204.13.8.18) Connection 7 - cetrk.com (208.67.183.100) Connection 8 - in.getclicky.com (204.13.8.18) Connection 9 - crazyegg.com (208.67.180.236) Connection 10 - www.google-analytics.com (72.14.223.147) Connection 11 - www.apture.com (67.192.46.19) Connection 12 - static.apture.com (67.192.46.25) Connection 13 - s.clicktale.net (66.114.49.24) Connection 14 - www.clicktale.net (75.125.82.70)
  • 111. Analytics site Server Data center Browser Server Mashup Server site
  • 112. Analytics site Server Data center Browser Server Snore. Mashup Server site
  • 113. What ultimately matters: When can the user start using the application as its designers intended?
  • 114. Part of the problem You control You’re blamed for Server latency Page rendering Network latency for Total network latency known content and User environment network parameters
  • 115. Part of the problem You control You’re blamed for Server latency Page rendering Network latency for Total network latency known content and User environment network parameters You need diagnostic metrics so you can fix it.
  • 116. Part of the problem You control You’re blamed for Server latency Page rendering Network latency for Total network latency known content and User environment network parameters You need escalation You need metrics so you can prove diagnostic metrics it and make it someone so you can fix it. else’s problem.
  • 117. Why measure Tactical, to find and fix Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part three Where to measure
  • 118. Three tiers of data WAN accessibility: One test from many locations Can everybody get here? App functionality: Several tests of key processes Is my business model working correctly? Tiered tests: Frequent metrics of each tier Is network, service, CPU, data I/O to blame?
  • 119. WAN accessibility Place A Task B Client Goal C ... Load Web App Internet balancer server server DB Client
  • 120. Analytics can tell you a lot.
  • 121.
  • 122.
  • 123.
  • 124.
  • 125. App functionality Page A Page B Client Event C Load Web App Internet balancer server server DB Client
  • 128.
  • 130. Landing page: View one story Task: Log in Enter credentials Verify Recovery
  • 131. Landing page: View one story Task: Log in Enter credentials Verify Recovery Task: Forward a story Enter recipients Enter message Send
  • 132. Landing page: Task: View one story Create account Task: Log in Pick name Check if free Enter credentials Set Password Verify CAPTCHA Recovery Send mail Get confirm Task: Forward a story Enter recipients Enter message Send
  • 133. Landing page: Task: View one story Create account Task: Log in Pick name Check if free Enter credentials Set Password Verify CAPTCHA Recovery Send mail Get confirm Task: Forward a story Task: Submit Enter recipients a new story Enter message Send Enter URL Describe Deduplicate Post it
  • 134. Landing page: Task: View one story Create account Task: Log in Pick name Place: View stories Check if free Enter credentials Vote up Next 25 Set Password Verify Vote down Last 25 CAPTCHA Recovery Send mail Get confirm Task: Forward a story Task: Submit Enter recipients a new story Enter message Send Enter URL Describe Deduplicate Post it
  • 135. Landing page: Task: View one story Create account Task: Log in Pick name Place: View stories Check if free Enter credentials Vote up Next 25 Set Password Verify Vote down Last 25 CAPTCHA Recovery Send mail Place: Read Get confirm poster comments Vote up Next 25 Task: Vote down Last 25 Forward a story Task: Submit Enter recipients a new story Enter message Send Enter URL Describe Deduplicate Post it
  • 136. Landing page: Task: View one story Create account Task: Log in Pick name Place: View stories Check if free Enter credentials Vote up Next 25 Set Password Verify Vote down Last 25 CAPTCHA Recovery Send mail Place: Read Get confirm poster comments Vote up Next 25 Task: Vote down Last 25 Forward a story Task: Submit Enter recipients a new story Place: My Enter message Enter URL account Send Describe Change My address comments Deduplicate Change PW See karma Post it
  • 137. Landing page: Create acct. View one story Task: Log in Place: View stories Place: Read poster comments Task: Forward a story Task: Submit a new story Place: My account
  • 138. Landing page: Create acct. Create acct. View one story Form uptime Place: View stories Task: Log in # started Bad form Place: Read # CAPTCHA poster comments Mail uptime Task: Forward a story Mail bounced Task: Submit a new story Place: My Confirm & return account Return 3x
  • 139. Landing page: Create acct. View one story Task: Log in Place: View stories Place: View stories Stories/visit Place: Read # up/down poster comments Time/story Top stories Task: Forward a story Task: Submit Refresh time Views/page a new story Place: My account
  • 140. Landing page: Create acct. View one story Task: Log in Place: View stories Place: Read poster comments Task: Forward a story Task: Submit a new story Place: My account
  • 141. Places Efficiency matters How quickly, how many, productivity Learning curve OK Leave when they’re bored Collect “aha” feedback A/B test content for pages/session, exits
  • 142. Tasks Effectiveness matters Completion, abandonment Intuitiveness rules Leave when they change their mind or it breaks Collect “motivation” feedback A/B test layouts for conversion
  • 143. 2 sides of the same coin End user Web analytics monitoring What did Could they visitors do? do it?
  • 144. For e-commerce sites Can people buy things?
  • 145. For media sites Are ads loading quickly and successfully clicked through? Is content loading fast enough for visitors?
  • 146. For collaboration sites Can visitors contribute (posting content, voting?) Is bad content being mitigated (trolling, spam)?
  • 147. For SaaS sites Are your end users productive? Are they making fewer mistakes? Is the site working during customers’ business hours?
  • 148. Tiered tests Place A Task B Client Goal C Load Web App Internet balancer server server DB Client
  • 149. Testing the tiers Load Web App Internet balancer server server DB Client Request Do some Search a Request a uncached heavy dataset for big object object computing a string (Or watch (Or track CPU) query time)
  • 150. ,)-$(&./01+2(3/04(#$+#+( &)$ %,$ %+$ %*$ !"#$%&'()%(*+( %&$ %)$ ,$ '""#$($ +$ '""#$&$ *$ '""#$%$ &$ )$ !""#$&$ -./01$2341$ !""#$%$ 53"67$2341$ 8!9$2341$ ':$2341$
  • 151. Why measure Tactical, to find and fix Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part four Where to measure: How to measure WAN, from everywhere Core app functionality performance data Tiers of components
  • 153. Load Web App Internet balancer server server DB Client
  • 154. Management tool Load Web App Internet balancer server server DB Client
  • 155. Load Web App Internet balancer server server DB Client
  • 156. Load Web App Internet balancer server server DB Client
  • 157. Test Testing config node Data center Testing node Website Testing node
  • 158. Test Testing config node Data center Testing node Website Testing node
  • 159. Test Testing config node Data center Testing node Website Testing node
  • 160. Test Testing config node Data center Testing node Website Reporting service Testing node
  • 161.
  • 162.
  • 163.
  • 164.
  • 165.
  • 166.
  • 167.
  • 168. Three things to watch for Cached vs. uncached Scripts vs. puppetry Simultaneous vs. sequential
  • 169. 0 10 Load time (seconds) Cached Uncached
  • 170. 10 Load time (seconds) 3.157s 0 Cached Uncached
  • 171. 13.349s 10 Load time (seconds) 3.157s 0 Cached Uncached
  • 173. Testing script Site: test.com Page: index.html Script interpreter
  • 174. Testing script Site: test.com Page: index.html Script interpreter HTTP GET www.test.com/index.html
  • 175. Testing script Site: test.com Page: index.html Script interpreter 200 OK index.html image.gif stylesheet.css etc...
  • 176. Testing script Site: test.com Test complete Page: index.html Script interpreter
  • 178. Browser controller DOM actions (“click on button 4”) Actual browser
  • 179. Browser controller DOM actions (“click on button 4”) Actual browser HTTP GET www.test.com/index.html
  • 180. Browser controller DOM actions (“click on button 4”) Actual browser 200 OK index.html image.gif stylesheet.css etc...
  • 181. Browser controller DOM actions DOM contents (“click on button 4”) (“DIV contains ‘error’”) Actual browser
  • 182.
  • 184. Simultaneous Sequential 5 tests from 5 tests at 15:00 15:00 to 15:05
  • 185. Synthetic pros & cons Pros Cons Easy to set up Brittle Only way to test without Detects macro outages, not actual visitor traffic user events Can compare to Good geographic & network competitors coverage costs money, Easy baseline establishment generates load Detects a problem before No measurement of traffic visitors sees it volume Consistent data over time Places load on the site under test
  • 186. Ultimately, Synthetic testing shows you if the site’s working.
  • 190.
  • 191.
  • 192.
  • 193. Browser Web server
  • 194. Browser Load Web balancer server
  • 195. Browser Load Web Network balancer server tap
  • 196. Browser Load Web Network balancer server tap
  • 197. Browser Load Web Network balancer server tap
  • 198. Browser Load Web Network balancer server tap
  • 199. Browser Load Web Network balancer server tap
  • 200. Browser Load Web Network balancer server tap
  • 201. Browser Load Web Network balancer server tap User A
  • 202. Browser Load Web Network balancer server tap User A User B User C
  • 203. Browser Load Web Network balancer server tap User A User B User C Visit history P1 P2 P3
  • 204. Browser Load Web Network balancer server tap User A User B User C Visit Aggregate history reports P1 P2 P3
  • 205. Browser Load Web Network balancer server tap User A User B User C Visit Aggregate Alerts history reports ! P1 P2 P3
  • 206.
  • 207.
  • 208.
  • 209.
  • 210.
  • 211.
  • 212. TopN, worstN RUM tools are excellent for more qualitative data What’s most broken? What’s biggest? What’s slowest? What’s most inconsistent?
  • 213. RUM pros & cons Pros Cons Directly correlated with May require physical clickstream, analytics installation Watches everything, not just Can be a privacy risk the things you know about Doesn’t work if there’s no Can be used to reproduce traffic problems Need to filter out your own Measures traffic as well as visits, crawlers, etc. performance
  • 214. Ultimately RUM shows you if the site’s working.
  • 215. Why measure Tactical, to find and fix Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part five Where to measure: Getting the math right WAN, from everywhere Core app functionality Tiers of components How to measure it: Synth, to ensure it’s working RUM, to see where it’s broken
  • 217. 0 10 20 30 40 50 60 70 80 90 Age
  • 218. Average age = 10 0 10 20 30 40 50 60 70 80 90 Age
  • 219. 20 Average age = 10 Count 0 0 10 20 30 40 50 60 70 80 90 Age
  • 220. 20 Average age = 10 Count 0 0 10 20 30 40 50 60 70 80 90 Age
  • 221. 20 Average age = 10 Count 0 0 10 20 30 40 50 60 70 80 90 Age
  • 222.
  • 223.
  • 224. Average varies wildly, making it hard to threshold properly or see a real slow-down.
  • 225. 80th percentile only spikes once for a legitimate slow-down (20% of users affected)
  • 226. Setting a useful threshold on percentiles gives less false positives and more real alerts
  • 227. 200 # of requests 0 0 2 4 6 8 10 12 14 16 18 20 Page load time (in seconds)
  • 228. 200 Average latency = 5s # of requests 0 0 2 4 6 8 10 12 14 16 18 20 Page load time (in seconds)
  • 229. 0 # of requests 200 0 2 4 Average latency = 5s 6 Page load time (in seconds) 8 10 12 14 16 18 95th percentile latency = 19s 20
  • 230.
  • 231. KISS
  • 232. “It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.” http://media.photobucket.com/image/einstein/derekabril/einstein_010.png
  • 233. “As simple as possible, but no simpler.” (FYI, this is irony.)
  • 235. Login 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
  • 236. Login Average 4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout Average 6s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite Average 9s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
  • 237. Login Average 4s 95% 8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout Average 6s 95% 10s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite Average 9s 95% 12s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
  • 238. Login Average 4s 95% 8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Mode 2s Checkout Average 6s 95% 10s Mode 5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite Average 9s 95% 12s Mode 1s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
  • 239. Login Average 4s 95% 8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Mode 2s Checkout Average 6s 95% 10s Mode 5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite Average 9s 95% 12s Mode 1s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Aggregate? Average 6s 95% 12s Mode 5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
  • 240. 740 260 Login: <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
  • 241. 740 260 Total samples 1000 Login: Below threshold 740 <=4s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s target threshold 74%
  • 242. 740 260 Total samples 1000 Login: Below threshold 740 <=4s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s target threshold 74% 370 630 Checkout: Total samples 1000 Below threshold 370 <=5s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s target threshold 37% 610 390 Invite: Total samples 1000 <=8s Below threshold 610 Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s target threshold 61%
  • 243. 740 260 Total samples 1000 Login: Below threshold 740 <=4s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 74% target threshold 370 630 Checkout: Total samples 1000 Below threshold 370 <=5s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 37% target threshold 610 390 Invite: Total samples 1000 Below threshold 610 <=8s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 61% target threshold Aggregate? Total samples 3000 Below threshold 1720 Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 57% target threshold
  • 244. 740 260 Total samples 1000 Login: Below threshold 740 <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 148 252 Checkout: Total samples 400 Below threshold 148 <=5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 366 366 Invite: Total samples 600 <=8s Below threshold 366 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Total samples 2000 Below threshold 1254 Percent below 63% target threshold
  • 245. 740 260 Total samples 1000 Login: Below threshold 740 <=4s Weight 1 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 148 252 Checkout: Total samples 400 Below threshold 148 <=5s Weight 5 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 366 366 Invite: Total samples 600 <=8s Below threshold 366 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Weight 2
  • 246. Login: <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout: <=5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite: <=8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Total requests inside target Login page 740/1000 Checkout page 148/400 Invite process 366/600
  • 247. Login: <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout: <=5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite: <=8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Total requests inside target Weight Weighted Login page 740/1000 1 740/1000 Checkout page 148/400 5 740/2000 Invite process 366/600 2 732/1200
  • 248. Login: <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout: <=5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite: <=8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Total requests inside target Weight Weighted Login page 740/1000 1 740/1000 Checkout page 148/400 5 740/2000 Invite process 366/600 2 732/1200 Total score 2212/4200 53%
  • 250. )" &!!!" !"#$%&'()*+(,&-*.*/0".1& !*2,&'.,%0)3.1& (" %#!!" #" %!!!" '" $#!!" &" $!!!" %" #!!" $" !" !" $" %" &" '" #" (" )" *" +" $!"$$"$%"$&"$'"$#"$("$)"$*"$+"%!"%$"%%"%&"%'"%#"%("%)"%*"%+"&!"&$"&%"&&"&'"&#"&("&)"&*" 4#5& ,-./0" 12-34-5.602" 14789:12-34-5.602;"
  • 251. )" &!!!" !"#$%&'()*+(,&-*.*/0".1& !*2,&'.,%0)3.1& (" %#!!" #" 71% correlation %!!!" '" $#!!" between traffic &" $!!!" %" $" !" #!!" !" and latency. $" %" &" '" #" (" )" *" +" $!"$$"$%"$&"$'"$#"$("$)"$*"$+"%!"%$"%%"%&"%'"%#"%("%)"%*"%+"&!"&$"&%"&&"&'"&#"&("&)"&*" 4#5& ,-./0" 12-34-5.602" 14789:12-34-5.602;" If you have traffic predictions, and latency is correlated with performance, you may be able to estimate performance in the future from the business plan.* *It’s seldom this simple.
  • 252.
  • 253.
  • 254. Baselines Establish an agreed-upon set of metrics, and always compare to these baselines. What does “normal” look like? Weekly variance? Seasonality?
  • 255. Why measure Tactical, to find and fix Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part six Where to measure: Targeting metrics WAN, from everywhere Core app functionality to your audience Tiers of components How to measure it: Synth, to ensure it’s working RUM, to see where it’s broken Get the math right
  • 256. Your goal is to be clearly understood.
  • 257. How technical are they? Your goal is to be clearly understood.
  • 258. How technical are they? Your goal is How will they to be clearly use it? understood.
  • 259. How technical are they? To fix Your goal is something How will they to be clearly use it? understood.
  • 260. How technical are they? To fix Your goal is something How will they To escalate to be clearly use it? to others understood.
  • 261. How technical are they? To fix Your goal is something How will they To escalate to be clearly use it? to others understood. To plan the future
  • 262. How technical are they? To fix Your goal is something How will they To escalate to be clearly use it? to others understood. To plan the future Translate to their jargon
  • 263. How technical are they? To fix Your goal is something How will they To escalate to be clearly use it? to others understood. To plan the future What words do Translate to they use? their jargon
  • 264. By timeframe Type of metric Timeframe Delivery Detail
  • 265. By timeframe Type of metric Timeframe Delivery Detail Break/fix monitoring
  • 266. By timeframe Type of metric Timeframe Delivery Detail Break/fix monitoring Daily reports
  • 267. By timeframe Type of metric Timeframe Delivery Detail Break/fix monitoring Daily reports Quarterly planning
  • 268. By timeframe Type of metric Timeframe Delivery Detail Break/fix Push alerts Simple Urgent monitoring to PDA messages Daily reports Quarterly planning
  • 269. By timeframe Type of metric Timeframe Delivery Detail Break/fix Push alerts Simple Urgent monitoring to PDA messages Daily Historical Automated Mail PDF reports context Quarterly planning
  • 270. By timeframe Type of metric Timeframe Delivery Detail Break/fix Push alerts Simple Urgent monitoring to PDA messages Daily Historical Automated Mail PDF reports context Quarterly Part of big Prepared Slide deck planning picture
  • 271. By medium Where will this wind up? Dashboard NOC screen Log file Someone’s spreadsheet Inbox http://www.flickr.com/photos/warrenski/4190341621/
  • 272. Why measure Tactical, to find and fix Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part seven Where to measure: Marching orders WAN, from everywhere Core app functionality Tiers of components How to measure it: Synth, to ensure it’s working RUM, to see where it’s broken Get the math right
  • 273. So what should you Some homework. do?
  • 274. First Meet your analytics team Find out What are the key goals they’re monitoring? Where are visitors coming from? What are the most common entrance and exit pages?
  • 275. Second Pick the three processes, pages, or functions that matter most to you Landing pages, or part of a conversion funnel
  • 276. Third Set up monitoring of: Your site from many places (synthetic testing) Your top 3 core business processes (synthetic or RUM) Your important infrastructure tiers (from agents + synthetic, or RUM)
  • 277. Fourth Wait a week or two To establish a baseline To detect seasonal variance To show others and get buy-in
  • 278. Fifth While you’re waiting, understand the elements latency and how they affect your performance DNS SSL Network latency Host (server) latency Client page load time
  • 279. Set a target threshold Now that you have an idea of what “normal” is, set a threshold ... but not just any threshold.
  • 280. The login page Function will have a total latency Metric of under 4 seconds Target with a cached browser copy User situation from any US branch office Testing point 95% of the time Percentile weekdays, 8AM ET to 6M PST Time window by synth test at 5m intervals Collection type
  • 281. Apdex score = ( ) ( Satisfied requests + Tolerating requests /2 ) All requests
  • 282. How Apdex works Frustrated: over 8 seconds Tolerating: 2-8 seconds Satisfied: 0-2 seconds
  • 283. How Apdex works Frustrated: 5 hits Total requests: 100 Tolerating: 30 hits (65) + (30/2) = 0.80 100 Satisfied: 65 hits
  • 284. Train your audience Visit key stakeholders and walk them through the report Get them used to the information In the same format At the same time From the same place
  • 285. Put monitoring into your release cycle Talk to the development team Adding instrumentation Identifying new code functions that need testing Verifying whether optimization worked
  • 286. Part eight Some tools to check out
  • 287. Paid
  • 288. Synthetic Keynote Systems Gomez Webmetrics Alertsite Dotcom Monitor Pingdom ...and many others
  • 289. RUM Client-side AJAX (Gomez, Coradiant TrueSight Edge) Full Agent-based (Aternity) disclosure: We both worked at Inline (sniffer/tap) Coradiant. Coradiant, Tealeaf, Beatbox(HP), Atomic Labs, Compuware Apdex Server-side (logfile, agent)
  • 290. Analytics Omniture Webtrends Coremetrics Woopra etc. (lots of specialization)
  • 293. Firebug Also: Webkit inspector, getfirebug.com Google Page Speed
  • 296. Monitor.us (Free ain’t pretty, and pretty ain’t free, but it works.) mon.itor.us
  • 297. AJAX measurement libraries Collecting from visitors: Jiffy (http://code.google.com/p/jiffy-web/) AJAX client sends measurements to Apache collector Other resources ZK-Gazer (http://code.google.com/p/zk-gazer/) http://www.ajaxperformance.com/ (Ryan Breen) http://www.opensourcetesting.org/performance.php
  • 298. YSlow http://justtalkaboutweb.com/wp-content/uploads/2008/06/yslow.gif http://events.stanford.edu/events/196/19695/souders.jpg
  • 299. Sites Dashboard Juice analytics’ Dashboard spy blog Insight’s gallery Simple Complexity
  • 300. Part nine Planning for the future
  • 301. AJAX
  • 302. AJAX As for your male and female slaves whom you may have: you may buy male and female slaves from among the nations that are around you. - Leviticus 25:44
  • 305. GET index.html HTTP/1.1 Host: www.stockprice.com Cookie: sessionID=KDF74INED6 Accept: */* <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/ xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <head profile="http://gmpg.org/xfn/11"> <title>Stock price for Apple</title> <script type="text/javascript" src="http:// www.bitcurrent.com/wp-content/themes/ grid_focus_public/js/perftracker.js"></script> <script> <body id=gsr topmargin=3 marginheight=3> AAPL:243.20 </body> </html>
  • 306. GET index.html Host: www.stockprice.com Cookie: sessionID=KDF74INED6 AAPL:243.20
  • 307. Web of Web of documents events (circa 1999) (circa 2008) http://www.flickr.com/photos/dnorman/2781572080/ http://www.flickr.com/photos/adamkr/4650637393/
  • 308. Recap What you need to go and do now.
  • 309. Metrics must be Relevant: related to a core business assumption Actionable: the basis for a decision or improvement Reproducible: documented and generated cleanly Understandable: easy for stakeholders to grok Accurate: providing the correct view of what happened
  • 310. Visit your analytics team & read your business model Pick three core business functions to watch Start monitoring One page from many places Key business functions Infrastructure tiers Take a deep breath and establish a baseline Analyze elements of latency while you wait Set target thresholds using a meaningful SLA Calculate a consistent score & train your audience Make it part of the release cycle
  • 311.
  • 312. Metric Source Target
  • 313. Metric Source Target Onload time
  • 314. Metric Source Target Onload From many time places
  • 315. Metric Source Target Onload From many To the top time places landing page
  • 316. Metric Source Target Onload From many To the top Uncached time places landing page Cached
  • 317. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server time
  • 318. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one time place, often
  • 319. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process
  • 320. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server time
  • 321. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one time place, often
  • 322. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU)
  • 323. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation
  • 324. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation TopN
  • 325. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst TopN pages
  • 326. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst TopN pages by
  • 327. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst TopN pages by Error rate
  • 328. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst Server TopN pages by Error rate latency
  • 329. Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst Server Network TopN pages by Error rate latency latency
  • 330. Got one report? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 331. Got one report? 5,000 Unique page views 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 332. Got one report? 5,000 Unique page views >4s or error 2-4s 0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 333. Got one report? 5,000 Unique page views >4s or error 2-4s 0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 334. Got one report? 5,000 Unique page views >4s or error 2-4s 0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 335. Got one report? 5,000 Unique page views >4s or error 2-4s 0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 336. Got one report? 5,000 Unique page views >4s or error 2-4s 0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 337. Got one report? 5,000 $10,000 Revenue (total sales) Unique page views >4s or error 2-4s 0 $0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 338. Got one report? 5,000 $10,000 Revenue (total sales) Unique page views Conversions >4s or error 2-4s 0 $0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  • 339. Got one report? 5,000 $10,000 Revenue (total sales) Unique page views Conversions >4s or error 2-4s 0 $0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16