Metrics 101

Metrics 101 What to watch

What we’ll cover Why collect metrics Understanding web latency How to target your ﬁndings Concrete steps to get started

Part one Why collect metrics?

http://www.ﬂickr.com/photos/chidorian/12411641/

Downtime costs

Downtime costs eBay ofﬂine ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)

Downtime costs eBay ofﬂine ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999) Financial company down ($100K/h) 53.2% of ﬁnance companies lose over $100,000/hour (nextslm.org)

Downtime costs Amazon offline ($1M/h) Amazon loses nearly $1M/hour if down (NYT, 2008) eBay offline ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999) Financial company down ($100K/h) 53.2% of finance companies lose over $100,000/hour (nextslm.org)

Downtime costs Amazon offline ($1M/h) Amazon loses nearly $1M/hour if down (NYT, 2008) Network downtime ($42K/h) 1 hour of network downtime costs $42,000 (Gartner, 2003) eBay offline ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999) Financial company down ($100K/h) 53.2% of finance companies lose over $100,000/hour (nextslm.org)

Downtime costs Amazon offline ($1M/h) Amazon loses nearly $1M/hour if down (NYT, 2008) Network downtime ($42K/h) 1 hour of network downtime costs $42,000 (Gartner, 2003) eBay offline ($90K/h) 22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999) Financial company down ($100K/h) 53.2% of finance companies lose over $100,000/hour (nextslm.org) Let’s say $50K/h if you’re serious.

Availability Downtime/year Loss @$50K/h 90% % 36.5 days Can$43,800,000 95% 18.25 days Can$21,900,000 98% 7.30 days Can$8,760,000 99% 3.65 days Can$4,380,000 99.5% 1.83 days Can$2,196,000 99.8% 17.52 hours Can$876,000 99.9% 8.76 hours Can$438,000 99.95% 4.38 hours Can$219,000 99.99% 52.6 minutes Can$43,833 99.999% 5.26 minutes Can$4,383 99.9999% 31.5 seconds Can$438

Availability Downtime/year Loss @$50K/h 90% % 36.5 days Can$43,800,000 95% 18.25 days Can$21,900,000 98% 7.30 days Can$8,760,000 99% 3.65 days Can$4,380,000 99.5% 1.83 days Can$2,196,000 99.8% 17.52 hours Can$876,000 99.9% 8.76 hours Can$438,000 Less than 99.95% 4.38 hours Can$219,000 an hour a 99.99% 52.6 minutes Can$43,833 year 99.999% 5.26 minutes Can$4,383 99.9999% 31.5 seconds Can$438

Availability Downtime/year Loss @$50K/h 90% % 36.5 days Can$43,800,000 95% 18.25 days Can$21,900,000 98% 7.30 days Can$8,760,000 99% 3.65 days Can$4,380,000 99.5% 1.83 days Can$2,196,000 99.8% 17.52 hours Can$876,000 99.9% 8.76 hours Can$438,000 Less than 99.95% 4.38 hours Can$219,000 an hour a 99.99% 52.6 minutes Can$43,833 year 99.999% 5.26 minutes Can$4,383 Less than 99.9999% 31.5 seconds Can$438 a minute a year

Harris poll conducted by Tealeaf in 2008

You really don’t want web users to call you. $15 $12 $9 $6 $3 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High

You really don’t want web users to call you. $15 $12 $9 $6 $3 Can$0.24 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High

You really don’t want web users to call you. $15 $12 $9 $6 $3 Can$0.24 Can$0.45 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High

You really don’t want web users to call you. $15 $12 $9 $6 Can$3.00 $3 Can$0.24 Can$0.45 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High

You really don’t want web users to call you. $15 $12 $9 $6 Can$5.50 Can$3.00 $3 Can$0.24 Can$0.45 $0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High

http://www.ﬂickr.com/photos/pagedooley/2811157950/

If you don’t know the past you can’t know the future. If you don’t know the future, you can’t budget for it. Photo by Alan Cleaver from his Flicker Freestock set. Thanks, Alan! http://www.ﬂickr.com/photos/alancleaver/2638883650/

“A plan so crazy, it just might work.”

http://www.ﬂickr.com/photos/genewolf/147722350

http://www.ﬂickr.com/photos/billselak/366692332/

Everything starts with a baseline.

Everything starts with a baseline. Know what’s worst.

Everything starts with a baseline. Know what’s Prove you worst. made it better.

The cycle of optimization Metrics & strategy

The cycle of optimization Metrics & strategy Collection

The cycle of optimization Metrics & strategy Collection Reporting

The cycle of optimization Metrics & strategy Collection Reporting Institutionalizing the results

The cycle of optimization Metrics & strategy Collection Link to KPI/ Reporting ROI Institutionalizing the results

The cycle of optimization Metrics & strategy Optimization Collection & change Link to KPI/ Reporting ROI Institutionalizing the results

http://www.ﬂickr.com/photos/elsie/8229790/

Understanding your goals. http://www.ﬂickr.com/photos/itsgreg/446061432/

Organic Ad Campaigns search network $ 1 1 1 Advertiser site Visitor 2 O er 3 $ 8 Upselling 4 Abandonment Reach 5 Purchase step $ Mailing, alerts, Purchase step $ 9 promotions $ Conversion $ Disengagement 7 Enrolment 6 Impact on site $ Positive $ Negative

Bad $ 4 content Social Search Invitation network link results 4 Good content 1 $ 1 1 Collaboration site 2 Visitor Content creation Moderation $ 3 Spam & trolls $ Engagement 5 Viral 6 Social graph spread 7 Disengagement $ Impact on site $ Positive $ Negative

Enterprise subscriber $ 1 End user (employee) $ Refund $ 2 Renewal, upsell, SLA reference SaaS site violation Performance Good Bad 3 Helpdesk Support 5 $ Usability escalation costs 7 4 Good Bad Productivity Good Bad 6 Churn $ Impact on site $ Positive $ Negative

$ Media site Enrolment Targeted 2 embedded ad 5 $ 6 1 Ad Visitor network 4 3 5 Advertiser $ Departure $ site Impact on site $ Positive $ Negative

Why measure Tactical, to ﬁnd and ﬁx Strategic, to plan/trend Part two The elements of web latency

Slow sites suck

Slow sites suck Lower conversion rates

Slow sites suck Lower conversion rates Less likely to attract a loyal following

Slow sites suck Lower conversion rates Less likely to attract a loyal following Liable for damages

Slow sites suck Lower conversion rates Less likely to attract a loyal following Liable for damages Liable for refunds or service credits

Slow sites suck Lower conversion rates Less likely to attract a loyal following Liable for damages Liable for refunds or service credits Customers ﬁnd other channels that cost more

Why the web is slow A crash course in performance & availability.

Load Web App Internet balancer server server DB Client www.example.com

Your website Load Web App Internet balancer server server DB Client www.example.com

DNS Load Web App Internet balancer server server DB Client DNS “www.example.com”

DNS DNS lookup Load Web App Internet balancer server server DB Client DNS “www.example.com”

IP IP Load Web App Internet balancer server server DB Client

IP IP Load Web App Internet balancer server server DB Client Internet routing

IP R IP R Load Web App Internet R balancer server server DB Client R R Internet routing

IP R IP R Load Web App Internet R balancer server server DB Client R R TCP session

Letter writing Postal service

You Them (sender) (receiver)

This is a sentence You Them (sender) (receiver)

sentence This is a You Them (sender) (receiver)

This is a sentence 3 2 1 4 You Them (sender) (receiver)

This is sentence 2 1 You Them (sender) (receiver)

This is sentence 2 1 4 You Them (sender) (receiver)

This WTF? is sentence 2 1 4 You Them (sender) (receiver)

sentence a This 4 3 1

sentence a This 4 3 1 “Can you send #2 again?”

sentence a This 4 3 1 “Can you send #2 again?” is “Sure. Here you go.” 2

How computers “connect”

IP IP Load Web App Internet balancer server server DB Client

The HTTP “stack” IP IP Load Web App Internet balancer server server DB Client

The HTTP “stack” TCP TCP IP IP Load Web App Internet balancer server server DB Client

The HTTP “stack” SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client

The HTTP “stack” HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client

Getting a page by hand

Getting a page by hand Trying 67.205.65.12... Connected to bitcurrent.com. Escape character is '^]'.

Getting a page by hand Trying 67.205.65.12... Connected to bitcurrent.com. Escape character is '^]'. GET / <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/ xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head profile="http://gmpg.org/xfn/11"> <script type="text/javascript" src="http:// www.bitcurrent.com/wp-content/themes/ grid_focus_public/js/perftracker.js"></script> <script> </body> </html> Connection closed by foreign host.

Static content HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client image.gif GET www.example.com/image.gif

Static content Dynamic content HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client dynamic.jsp GET www.example.com/dynamic.jsp

Static content Dynamic Stored content data HTTP HTTP HTTP SSL SSL TCP TCP IP IP Load Web App Internet balancer server server DB Client (Database) POST www.example.com/data.cgi

Browser Data center Server

Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking)

Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking) SSL (“Someone might be listening!”) SSL (“Here’s a decoder ring”)

Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking) SSL (“Someone might be listening!”) SSL (“Here’s a decoder ring”) HTTP GET / (“Can I have your home page?”) HTTP 200 OK (“Sure!”) (thinks [index.html] (“Here it is!”) a bit) (Renders furiously) Bump, bump. [img js css] (“Have this too!”)

Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking) SSL (“Someone might be listening!”) SSL (“Here’s a decoder ring”) HTTP GET / (“Can I have your home page?”) HTTP 200 OK (“Sure!”) (thinks [index.html] (“Here it is!”) a bit) (Renders furiously) Bump, bump. [img js css] (“Have this too!”) TCP FIN (“Thanks! I’m done now.”) TCP FIN ACK (“You’re welcome. Have a nice day.”)

“Page load time” isn’t simple Documents versus event models AJAX Mobility CDNs Third-party content Embedded objects and plug-ins

Connections to load Connection 0 - www.bitcurrent.com (67.205.65.12) Connection 1 - www.bitcurrent.com (67.205.65.12) Connection 2 - 4qinvite.4q.iperceptions.com (64.18.71.70) Connection 3 - static.slideshare.net (66.114.49.24) Connection 4 - static.slideshare.net (66.114.49.24) Connection 5 - www.feedburner.com (66.150.96.123) Connection 6 - static.getclicky.com (204.13.8.18) Connection 7 - cetrk.com (208.67.183.100) Connection 8 - in.getclicky.com (204.13.8.18) Connection 9 - crazyegg.com (208.67.180.236) Connection 10 - www.google-analytics.com (72.14.223.147) Connection 11 - www.apture.com (67.192.46.19) Connection 12 - static.apture.com (67.192.46.25) Connection 13 - s.clicktale.net (66.114.49.24) Connection 14 - www.clicktale.net (75.125.82.70)

Analytics site Server Data center Browser Server Mashup Server site

Analytics site Server Data center Browser Server Snore. Mashup Server site

What ultimately matters: When can the user start using the application as its designers intended?

Part of the problem You control You’re blamed for Server latency Page rendering Network latency for Total network latency known content and User environment network parameters

Part of the problem You control You’re blamed for Server latency Page rendering Network latency for Total network latency known content and User environment network parameters You need diagnostic metrics so you can ﬁx it.

Part of the problem You control You’re blamed for Server latency Page rendering Network latency for Total network latency known content and User environment network parameters You need escalation You need metrics so you can prove diagnostic metrics it and make it someone so you can ﬁx it. else’s problem.

Why measure Tactical, to ﬁnd and ﬁx Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part three Where to measure

Three tiers of data WAN accessibility: One test from many locations Can everybody get here? App functionality: Several tests of key processes Is my business model working correctly? Tiered tests: Frequent metrics of each tier Is network, service, CPU, data I/O to blame?

WAN accessibility Place A Task B Client Goal C ... Load Web App Internet balancer server server DB Client

Analytics can tell you a lot.

App functionality Page A Page B Client Event C Load Web App Internet balancer server server DB Client

http://www.ﬂickr.com/photos/tinfoilraccoon/197640807/

Places and Tasks.

Landing page: View one story

Landing page: View one story Task: Log in Enter credentials Verify Recovery

Landing page: View one story Task: Log in Enter credentials Verify Recovery Task: Forward a story Enter recipients Enter message Send

Landing page: Task: View one story Create account Task: Log in Pick name Check if free Enter credentials Set Password Verify CAPTCHA Recovery Send mail Get conﬁrm Task: Forward a story Enter recipients Enter message Send

Landing page: Task: View one story Create account Task: Log in Pick name Check if free Enter credentials Set Password Verify CAPTCHA Recovery Send mail Get conﬁrm Task: Forward a story Task: Submit Enter recipients a new story Enter message Send Enter URL Describe Deduplicate Post it

Landing page: Task: View one story Create account Task: Log in Pick name Place: View stories Check if free Enter credentials Vote up Next 25 Set Password Verify Vote down Last 25 CAPTCHA Recovery Send mail Get conﬁrm Task: Forward a story Task: Submit Enter recipients a new story Enter message Send Enter URL Describe Deduplicate Post it

Landing page: Task: View one story Create account Task: Log in Pick name Place: View stories Check if free Enter credentials Vote up Next 25 Set Password Verify Vote down Last 25 CAPTCHA Recovery Send mail Place: Read Get conﬁrm poster comments Vote up Next 25 Task: Vote down Last 25 Forward a story Task: Submit Enter recipients a new story Enter message Send Enter URL Describe Deduplicate Post it

Landing page: Task: View one story Create account Task: Log in Pick name Place: View stories Check if free Enter credentials Vote up Next 25 Set Password Verify Vote down Last 25 CAPTCHA Recovery Send mail Place: Read Get conﬁrm poster comments Vote up Next 25 Task: Vote down Last 25 Forward a story Task: Submit Enter recipients a new story Place: My Enter message Enter URL account Send Describe Change My address comments Deduplicate Change PW See karma Post it

Landing page: Create acct. View one story Task: Log in Place: View stories Place: Read poster comments Task: Forward a story Task: Submit a new story Place: My account

Landing page: Create acct. Create acct. View one story Form uptime Place: View stories Task: Log in # started Bad form Place: Read # CAPTCHA poster comments Mail uptime Task: Forward a story Mail bounced Task: Submit a new story Place: My Conﬁrm & return account Return 3x

Landing page: Create acct. View one story Task: Log in Place: View stories Place: View stories Stories/visit Place: Read # up/down poster comments Time/story Top stories Task: Forward a story Task: Submit Refresh time Views/page a new story Place: My account

Landing page: Create acct. View one story Task: Log in Place: View stories Place: Read poster comments Task: Forward a story Task: Submit a new story Place: My account

Places Efﬁciency matters How quickly, how many, productivity Learning curve OK Leave when they’re bored Collect “aha” feedback A/B test content for pages/session, exits

Tasks Effectiveness matters Completion, abandonment Intuitiveness rules Leave when they change their mind or it breaks Collect “motivation” feedback A/B test layouts for conversion

2 sides of the same coin End user Web analytics monitoring What did Could they visitors do? do it?

For e-commerce sites Can people buy things?

For media sites Are ads loading quickly and successfully clicked through? Is content loading fast enough for visitors?

For collaboration sites Can visitors contribute (posting content, voting?) Is bad content being mitigated (trolling, spam)?

For SaaS sites Are your end users productive? Are they making fewer mistakes? Is the site working during customers’ business hours?

Tiered tests Place A Task B Client Goal C Load Web App Internet balancer server server DB Client

Testing the tiers Load Web App Internet balancer server server DB Client Request Do some Search a Request a uncached heavy dataset for big object object computing a string (Or watch (Or track CPU) query time)

,)-$(&./01+2(3/04(#$+#+( &)$ %,$ %+$ %*$ !"#$%&'()%(*+( %&$ %)$ ,$ '""#$($ +$ '""#$&$ *$ '""#$%$ &$ )$ !""#$&$ -./01$2341$ !""#$%$ 53"67$2341$ 8!9$2341$ ':$2341$

Why measure Tactical, to ﬁnd and ﬁx Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part four Where to measure: How to measure WAN, from everywhere Core app functionality performance data Tiers of components

Synthetic testing.

Load Web App Internet balancer server server DB Client

Management tool Load Web App Internet balancer server server DB Client

Load Web App Internet balancer server server DB Client

Test Testing conﬁg node Data center Testing node Website Testing node

Test Testing conﬁg node Data center Testing node Website Reporting service Testing node

Three things to watch for Cached vs. uncached Scripts vs. puppetry Simultaneous vs. sequential

0 10 Load time (seconds) Cached Uncached

10 Load time (seconds) 3.157s 0 Cached Uncached

13.349s 10 Load time (seconds) 3.157s 0 Cached Uncached

Testing script Script interpreter

Testing script Site: test.com Page: index.html Script interpreter

Testing script Site: test.com Page: index.html Script interpreter HTTP GET www.test.com/index.html

Testing script Site: test.com Page: index.html Script interpreter 200 OK index.html image.gif stylesheet.css etc...

Testing script Site: test.com Test complete Page: index.html Script interpreter

Browser controller Actual browser

Browser controller DOM actions (“click on button 4”) Actual browser

Browser controller DOM actions (“click on button 4”) Actual browser HTTP GET www.test.com/index.html

Browser controller DOM actions (“click on button 4”) Actual browser 200 OK index.html image.gif stylesheet.css etc...

Browser controller DOM actions DOM contents (“click on button 4”) (“DIV contains ‘error’”) Actual browser

Simultaneous 5 tests at 15:00

Simultaneous Sequential 5 tests from 5 tests at 15:00 15:00 to 15:05

Synthetic pros & cons Pros Cons Easy to set up Brittle Only way to test without Detects macro outages, not actual visitor trafﬁc user events Can compare to Good geographic & network competitors coverage costs money, Easy baseline establishment generates load Detects a problem before No measurement of trafﬁc visitors sees it volume Consistent data over time Places load on the site under test

Ultimately, Synthetic testing shows you if the site’s working.

Real User Monitoring.

Synthetic isn’t enough

Browser Web server

Browser Load Web balancer server

Browser Load Web Network balancer server tap

Browser Load Web Network balancer server tap User A

Browser Load Web Network balancer server tap User A User B User C

Browser Load Web Network balancer server tap User A User B User C Visit history P1 P2 P3

Browser Load Web Network balancer server tap User A User B User C Visit Aggregate history reports P1 P2 P3

Browser Load Web Network balancer server tap User A User B User C Visit Aggregate Alerts history reports ! P1 P2 P3

TopN, worstN RUM tools are excellent for more qualitative data What’s most broken? What’s biggest? What’s slowest? What’s most inconsistent?

RUM pros & cons Pros Cons Directly correlated with May require physical clickstream, analytics installation Watches everything, not just Can be a privacy risk the things you know about Doesn’t work if there’s no Can be used to reproduce traffic problems Need to filter out your own Measures traffic as well as visits, crawlers, etc. performance

Ultimately RUM shows you if the site’s working.

Why measure Tactical, to find and fix Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part five Where to measure: Getting the math right WAN, from everywhere Core app functionality Tiers of components How to measure it: Synth, to ensure it’s working RUM, to see where it’s broken

http://upload.wikimedia.org/wikipedia/commons/0/0e/Count-von-count.jpg

0 10 20 30 40 50 60 70 80 90 Age

Average age = 10 0 10 20 30 40 50 60 70 80 90 Age

20 Average age = 10 Count 0 0 10 20 30 40 50 60 70 80 90 Age

Average varies wildly, making it hard to threshold properly or see a real slow-down.

80th percentile only spikes once for a legitimate slow-down (20% of users affected)

Setting a useful threshold on percentiles gives less false positives and more real alerts

200 # of requests 0 0 2 4 6 8 10 12 14 16 18 20 Page load time (in seconds)

200 Average latency = 5s # of requests 0 0 2 4 6 8 10 12 14 16 18 20 Page load time (in seconds)

0 # of requests 200 0 2 4 Average latency = 5s 6 Page load time (in seconds) 8 10 12 14 16 18 95th percentile latency = 19s 20

“It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.” http://media.photobucket.com/image/einstein/derekabril/einstein_010.png

“As simple as possible, but no simpler.” (FYI, this is irony.)

http://www.ﬂickr.com/photos/evilerin/3540381299/ http://www.ﬂickr.com/photos/golf_pictures/2538894627/

Login 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

Login Average 4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout Average 6s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite Average 9s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

Login Average 4s 95% 8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout Average 6s 95% 10s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite Average 9s 95% 12s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

Login Average 4s 95% 8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Mode 2s Checkout Average 6s 95% 10s Mode 5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite Average 9s 95% 12s Mode 1s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

Login Average 4s 95% 8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Mode 2s Checkout Average 6s 95% 10s Mode 5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite Average 9s 95% 12s Mode 1s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Aggregate? Average 6s 95% 12s Mode 5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

740 260 Login: <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

740 260 Total samples 1000 Login: Below threshold 740 <=4s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s target threshold 74%

740 260 Total samples 1000 Login: Below threshold 740 <=4s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s target threshold 74% 370 630 Checkout: Total samples 1000 Below threshold 370 <=5s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s target threshold 37% 610 390 Invite: Total samples 1000 <=8s Below threshold 610 Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s target threshold 61%

740 260 Total samples 1000 Login: Below threshold 740 <=4s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 74% target threshold 370 630 Checkout: Total samples 1000 Below threshold 370 <=5s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 37% target threshold 610 390 Invite: Total samples 1000 Below threshold 610 <=8s Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 61% target threshold Aggregate? Total samples 3000 Below threshold 1720 Percent below 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 57% target threshold

740 260 Total samples 1000 Login: Below threshold 740 <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 148 252 Checkout: Total samples 400 Below threshold 148 <=5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 366 366 Invite: Total samples 600 <=8s Below threshold 366 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Total samples 2000 Below threshold 1254 Percent below 63% target threshold

740 260 Total samples 1000 Login: Below threshold 740 <=4s Weight 1 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 148 252 Checkout: Total samples 400 Below threshold 148 <=5s Weight 5 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 366 366 Invite: Total samples 600 <=8s Below threshold 366 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Weight 2

Login: <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout: <=5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite: <=8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Total requests inside target Login page 740/1000 Checkout page 148/400 Invite process 366/600

Login: <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout: <=5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite: <=8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Total requests inside target Weight Weighted Login page 740/1000 1 740/1000 Checkout page 148/400 5 740/2000 Invite process 366/600 2 732/1200

Login: <=4s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Checkout: <=5s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Invite: <=8s 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s Total requests inside target Weight Weighted Login page 740/1000 1 740/1000 Checkout page 148/400 5 740/2000 Invite process 366/600 2 732/1200 Total score 2212/4200 53%

(Snore?)

)" &!!!" !"#$%&'()*+(,&-*.*/0".1& !*2,&'.,%0)3.1& (" %#!!" #" %!!!" '" $#!!" &" $!!!" %" #!!" $" !" !" $" %" &" '" #" (" )" *" +" $!"$$"$%"$&"$'"$#"$("$)"$*"$+"%!"%$"%%"%&"%'"%#"%("%)"%*"%+"&!"&$"&%"&&"&'"&#"&("&)"&*" 4#5& ,-./0" 12-34-5.602" 14789:12-34-5.602;"

)" &!!!" !"#$%&'()*+(,&-*.*/0".1& !*2,&'.,%0)3.1& (" %#!!" #" 71% correlation %!!!" '" $#!!" between trafﬁc &" $!!!" %" $" !" #!!" !" and latency. $" %" &" '" #" (" )" *" +" $!"$$"$%"$&"$'"$#"$("$)"$*"$+"%!"%$"%%"%&"%'"%#"%("%)"%*"%+"&!"&$"&%"&&"&'"&#"&("&)"&*" 4#5& ,-./0" 12-34-5.602" 14789:12-34-5.602;" If you have trafﬁc predictions, and latency is correlated with performance, you may be able to estimate performance in the future from the business plan.* *It’s seldom this simple.

Baselines Establish an agreed-upon set of metrics, and always compare to these baselines. What does “normal” look like? Weekly variance? Seasonality?

Why measure Tactical, to ﬁnd and ﬁx Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part six Where to measure: Targeting metrics WAN, from everywhere Core app functionality to your audience Tiers of components How to measure it: Synth, to ensure it’s working RUM, to see where it’s broken Get the math right

Your goal is to be clearly understood.

How technical are they? Your goal is to be clearly understood.

How technical are they? Your goal is How will they to be clearly use it? understood.

How technical are they? To ﬁx Your goal is something How will they to be clearly use it? understood.

How technical are they? To ﬁx Your goal is something How will they To escalate to be clearly use it? to others understood.

How technical are they? To ﬁx Your goal is something How will they To escalate to be clearly use it? to others understood. To plan the future

How technical are they? To ﬁx Your goal is something How will they To escalate to be clearly use it? to others understood. To plan the future Translate to their jargon

How technical are they? To ﬁx Your goal is something How will they To escalate to be clearly use it? to others understood. To plan the future What words do Translate to they use? their jargon

By timeframe Type of metric Timeframe Delivery Detail

By timeframe Type of metric Timeframe Delivery Detail Break/ﬁx monitoring

By timeframe Type of metric Timeframe Delivery Detail Break/ﬁx monitoring Daily reports

By timeframe Type of metric Timeframe Delivery Detail Break/ﬁx monitoring Daily reports Quarterly planning

By timeframe Type of metric Timeframe Delivery Detail Break/ﬁx Push alerts Simple Urgent monitoring to PDA messages Daily reports Quarterly planning

By timeframe Type of metric Timeframe Delivery Detail Break/ﬁx Push alerts Simple Urgent monitoring to PDA messages Daily Historical Automated Mail PDF reports context Quarterly planning

By timeframe Type of metric Timeframe Delivery Detail Break/ﬁx Push alerts Simple Urgent monitoring to PDA messages Daily Historical Automated Mail PDF reports context Quarterly Part of big Prepared Slide deck planning picture

By medium Where will this wind up? Dashboard NOC screen Log ﬁle Someone’s spreadsheet Inbox http://www.ﬂickr.com/photos/warrenski/4190341621/

Why measure Tactical, to ﬁnd and ﬁx Strategic, to plan/trend What to measure: How long until a user can use the app as you intended? Part seven Where to measure: Marching orders WAN, from everywhere Core app functionality Tiers of components How to measure it: Synth, to ensure it’s working RUM, to see where it’s broken Get the math right

So what should you Some homework. do?

First Meet your analytics team Find out What are the key goals they’re monitoring? Where are visitors coming from? What are the most common entrance and exit pages?

Second Pick the three processes, pages, or functions that matter most to you Landing pages, or part of a conversion funnel

Third Set up monitoring of: Your site from many places (synthetic testing) Your top 3 core business processes (synthetic or RUM) Your important infrastructure tiers (from agents + synthetic, or RUM)

Fourth Wait a week or two To establish a baseline To detect seasonal variance To show others and get buy-in

Fifth While you’re waiting, understand the elements latency and how they affect your performance DNS SSL Network latency Host (server) latency Client page load time

Set a target threshold Now that you have an idea of what “normal” is, set a threshold ... but not just any threshold.

The login page Function will have a total latency Metric of under 4 seconds Target with a cached browser copy User situation from any US branch ofﬁce Testing point 95% of the time Percentile weekdays, 8AM ET to 6M PST Time window by synth test at 5m intervals Collection type

Apdex score = ( ) ( Satisﬁed requests + Tolerating requests /2 ) All requests

How Apdex works Frustrated: over 8 seconds Tolerating: 2-8 seconds Satisﬁed: 0-2 seconds

How Apdex works Frustrated: 5 hits Total requests: 100 Tolerating: 30 hits (65) + (30/2) = 0.80 100 Satisﬁed: 65 hits

Train your audience Visit key stakeholders and walk them through the report Get them used to the information In the same format At the same time From the same place

Put monitoring into your release cycle Talk to the development team Adding instrumentation Identifying new code functions that need testing Verifying whether optimization worked

Part eight Some tools to check out

Synthetic Keynote Systems Gomez Webmetrics Alertsite Dotcom Monitor Pingdom ...and many others

RUM Client-side AJAX (Gomez, Coradiant TrueSight Edge) Full Agent-based (Aternity) disclosure: We both worked at Inline (sniffer/tap) Coradiant. Coradiant, Tealeaf, Beatbox(HP), Atomic Labs, Compuware Apdex Server-side (logﬁle, agent)

Analytics Omniture Webtrends Coremetrics Woopra etc. (lots of specialization)

Open Source

Firebug getﬁrebug.com

Firebug Also: Webkit inspector, getﬁrebug.com Google Page Speed

Google Analytics analytics.google.com

webpagetest.org

Monitor.us (Free ain’t pretty, and pretty ain’t free, but it works.) mon.itor.us

AJAX measurement libraries Collecting from visitors: Jiffy (http://code.google.com/p/jiffy-web/) AJAX client sends measurements to Apache collector Other resources ZK-Gazer (http://code.google.com/p/zk-gazer/) http://www.ajaxperformance.com/ (Ryan Breen) http://www.opensourcetesting.org/performance.php

YSlow http://justtalkaboutweb.com/wp-content/uploads/2008/06/yslow.gif http://events.stanford.edu/events/196/19695/souders.jpg

Sites Dashboard Juice analytics’ Dashboard spy blog Insight’s gallery Simple Complexity

Part nine Planning for the future

AJAX As for your male and female slaves whom you may have: you may buy male and female slaves from among the nations that are around you. - Leviticus 25:44

http://www.ﬂickr.com/photos/farhannasir/4577508824/ Mobility

http://www.ﬂickr.com/photos/andrewparnell/2738598951/

GET index.html HTTP/1.1 Host: www.stockprice.com Cookie: sessionID=KDF74INED6 Accept: */* <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/ xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <head profile="http://gmpg.org/xfn/11"> <title>Stock price for Apple</title> <script type="text/javascript" src="http:// www.bitcurrent.com/wp-content/themes/ grid_focus_public/js/perftracker.js"></script> <script> <body id=gsr topmargin=3 marginheight=3> AAPL:243.20 </body> </html>

GET index.html Host: www.stockprice.com Cookie: sessionID=KDF74INED6 AAPL:243.20

Web of Web of documents events (circa 1999) (circa 2008) http://www.ﬂickr.com/photos/dnorman/2781572080/ http://www.ﬂickr.com/photos/adamkr/4650637393/

Recap What you need to go and do now.

Metrics must be Relevant: related to a core business assumption Actionable: the basis for a decision or improvement Reproducible: documented and generated cleanly Understandable: easy for stakeholders to grok Accurate: providing the correct view of what happened

Visit your analytics team & read your business model Pick three core business functions to watch Start monitoring One page from many places Key business functions Infrastructure tiers Take a deep breath and establish a baseline Analyze elements of latency while you wait Set target thresholds using a meaningful SLA Calculate a consistent score & train your audience Make it part of the release cycle

Metric Source Target

Metric Source Target Onload time

Metric Source Target Onload From many time places

Metric Source Target Onload From many To the top time places landing page

Metric Source Target Onload From many To the top Uncached time places landing page Cached

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server time

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one time place, often

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server time

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one time place, often

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU)

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation TopN

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst TopN pages

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst TopN pages by

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst TopN pages by Error rate

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst Server TopN pages by Error rate latency

Metric Source Target Onload From many To the top Uncached time places landing page Cached Server From one To your core time place, often business process Server From one To each tier time place, often (web, I/O, CPU) List Criteria Segmentation Worst Server Network TopN pages by Error rate latency latency

Got one report? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Got one report? 5,000 Unique page views 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Got one report? 5,000 Unique page views >4s or error 2-4s 0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Got one report? 5,000 $10,000 Revenue (total sales) Unique page views >4s or error 2-4s 0 $0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Got one report? 5,000 $10,000 Revenue (total sales) Unique page views Conversions >4s or error 2-4s 0 $0 <2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Thanks! @seanpower sean@httpd.org @acroll alistair@bitcurrent.com www.watchingwebsites.com (and go buy this.)

http://www.watchingwebsites.com	3556
http://www.royans.net	1164
http://bharath-marrivada.blogspot.com	600
http://seanpower.com	493
http://bharath-marrivada.blogspot.in	432
http://velocityconf.com	405
http://www.slideshare.net	49
http://bharath-marrivada.blogspot.tw	34
http://bharath-marrivada.blogspot.co.uk	24
http://gaverder.pl	22
http://static.slidesharecdn.com	15
http://blog.virtacore.com	13
http://bharath-marrivada.blogspot.ca	12
http://clintchao.tumblr.com	9
http://bharath-marrivada.blogspot.fr	9
http://bharath-marrivada.blogspot.de	7
http://www.techgig.com	6
http://translate.googleusercontent.com	5
http://bharath-marrivada.blogspot.nl	5
http://bharath-marrivada.blogspot.com.br	5
http://www.bharath-marrivada.blogspot.in	4
http://http.www.watchingwebsites.com.davekim.goinstant.com	3
http://bharath-marrivada.blogspot.cz	3
http://bharath-marrivada.blogspot.ro	3
http://bharath-marrivada.blogspot.com.au	3
http://bharath-marrivada.blogspot.ch	2
http://bharath-marrivada.blogspot.it	2
http://bharath-marrivada.blogspot.sg	2
http://bharath-marrivada.blogspot.ie	2
http://pkoraca.blogspot.com	2
http://bharath-marrivada.blogspot.se	2
http://web.archive.org	2
https://vpn.chula.ac.th	2
http://us-w1.rockmelt.com	2
http://bharath-marrivada.blogspot.fi	1
http://anonymouse.org	1
http://bharath-marrivada.blogspot.mx	1
http://bharath-marrivada.blogspot.kr	1
http://bharath-marrivada.blogspot.no	1
http://bharath-marrivada.blogspot.com.es	1
http://virtacore.com	1
http://xss.yandex.net	1
http://www.google.co.uk	1
http://feeds.feedburner.com	1
http://wildfire.gigya.com	1
http://www.tumblr.com	1
http://1.2.hidemyass.com	1
http://cache.baidu.com	1
http://bharath-marrivada.blogspot.be	1
http://bharath-marrivada.blogspot.co.nz	1
http://www.linkedin.com	1

Metrics 101

by Alistair Croll

Accessibility

Categories

Upload Details

Usage Rights

51 Embeds 6,916

Statistics

Metrics 101 Presentation Transcript