NY Web Perf Meetup: Peeling the Web Performance Onion

4,121 views

Published on

NY Web Perf Meetup presentation
Web Performance Monitoring

Published in: Technology
  • Be the first to comment

NY Web Perf Meetup: Peeling the Web Performance Onion

  1. 1. 5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED<br />Michael Mikikian - 1<br />Peeling the Web Performance Onion<br />Drit Suljoti, Co-founder Catchpoint Systems<br />drit@catchpoint.com<br />
  2. 2. Speed Matters<br />Shopzilla<br />Performance 6sec to 1.2 sec<br />Revenue +12%<br />Pageviews+25%<br />Yahoo<br />400ms improvement<br />Traffic +9%<br />Length of an eye blink is ~400ms<br />
  3. 3. Speed Matters<br />Amazon<br />Latency +100ms<br />Sales drop 1% / 250mil<br />Google<br />400ms latency<br />Searches drop 0.59%<br />
  4. 4. What Impacts Speed<br />5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED<br />4<br />
  5. 5. What Impacts Speed<br />5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED<br />5<br />Webpage Response (ms): 2,254<br /># Requests: 102<br /># Hosts: 11<br />
  6. 6. Web Performance Optimization<br />Reduce<br />Pack Tight<br />Merge<br />Leverage Cache<br />Partner Up<br />
  7. 7. WPO Works<br />5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED<br />For Ethan Selzer - 7<br />
  8. 8. 80 / 20 Rule<br />5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED<br />8<br />Steve Souders:<br />“80% of the end-user response time is spent <br />on the front-end. Start there.”<br />When things fail, a single request takes 80% of the time!<br />
  9. 9. Great Performance is More than SPEED<br />Availability can users access it?<br />Reliability is it working consistently and properly? <br />
  10. 10. Monitoring<br />
  11. 11. Web Performance Monitoring<br />Synthetic: simulates End User activity<br />Measures the end to end transaction<br />Continuous, 24x7<br />Detect anomalies<br />Track performance metrics over time<br />Measures every component<br />
  12. 12. The Foundations of the Web<br />IP<br />
  13. 13. Simple HTTP Transaction<br />DNS Lookup: The client tries to resolve the domain name for the request.<br />Client sends DNS Query to local ISP DNS server.<br />DNS server responds with the IP address for hostname.com<br />Connect: Client establishes TCP connection with the IP address of hostname.com<br />Client sends SYN packet.<br />Web server sends SYN-ACK packet.<br />Client answers with ACK packet, concluding the three-way TCP connection establishment.<br />Send: Client sends the HTTP request to the web server.<br />Wait: Client waits for the server to respond to the request.<br />Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content.<br />Load: Client loads the content of the response.<br />Web server sends second TCP segment with the PSH flag set.<br />Client sends ACK. (Client sends ACK every two segments it receives. from the host)<br />Web server sends third TCP segment with HTTP_Continue.<br />Close: Client sends a a FIN packet to close the TCP connection.<br />
  14. 14. DNS<br />
  15. 15. DNS Recommendations<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED<br />15<br /><ul><li>Avoid Short TTL if possible.
  16. 16. Avoid multiple CNAMEs.
  17. 17. Rely on distributed NS server or utilize companies like 3Crowd, Cotendo, Dyn, UltraDNS,...
  18. 18. Ensure DNS servers capacity.
  19. 19. Avoid using Registrars, usually unreliable and slow.
  20. 20. Do not mix Private & Public IP addresses.</li></li></ul><li>Use Case: DNS<br />
  21. 21. Use Case: DNS<br />DNS Provider Switch<br />
  22. 22. Connect<br />DNS Lookup: The client tries to resolve the domain name for the request.<br />Client sends DNS Query to local ISP DNS server.<br />DNS server responds with the IP address for hostname.com<br />Connect: Client establishes TCP connection with the IP address of hostname.com<br />Client sends SYN packet.<br />Web server sends SYN-ACK packet.<br />Client answers with ACK packet, concluding the three-way TCP connection establishment.<br />Send: Client sends the HTTP request to the web server.<br />Wait: Client waits for the server to respond to the request.<br />Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content.<br />Load: Client loads the content of the response.<br />Web server sends second TCP segment with the PSH flag set.<br />Client sends ACK. (Client sends ACK every two segments it receives. from the host)<br />Web server sends third TCP segment with HTTP_Continue.<br />Close: Client sends a a FIN packet to close the TCP connection.<br />
  23. 23. Connect<br />What impacts Connect times?<br />Packet loss / network stability <br />Latency <br />Routers<br />Load Balancers<br />Network Cards<br />Server Load<br />Common issues:<br />Bad configuration of Load Balancers, Routers, Switches and the NIC cards! <br />Latency/Packet Loss/Link speed<br />TCP offloading <br />Capacity<br />
  24. 24. Use Case: Connect<br />
  25. 25. Use Case: Connect<br />Capacity Problems at Datacenter<br />
  26. 26. Wait<br />DNS Lookup: The client tries to resolve the domain name for the request.<br />Client sends DNS Query to local ISP DNS server.<br />DNS server responds with the IP address for hostname.com<br />Connect: Client establishes TCP connection with the IP address of hostname.com<br />Client sends SYN packet.<br />Web server sends SYN-ACK packet.<br />Client answers with ACK packet, concluding the three-way TCP connection establishment.<br />Send: Client sends the HTTP request to the web server.<br />Wait: Client waits for the server to respond to the request.<br />Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content.<br />Load: Client loads the content of the response.<br />Web server sends second TCP segment with the PSH flag set.<br />Client sends ACK. (Client sends ACK every two segments it receives. from the host)<br />Web server sends third TCP segment with HTTP_Continue.<br />Close: Client sends a a FIN packet to close the TCP connection.<br />
  27. 27. Wait<br />What impacts Wait times?<br />Network factors<br />The Application code<br />The Application Load / Capacity<br />Any other dependencies (database lookups, other servers, apis…)<br />Common issues:<br />Application Load - Profile your capacity / perform load testing<br />Database utilization, locks…, <br />Memory Leaks, Bugs…<br />
  28. 28. Use Case: Wait<br />
  29. 29. Use Case: Wait<br />
  30. 30. Use Case: Wait - Database Impact<br />DB Problems<br />
  31. 31. Load/Receive<br />DNS Lookup: The client tries to resolve the domain name for the request.<br />Client sends DNS Query to local ISP DNS server.<br />DNS server responds with the IP address for hostname.com<br />Connect: Client establishes TCP connection with the IP address of hostname.com<br />Client sends SYN packet.<br />Web server sends SYN-ACK packet.<br />Client answers with ACK packet, concluding the three-way TCP connection establishment.<br />Send: Client sends the HTTP request to the web server.<br />Wait: Client waits for the server to respond to the request.<br />Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content.<br />Load: Client loads the content of the response.<br />Web server sends second TCP segment with the PSH flag set.<br />Client sends ACK. (Client sends ACK every two segments it receives. from the host)<br />Web server sends third TCP segment with HTTP_Continue.<br />Close: Client sends a a FIN packet to close the TCP connection.<br />
  32. 32. Load/Receive<br />What impacts Wait times?<br />Network factors<br />The Application code<br />The Application Load / Capacity<br />Any other dependencies (database lookups, other servers, apis…)<br />Common issues:<br />Application Load - Profile your capacity / Load testing<br />Database utilization, locks…, <br />Memory Leaks, Bugs…<br />Bad chunked encoding!<br />
  33. 33. Factors You Control<br />Applications/System<br />Architecture, Code, DB, SQL Queries, …<br />Configurations<br />Partners: <br />DNS, CDN, ISP Links, Datacenter, Optimization, Ads…<br />ISP Peering <br />Hardware<br />Server, CPU, RAM, I/O, Routers,…<br />5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED<br />For Ethan Selzer - 29<br />
  34. 34. Factors You CANNOT Control<br />Internet <br />Fiber cut in DC, Latency East Coast to West Coast<br />Last Mile<br />The internet link to the end user!<br />End User Local Network<br />Misconfigured router, computers downloading/uploading torrents!<br />OS/Browser of the end user<br />Configurations, Add-ons, installed malware…<br />5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED<br />For Ethan Selzer - 30<br />
  35. 35. Parallel & Persistent HTTP Connections<br /><ul><li>Connect – Once per connection
  36. 36. DNS – Once or less per transaction per hostname</li></li></ul><li>Parallel & Persistent HTTP Connections<br />http://www.webpagetest.org<br />
  37. 37. HTML Content<br />Each request relies on HTTP - same rules apply!<br />Pay attention to bottlenecks<br />Primary URL<br />Inline JavaScript requests<br />Front end code impacts browser processing!<br />
  38. 38. Oops … bad httpd.conf file<br />5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED<br />34<br />
  39. 39. Oops … bad httpd.conf file<br />5/19/2011<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED<br />35<br />Apache configuration!<br />No More Keep Alives<br />
  40. 40. Monitoring Best Practices<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED<br />36<br />Monitor every component:<br /><ul><li>Key webpages with real browser
  41. 41. Setup monitoring for every key provider / hostname / requests
  42. 42. DNS
  43. 43. Origin servers
  44. 44. if Multiple datacenters, monitor them individually</li></ul>Baseline very often, track improvement or degradation<br />Keep an eye on performance during critical business hours (Volume)<br />Compare data pre-release and post release<br />Review Daily / Weekly /Monthly<br />Alerts on % delta<br />
  45. 45. Troubleshooting Tool Box<br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED<br />37<br />Standalone<br /><ul><li>Chrome – Developer tool
  46. 46. Firebug
  47. 47. IE 9 – Developer tool
  48. 48. HTTPWatch
  49. 49. Wireshark
  50. 50. Fiddler
  51. 51. Yslow
  52. 52. Dynatrace
  53. 53. Dig for DNS
  54. 54. MTR (Matt traceroute)</li></ul>Online:<br /><ul><li>WebPageTest
  55. 55. Google Page speed (http://pagespeed.googlelabs.com/)</li></ul><br />
  56. 56. Closing the Loop<br />Monitor Action Plan  Optimize Monitor… do it again<br />It isn’t over till it’s over.<br />And it’s never over.<br />It is not a sprint, it is a never ending marathon!<br />
  57. 57. CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED<br />39<br />THANK YOU.<br />http://blog.catchpoint.com<br />@catchpoint<br />
  58. 58. Troubleshooting: My 6 Best Friends <br />CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED<br />40<br /> Rudyard Kipling: “I keep six honest serving-men (They taught me all I knew);Their names are What and Why and When And How and Where and Who.”<br /><ul><li>What is there a problem? Is it one monitoring tool or multiple that turned red? is it only one node or multiple monitoring nodes in the case of a 3rd party tool? Is it the DB?…
  59. 59. Where is the problem? is it specific to a country? a city? a customer? a datacenter?
  60. 60. Who is affected? All customers? Some customers? Just the monitoring tools?
  61. 61. When did it start? Does it always happen? is it sporadic?
  62. 62. Why?Did something change? Did we roll out something? Did the environment change? Did the customer behavior change?
  63. 63. How did this happen? Change management? Router upgrade? Did someone upload a crazy ad?</li>

×