Performance in business terms

3,396 views

Published on

You know that performance is crucial to your company's success, but do the people in the corner office know this? You need to get the message across using the language they speak and targeting the goals they care about.

This session -- presented by Strangeloop president Joshua Bixby at the 2011 Web Performance Summit -- summarizes the benefits of a faster website or web app, then delves into a series of how-tos for creating a business case for web performance in your organization.

Published in: Technology
1 Comment
11 Likes
Statistics
Notes
No Downloads
Views
Total views
3,396
On SlideShare
0
From Embeds
0
Number of Embeds
1,690
Actions
Shares
0
Downloads
0
Comments
1
Likes
11
Embeds 0
No embeds

No notes for slide
  • If you’re here, you’re a performance convert. But one of your biggest problems may be trying to explain the urgency of performance to non-techies in your company… and getting higher-ups to commit to long-term investing in performance. Sure, there are lots of high-profile stories of how speeding up pages has been successful for mega-sites…
  • I talk to a lot of execs, and I hear this a lot. I get why. It’s difficult for mortal companies to see themselves in relation to ecommerce mega-giants thatmake billions of dollars a year and have teams of in-house performance engineers to do their bidding.
  • Again, performed 50/50 test to see the impact of faster pages on performance:Conversions increased by 9%Cartsize by 11%Sales by 13%
  • Again, 50/50 test of the site after implementing the Site Optimizer service. Wetracked three metrics among both test groups: revenue per visitor, revenue per visit, and overall revenue. The goal was a baseline increase of 2.5% across all three metrics. The accelerated site dramatically outperformed this goal:Revenue per visit +8%Overall revenue +6%
  • I talk to a lot of execs, and I hear this a lot. I get why. It’s difficult for mortal companies to see themselves in relation to ecommerce mega-giants thatmake billions of dollars a year and have teams of in-house performance engineers to do their bidding.
  • The obvious answer is to just implement Site Optimizer, speed things up, then check out conversion rate changes using a segmentation test, but this isn’t an option for everyone. So we developed a hack. It lets you use two tools you’re probably already familiar with – Google Analytics and WebPagetest – to slice your own data a bunch of different ways, and create decent proxies for performance.
  • Last fall, we performed a study which showed that IE8 is about 25% faster than IE7 across almost 200 websites. Based on this, we felt that browser version is a solid performance proxy for exploring conversion behaviour. Reference original case study: “The first thing we did was perform a Webpagetest in IE7 and IE8. We found that his site was 30% faster in IE8.”
  • Explore different connection speeds within IE8. First, perform Webpagetests on the different connection speeds, and then compare them to the results in Google Analytics.If you’re using Google Analytics, I think the easiest way to get this data is with a custom report that looks something like this. (reference Google Analytics screen grab)From case study: “Again we found a remarkable relationship between connection speed and order value. On average, online shoppers using T1 connections spent about 11% more than shoppers with DSL connections. And shoppers with T1 connections spent roughly 32% more than those using dialup.”
  • To pass any hardcore statistical muster, a much more in-depth regression test would need to occur. But these early proof points are enough to convince many non-believers that performance matters and they should invest in it.
  • In a real-world application of this approach, with all variables accounted for, the optimized site still outperformed the unoptimized site.
  • From case study:Before I released this hack into the wild, I needed to apply this methodology to one last test to determine if it had any validity in the real world. I took a Strangeloop customer who had been through a rigorous month-long 50/50 test. In this particular case, with all other variables accounted for, the optimized site outperformed the unoptimized site. On average, order value for the optimized site was 20% greater than for the unoptimized site
  • This is what you need to start with.You may have your own variation on this. What’s the best way of showing this?
  • With a table?This is a table showing the load time and start render time for the top 20 ecommerce sites of the 2009 holiday shopping season. There’s some interesting data here, but if you were a performance newbie, you wouldn’t know it.Tables show that you’ve done your homework and tabulated your data. Okayin the appendix of a performance report, but they’re not goingto light fires under any butts.
  • Better…Side by side comparison graphic. (It’s easy to create using Webpagetest’s visual comparison feature. First, run your side-by-side test, then click the “Export filmstrip as an image” text link on the bottom of the results page.)You can see how your site loads, frame by frame, compared to your competitors.Interesting, but requires a bit of scrutiny to understand.Good to include in a competitive analysis section of your performance audit, but not a showstopper.
  • This is when you share the findings of your 5-minute speed/benefit analysis. It can be as simple as Google Analytics screenshots like this.
  • After you’ve grabbed attention, then start providing big-picture context.Use concrete benchmarks to create goals… and introduce competitiveness.You can get this data from companies like Gomez, which updates benchmarks every week.This index may focus on larger companies, but these numbers are relevant to all companies. Here’s why…
  • Users don’t care if your company is large or small. They expect all sites to load fast.
  • Twitter @joshuabixby
  • Performance in business terms

    1. 1. Performance in business terms<br />
    2. 2. Agenda<br /><ul><li> Introduction
    3. 3. Basic terminology and concepts
    4. 4. Case study
    5. 5. Performance stats and examples
    6. 6. 5 minute performance benchmark
    7. 7. Business case
    8. 8. Q&A</li></li></ul><li>
    9. 9.
    10. 10. Web Performance<br />
    11. 11.
    12. 12. Basic terminology and concepts<br />
    13. 13. 8<br />www.strangeloopnetworks.com<br />
    14. 14.
    15. 15. Waterfall chart<br />10<br />
    16. 16. Waterfall chart<br />HTML<br />11<br />
    17. 17. Waterfall chart<br />HTML<br />Resources<br />12<br />
    18. 18. Waterfall chart<br />Start <br />Render<br />Document <br />Complete<br />Fully<br />Loaded<br />13<br />
    19. 19. Waterfall component breakdown<br />DNS lookup<br />DNS Lookup<br /><ul><li>Takes one packet in each direction
    20. 20. Time is limited to the latency of the connection (DSL/Cable/etc) for the single round trip</li></ul>14<br />
    21. 21. Waterfall component breakdown<br />TCP Connection<br />TCP Connection<br /><ul><li>3 packets</li></ul> - ClientServer (SYN)<br /> - ServerClient (SYN/ACK)<br /> - ClientServer (ACK)<br /><ul><li>Time is limited to the latency of the connection (DSL/Cable/etc) for the single round trip</li></ul>15<br />
    22. 22. Waterfall component breakdown<br />Time To First Byte<br />Time to First Byte<br /><ul><li>Time from the request packet (usually 1) to the first packet of the response
    23. 23. Includes the latency of the connect ion (DSL/Cable/etc) for a single round trip
    24. 24. Also includes whatever server think time was involved in generating the response</li></ul>16<br />
    25. 25. Waterfall component breakdown<br />Content Download<br />Download<br /><ul><li>Time it takes for the entire content of the response to get to the browser
    26. 26. Mostly limited by the bandwidth limit of the connection (DSL/Cable/etc)</li></ul>17<br />
    27. 27. First view vs Repeat View<br />18<br />
    28. 28. Concurrency<br />Concurrency<br /><ul><li>Blocking script don't let the browser use available resources (network, connections, etc) to fetch more content from the server
    29. 29. The overall page load time and render time are both affected negatively)</li></ul>Blocking<br />Concurrent<br />19<br />
    30. 30. Case Study: Automation in action<br />
    31. 31. Methodology<br />
    32. 32. Let’s Automate<br />
    33. 33.
    34. 34. A Few Points of Clarification<br />We’ll use it to describe where performance pain points are, but that doesn’t mean the page actually has these problems<br />What we’re going to do:<br />Improve performance incrementally<br />Not so good<br />(slow)<br />Awesome<br />(fast)<br />* The real Velocity site is somewhere in the middle!<br />
    35. 35. Performance Summary<br />http://bit.ly/au01VY /<br />
    36. 36. Waterfall<br />First View<br />Repeat View<br />
    37. 37. Per Object<br />
    38. 38. Content Analysis by Type<br />
    39. 39. Content Analysis by Domain<br />
    40. 40. Show Me Where It Hurts<br />Problem Analysis<br />
    41. 41. Performance Problems<br />Too many connections (too much orange)<br />
    42. 42. Too Many Connections<br />97 Connections<br />(almost one per request)<br />
    43. 43. Too Many Connections<br />
    44. 44. Performance Problems<br />Too many connections (too much orange)<br />Too many bytes (too much blue)<br />
    45. 45. Too Many Bytes<br />By Type<br />By Domain<br />
    46. 46. Performance Problems<br />Too many connections (too much orange)<br />Too many bytes (too much blue)<br />Concurrency<br />
    47. 47. Concurrency<br />Browsers can only open and use so many connections at once<br />www.browserscope.org<br />
    48. 48. Performance Problems<br />Too many connections (too much orange)<br />Too many bytes (too much blue)<br />Concurrency<br />Bad Caching for repeat views<br />
    49. 49. Bad Repeat View<br />Full Fetches<br />Validation<br />
    50. 50. Bad Repeat View<br />Connections<br />Bytes<br />
    51. 51. Performance Problems<br />Too many connections (too much orange)<br />Too many bytes (too much blue)<br />Concurrency<br />Bad Caching for repeat views<br />No CDN (the greens are too big)<br />
    52. 52. The Green Problem #1: No CDN<br />TTFB<br />
    53. 53. Performance Problems<br />Too many connections (too much orange)<br />Too many bytes (too much blue)<br />Concurrency<br />Bad Caching for repeat views<br />No CDN (the greens are too big)<br />Too Many Roundtrips (too many greens)<br />
    54. 54. The Green Problem #2: Roundtrips<br />Repeat View<br />First View<br />80 Requests<br />78 Requests<br />27 Requests<br />14 Requests<br />
    55. 55. The Green Problem #2: Roundtrips<br />Every fetch still pays the HTTP overhead penalty  TTFB is still a problem<br />Exacerbated by concurrency issues<br />Getting worse as number of objects per page grows<br />Generally, the hardest problem to solve<br />
    56. 56. Performance Problems<br />Too many connections<br />Too many bytes (too much blue)<br />Concurrency<br />Bad Caching for repeat views<br />No CDN (the greens are too big)<br />Too Many Roundtrips (too many greens)<br />Others<br />
    57. 57. Examples of Other Problems<br />Blocking Javascript<br />3rd party calls (http://stevesouders.com/p3pc/)<br />
    58. 58. Before and after: Keep-alives & compression<br />http://www.youtube.com/watch?v=KPYBF41yiFw<br />
    59. 59. Summary<br />
    60. 60. Let’s Fix It!!<br />
    61. 61. Set-Up<br />
    62. 62. Testing Environment<br />WebPageTest Client<br />(East Coast)<br />O’Reilly Datacenter <br />(West Coast)<br />Automation <br />SERVICE (CLOUD)<br />
    63. 63. Acceleration Methodology<br />
    64. 64. Stepwise Acceleration<br />Start from the beginning and fix the easy stuff<br />Step by step acceleration of the page<br />Apply techniques/methods/etc and see the result<br />Try to make it as fast as possible<br />
    65. 65. Step 1: <br />Low Hanging Fruit<br />
    66. 66. Keep-Alive<br />Solves the too-many connection problem (Less Orange!)<br />Will help alleviate the TCP connection setup overhead<br />97 <br />Connections<br />
    67. 67. Compression<br />Addresses the too-many-bytes problem (Less Blue!)<br />We’ll compress textual content (html/css/etc)<br />Not the only solution to less blue, but the easiest<br />
    68. 68. WebPageTest<br />http://bit.ly/cKkjGz<br />
    69. 69. Before and After<br />~17.8sec<br />~10.5sec<br />
    70. 70. What We Helped: Keep-Alive<br />97 Connections  19 Connections<br />
    71. 71. What We Helped: Compression<br />
    72. 72. How Did We Do?<br />Original<br />KA+Comp<br />Improvement<br />First View <br />Repeat View<br />52%<br />71%<br />34%<br />94%<br />31%<br />51%<br />23%<br />75%<br />40%<br />62%<br />
    73. 73.
    74. 74. Pros and Cons<br />Pros<br />Really easy to do<br />Single configuration switches in servers, proxies, or load balancers<br />Good benefit seen right away<br />Cons<br />Compression has processing overhead<br />On their own they’re just not enough<br />
    75. 75. Step 2: <br />Repeat View Problem<br />
    76. 76. Poor Client Side Caching<br />Original<br />KA+Comp<br />
    77. 77. Problem Still Exists<br />~6.2sec<br />
    78. 78. How Do We Get Better Caching<br />RFC 2616, Section 13<br />Caching headers should be used on static (non-changing) objects, so they can be cached browser-side <br />And by intermediate caching proxies<br />Validators are not enough<br />
    79. 79. WebPageTest<br />http://bit.ly/aCP3iX<br />
    80. 80. Before and After<br />~2.0sec<br />~6.2sec<br />
    81. 81. How Did We Do?<br />KA+Comp<br />With Good Caching<br />Improvement<br />Repeat View<br />70%<br />42%<br />67%<br />
    82. 82. Pros and Cons<br />Pros<br />Good caching can have a major performance impact on repeat visits to a page<br />Sometimes it’s easy to do<br />Browsers generally pay attention (although interpretation may vary slightly)<br />Cons<br />The spec appears scary<br />Invalidation and stale content<br />
    83. 83. Step 3: <br />CDN<br />
    84. 84. What Does a CDN do?<br />Global network of proxy caches<br />Get cacheable content close to users<br />Reduce TTFB (smaller greens)<br />
    85. 85. TTFB Problem<br />
    86. 86. WebPageTest<br />http://bit.ly/a9ZJcF<br />
    87. 87. Before and After<br />~8.3sec<br />~10.5sec<br />
    88. 88. TTFB Savings<br />Per object TTFB savings of ~50%<br />
    89. 89. How Did We Do?<br />KA+Comp<br />+CDN<br />Improvement<br />First View<br />21%<br />17%<br />22%<br />0.7 sec<br />2.3 sec<br />2.7 sec<br />Seconds Gained<br />
    90. 90. Before and after: Adding a CDN<br />http://www.youtube.com/watch?v=BR5hO5rL8lE<br />
    91. 91. Pros and Cons<br />Pro<br />Good mitigation of the TTFB problem<br />Established industry: lots of vendors to choose from<br />Cons<br />Sometimes costly<br />May require code change (CDN’ed objects should be written to the CDN domain)<br />
    92. 92. Step 4: <br />Steroids- the Hard Stuff<br />
    93. 93. We Can Get Better!<br />Still too many roundtrips<br />Still too many bytes<br />Not Fast Enough!!<br />
    94. 94. What to do Next?<br />Reduce Roundtrips<br />Combine images<br />Combine JavaScript<br />Combine CSS<br />Reduce Payload even more<br />Minify CSS and JavaScript<br />Add Image Compression<br />Increase Concurrency<br />Add a couple of domains to the mix<br />
    95. 95. WebPageTest<br />http://bit.ly/bbT3v4<br />
    96. 96. Before and After<br />~3.8sec<br />~8.3sec<br />
    97. 97. How Did We Do?<br />+CDN<br />81<br />107<br />+Strangeloop<br />11<br />37<br />Improvement<br />First View<br />19%<br />30%<br />54%<br />45%<br />31%<br />0.5 sec<br />4.6 sec<br />4.1 sec<br />Seconds Gained<br />
    98. 98. Before and after: The final, fastest version<br />http://www.youtube.com/watch?v=IPn0T1UacIA<br />
    99. 99. Pros and Cons<br />Pros<br />Most significant benefit for the hardest part of the acceleration lifecycle<br />Address multiple performance points (somtimes multiple ones with the same technique)<br />Cons<br />It’s not easy<br />Regression<br />
    100. 100. Different Browsers<br />Safari Mobile<br />Chrome 10<br />IE 7<br />IE 8<br />IE 6<br />FireFox 4<br />Chrome 9<br />FireFox 3<br />Safari 5<br />
    101. 101. Flow<br />Conversion<br />?<br />?<br />?<br />3.8 Seconds<br />11 Roundtrips<br />
    102. 102.
    103. 103.
    104. 104.
    105. 105.
    106. 106.
    107. 107.
    108. 108. “These stats don’t apply to me.”<br />
    109. 109.
    110. 110.
    111. 111. “These stats don’t apply to me either.”<br />
    112. 112. How to perform a 5-minute speed/revenue benefit analysis of your site<br />
    113. 113. Step 1Perform WebPagetests in IE7 and IE8. Calculate the load time difference as a percentage.<br />
    114. 114. Step 2Use Google Analytics to calculate your current order value per visit for IE7 and IE8.<br />
    115. 115. Step 3Determine your order value per visit, by connection speed (cable, DSL, T1, dialup).<br />
    116. 116. Step 3 con’tDetermine your order value per visit, by connection speed (cable, DSL, T1, dialup).<br />
    117. 117. Step 4Correlate the results of steps 1 to 3 in simple graphs.<br />
    118. 118. Step 5Test your site to get a sense of how much faster it could be.<br />strangeloopnetworks.com/test-your-website<br />
    119. 119. Step 6Compare these gains to your graphs from step 4. What lift can you anticipate in value per visit?<br />
    120. 120. Caveats<br />Correlation does not imply causation. <br />Browser and connection speed might imply something about the buyer (i.e. s/he is more affluent) that is unrelated to the effects of speed.<br />
    121. 121. But this trend seems to hold true in the real world.<br />
    122. 122. Case study<br />
    123. 123. How to be your company’s in-house performance evangelist<br />
    124. 124. The average exec wants to know 3 things<br />What’s in it for the company?<br />What’s in it for me?<br />How do we compare to our competitors?<br />
    125. 125. When talking to an exec…<br />Tell the time, not how the watch works.<br />Most important, urgent point first.<br />Keep it short.<br />Keep it simple.<br />Make it visual.<br />Be ready to give context.<br />
    126. 126. What to say: #1<br />“Our site is slower than our competitors. We’re losing money.”<br />
    127. 127.
    128. 128.
    129. 129. What to say: #2<br />“We’ve proven that, when our site is faster for users, they spend more.”<br />
    130. 130. What to say: #3<br />“This is where we should be aiming.”<br />
    131. 131. What to say: #4<br />“Consumers expect EVERY website to load fast.”<br />

    ×