Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AB Testing, Ads and other 3rd party tags - SmashingConf London - 2018

1,795 views

Published on

Talk at Smashing Conf - 7th Feb 2018 (Video - https://vimeo.com/254703766)

Explores some of the issues that 3rd-party tags introduce when we add them to our sites, some ways of measuring the impact, and challenges we still have

Published in: Technology
  • Be the first to comment

AB Testing, Ads and other 3rd party tags - SmashingConf London - 2018

  1. 1. https://www.flickr.com/photos/9422878@N08/6884612488 AB Testing, Ads & other 3rd-Party Tags @AndyDavies · Smashing Conf · Feb ‘18
  2. 2. Infrastructure and code managed by someone else Third-party = Guy Podjarny, Velocity Santa Clara, 2014
  3. 3. We’re including more and more of them 0 25 50 75 100 125 150 175 200 225 01-Jan-13 01-Jul-13 01-Jan-14 01-Jul-14 01-Jan-15 01-Jul-15 01-Jan-16 01-Jul-16 01-Jan-17 01-Jul-17 01-Jan-1 50th 75th 90th 95th HTTPArchive Jan ’13 to Jan ‘18Percentile
  4. 4. Well… we’ve got plenty of choice!
  5. 5. And we’re told they’re going to make us more money www.flickr.com/photos/59937401@N07/5856660723
  6. 6. EVERYTHING SHOULD HAVE VALUE, BECAUSE EVERYTHING HAS A COST Tim Kadlec
  7. 7. How can we measure their cost? 1. Test a page in WebPageTest 2. Repeat with selected 3rd-Parties blocked 3. Compare the two results
  8. 8. Requests to be blocked
  9. 9. Use the filmstrip view to compare visual experience
  10. 10. Use the filmstrip view to compare visual experience
  11. 11. Feed the timings into RUM to understand the business impact Conversions (At least three RUM products support ‘What-If’ type analysis)
  12. 12. MedianPageLoadTime(s) 0 2 4 6 8 10 12 14 W eek 0 W eek 1 W eek 2 W eek 3 W eek 5 W eek 5 Android iOS 3rd-Parties really can cost you money…
  13. 13. MedianPageLoadTime(s) 0 2 4 6 8 10 12 14 W eek 0 W eek 1 W eek 2 W eek 3 W eek 5 W eek 5 Android iOSRemoved BazaarVoice for Android visitors 3rd-Parties really can cost you money…
  14. 14. -4s
  15. 15. £+26%
  16. 16. Understand how 3rd-parties affect your key milestones Working? Useful? Usable? What’s happening here? Or here?
  17. 17. Analysing activity before firstMeaningfulPaint
  18. 18. How do we do this for many pages? 1. Grab Chrome timelines from HTTPArchive 2. Analyse with modified version of Jean-Pierre Vincent’s 3rd-party-cpu-abuser 3. Aggregate the results
  19. 19. URL Count Mean (ms) SD (ms) www.google-analytics.com 115 232 85 cdn.optimizely.com 110 440 329 pagead2.googlesyndication.com 68 375 266 assets.adobedtm.com 62 335 243 ajax.googleapis.com 59 259 114 securepubads.g.doubleclick.net 57 380 367 connect.facebook.net 47 238 89 platform.twitter.com 22 241 127 s7.addthis.com 22 360 218 tags.tiqcdn.com 20 294 137 nexus.ensighten.com 17 368 230 www.googletagmanager.com 15 317 413 apis.google.com 13 317 153 cdn.onesignal.com 12 270 136cdnjs.cloudflare.com 12 244 87 Top 15 CPU consumers (by frequency) - HTTPArchive rank < 5000
  20. 20. URL Count Mean (ms) SD (ms) www.google-analytics.com 115 232 85 cdn.optimizely.com 110 440 329 pagead2.googlesyndication.com 68 375 266 assets.adobedtm.com 62 335 243 ajax.googleapis.com 59 259 114 securepubads.g.doubleclick.net 57 380 367 connect.facebook.net 47 238 89 platform.twitter.com 22 241 127 s7.addthis.com 22 360 218 tags.tiqcdn.com 20 294 137 nexus.ensighten.com 17 368 230 www.googletagmanager.com 15 317 413 apis.google.com 13 317 153 cdn.onesignal.com 12 270 136cdnjs.cloudflare.com 12 244 87 Top 15 CPU consumers (by frequency) - HTTPArchive rank < 5000
  21. 21. URL Count Mean (ms) SD (ms) www.google-analytics.com 115 232 85 cdn.optimizely.com 110 440 329 pagead2.googlesyndication.com 68 375 266 assets.adobedtm.com 62 335 243 ajax.googleapis.com 59 259 114 securepubads.g.doubleclick.net 57 380 367 connect.facebook.net 47 238 89 platform.twitter.com 22 241 127 s7.addthis.com 22 360 218 tags.tiqcdn.com 20 294 137 nexus.ensighten.com 17 368 230 www.googletagmanager.com 15 317 413 apis.google.com 13 317 153 cdn.onesignal.com 12 270 136cdnjs.cloudflare.com 12 244 87 Top 15 CPU consumers (by frequency) - HTTPArchive rank < 5000
  22. 22. URL Count Mean (ms) SD (ms) www.google-analytics.com 115 232 85 cdn.optimizely.com 110 440 329 pagead2.googlesyndication.com 68 375 266 assets.adobedtm.com 62 335 243 ajax.googleapis.com 59 259 114 securepubads.g.doubleclick.net 57 380 367 connect.facebook.net 47 238 89 platform.twitter.com 22 241 127 s7.addthis.com 22 360 218 tags.tiqcdn.com 20 294 137 nexus.ensighten.com 17 368 230 www.googletagmanager.com 15 317 413 apis.google.com 13 317 153 cdn.onesignal.com 12 270 136cdnjs.cloudflare.com 12 244 87 Top 15 CPU consumers (by frequency) - HTTPArchive rank < 5000
  23. 23. URL Count Mean (ms) SD (ms) www.google-analytics.com 115 232 85 cdn.optimizely.com 110 440 329 pagead2.googlesyndication.com 68 375 266 assets.adobedtm.com 62 335 243 ajax.googleapis.com 59 259 114 securepubads.g.doubleclick.net 57 380 367 connect.facebook.net 47 238 89 platform.twitter.com 22 241 127 s7.addthis.com 22 360 218 tags.tiqcdn.com 20 294 137 nexus.ensighten.com 17 368 230 www.googletagmanager.com 15 317 413 apis.google.com 13 317 153 cdn.onesignal.com 12 270 136cdnjs.cloudflare.com 12 244 87 Top 15 CPU consumers (by frequency) - HTTPArchive rank < 5000
  24. 24. URL Count Mean (ms) SD (ms) www.google-analytics.com 115 232 85 cdn.optimizely.com 110 440 329 pagead2.googlesyndication.com 68 375 266 assets.adobedtm.com 62 335 243 ajax.googleapis.com 59 259 114 securepubads.g.doubleclick.net 57 380 367 connect.facebook.net 47 238 89 platform.twitter.com 22 241 127 s7.addthis.com 22 360 218 tags.tiqcdn.com 20 294 137 nexus.ensighten.com 17 368 230 www.googletagmanager.com 15 317 413 apis.google.com 13 317 153 cdn.onesignal.com 12 270 136cdnjs.cloudflare.com 12 244 87 Top 15 CPU consumers (by frequency) - HTTPArchive rank < 5000
  25. 25. Let’s look at testing, using Optimizley as an example https://www.flickr.com/photos/pinkangelbabe/427255017
  26. 26. Optimizely’s cost varies by implementation 0 1.25 2.5 3.75 5 Duration of script execution before firstMeaningfulPaint (ms) 0 500 1000 1500 2000 2500 Optimizely (n=110) Circle size = number occurrences
  27. 27. On the critical rendering path
  28. 28. Improving performance Reducing number of • URL Matches • Experiments • Plugins (Widgets) reduces snippet size, download and execution times Removing jQuery from the bundle and using your own (if you already have it) can shrink bundle further
  29. 29. Add metrics from Optimizely to Analytics or RUM? optimizely.get("state").getActiveExperimentIds().length Some products have documented APIs, why not extract key metrics and send them to analytics or RUM? Number of experiments in snippet: Number of experiments active for this page view: Object.keys(optimizely.get('data').experiments).length
  30. 30. Use Performance APIs to build a picture Resource Timing • When was script requested, how long did it take to fetch, was it cached? User Timing • When did the initial script execute, how long did it take? LongTasks • Were there any script elements that took longer that 50ms?
  31. 31. Make sure tests are equivalent from a performance viewpoint Some things seen in the the wild: • One variant redirected to a new page, the other didn’t • Optimised vs unoptimised images (Can be good reasons why tests can’t be equivalent)
  32. 32. Could you do it server-side or or at CDN level? https://www.flickr.com/photos/kewl/8475764430
  33. 33. Know how to turn it off in an emergency https://www.flickr.com/photos/chrysics/4365802409
  34. 34. -150ms -300ms
  35. 35. £+5%
  36. 36. https://www.flickr.com/photos/gviciano/3128575291 What if I told you…
  37. 37. https://www.flickr.com/photos/gviciano/3128575291 a script loaded async, and even injected after page load can delay a page rendering? …
  38. 38. What was the first request in this navigation?
  39. 39. Request #61
  40. 40. ¯_( )_/¯
  41. 41. beforeUnload event took 400ms to execute! cloning DOM triggered extra network requests
  42. 42. beforeUnload event handlers took over 1,800ms to execute!!!
  43. 43. The examples aren’t isolated cases 0 3 6 9 12 15 Duration of beforeUnload event handlers (s) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Moat Ads (n=481) TeaLeaf (n=271)
  44. 44. Not always easy to spot these delays https://www.flickr.com/photos/fernandosanchez/2441716057
  45. 45. Navigation Timing API excludes beforeUnload Prompt for unload App Cache DNS TCP Request Response Processing onLoadRedirect unload navigation start
  46. 46. Navigation Timing API excludes beforeUnload Prompt for unload App Cache DNS TCP Request Response Processing onLoadRedirect unload Before Unload navigation start
  47. 47. Instrument addEventListener? !(function() { const callbacks = [] window.addEventListener = (function(_addEventListener){ _addEventListener.call(window, 'beforeunload', function(){ const args = Array.prototype.slice.call(arguments) let total = -1 * performance.now() callbacks.forEach(function(callback) { callback.apply(undefined, args) }) total += performance.now() console.info(`total: ${total}`) }) return function() { if (arguments[0] === 'beforeunload') { callbacks.push(arguments[1]) return } _addEventListener.apply(this, arguments) } })(window.addEventListener) })() This is just a sketch of an idea - don’t deploy it without improvements! https://gist.github.com/cvazac/de18cf4d045b01d179d8fdc2902d4997
  48. 48. WebPageTest? 1. WebPageTest script navigate https://www.game.co.uk/
 navigate https://www.game.co.uk/ 2. Download Timeline for step #2 3. Pipe through jq cat timeline.json | jq -r '[.[] | select(.name=="EventDispatch" and .args.data.type=="beforeunload") | .dur] | add / 1000')
  49. 49. DevTools?
  50. 50. https://www.flickr.com/photos/wnieke/4078161933
  51. 51. Do you include adverts? https://www.flickr.com/photos/helldini/6190516041
  52. 52. https://news.sky.com/story/hackers-behind-a-billion-dodgy-web-adverts-11219278
  53. 53. https://blog.trendmicro.com/trendlabs-security-intelligence/malvertising-campaign-abuses-googles-doubleclick-to-deliver-cryptocurrency-miners
  54. 54. https://discuss.httparchive.org/t/the-performance-impact-of-cryptocurrency-mining-on-the-web/1126/8
  55. 55. https://www.adlightning.com/ad-quality-report “…41% of the ads tracked exceeded the industry-approved maximum. Nearly 10% of the ads were larger than 5MB. Some ads are as large as 30 MBs.” “…average number of network requests and tracking scripts per ad was 56: 3.7x greater than the maximum suggested by the IAB.”
  56. 56. “…one third (32%) of the ads were overly processor- intensive. The study also found it common for processor-intensive ads, often video ads, to consume more than 3 seconds of CPU time to render a single ad within the user’s browser.” 4% of video ads were in Flash format and 19% of in-banner video ads were auto-play. https://www.adlightning.com/ad-quality-report
  57. 57. And we wonder why people use adblockers?
  58. 58. https://www.iab.com/newadportfolio/ Advertisers are trying to improve quality
  59. 59. Performance APIs can help to build a picture (within limits) Resource Timing • What was requested and when, how long did it take to fetch, was it cached? LongTasks • Were there any script elements that took longer that 50ms? But there are still plenty of blind spots
  60. 60. Can some of the security features of the web help? Tried using • Content-Security-Policy • Subresource Integrity to help a client police ads last year but didn’t have much success
  61. 61. TagManagers… A blessing or a curse? https://www.flickr.com/photos/jdhancock/9004372146
  62. 62. Makes it really easy to add new tags https://www.flickr.com/photos/lantzilla/29842854
  63. 63. Makes it easy to remove them too! https://www.flickr.com/photos/alesimages/4215559895
  64. 64. Provides a shared source of truth and audit trail https://www.flickr.com/photos/tim_ellis/3919090513
  65. 65. Help control loading order https://www.flickr.com/photos/rusty_clark/32889333561
  66. 66. Ultimately it’s about finding the balance https://www.flickr.com/photos/gilgamesh/183037111
  67. 67. VALUE COST EVERYTHING SHOULD HAVE VALUE, BECAUSE EVERYTHING HAS A COST Tim Kadlec
  68. 68. Understand the COST Establish the VALUEVALUE COST
  69. 69. https://www.flickr.com/photos/basheertome/4762529213
  70. 70. Thanks! @AndyDavies www.slideshare.net/andydavies

×