Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How BiggerPockets runs experiments faster with performance edge

209 views

Published on

Whelan Boyd, Product Manager at Optimizely will be joining Alex Wald and Mike McKee from BiggerPockets’s Growth and Engineering teams to share how using Performance Edge by Optimizely can help teams scale experimentation with minimal performance impact to drive better conversion rates and less abandonment.

In this webinar, you’ll learn:

- Best practices for measuring website performance
- Ways to experiment client-side without affecting site performance
- How BiggerPockets saw improved site performance by adopting Performance Edge

Published in: Business
  • Be the first to comment

  • Be the first to like this

How BiggerPockets runs experiments faster with performance edge

  1. 1. 1 How BiggerPockets runs experiments faster with Performance Edge + Whelan Boyd, Alex Wald, Mike McKee
  2. 2. 2 Introductions Alex Wald Growth Hacker Mike McKee Production Support Engineer Whelan Boyd Sr. Product Manager
  3. 3. 3 Housekeeping ● We are recording this webinar ● You will be emailed the slides and recording ● Ask your questions via the GoToWebinar questions box ● We’ll answer all questions at end
  4. 4. 4 Agenda 1. Why performance matters 2. How to think about measuring performance 3. How experimentation affects performance 4. Performance Edge 5. Case study: BiggerPockets 6. Q&A
  5. 5. 5 Why Performance Matters
  6. 6. 6 The business impact of speed 13% increase in sales when page load time was cut by half Found that 53% of mobile visits were abandoned if page took > 3 seconds to load 7% increase in conversion when page load time reduced by 850ms Source: https://developers.google.com/web/fundamentals/performance/why-performance-matters/
  7. 7. 7 How to measure performance
  8. 8. 8 Measure impact to user experience Source: Google’s Web Fundamentals
  9. 9. 9 Best practices for measuring Free Online Tools ■ Synthetic Traffic ■ Easy to run ■ Actionable Insights Real User Monitoring ■ Production Traffic ■ Segment by Device, Location, etc ■ Robust Analytics
  10. 10. 10 How Website Experimentation Affects Performance
  11. 11. 11 Standard Web Experiments Experiment A variation 1: CTA: Get Started variation 2: CTA: Try it out Experiment B variation 1: Promo Code variation 2: No promo code Standard CDN URL Targeting Audience Targeting Traffic Splitting Apply Variations Event Tracking Task load: 1. Snippet size increases with more experiments 1. For specific visitors, the downloaded snippet includes all project variations vs just active variations 1. Larger snippet results in more time to download and execute 1. Resource-constrained browsers have to execute several tasks site.com/home script Performance Challenges for Web Experiments
  12. 12. 12 Introducing Performance Edge
  13. 13. 13 CTA: Get Started Only returns visitor-specific variations Standard Web Experiments Experiment A variation 1: CTA: Get Started variation 2: CTA: Try it out Experiment B variation 1: Promo Code variation 2: No promo code URL Targeting Audience Targeting Traffic Splitting Apply Variations Event Tracking Task load: URL Targeting Audience Targeting Traffic Splitting Experiment Delivery Network Apply Variations Event Tracking Key experiment decisions are moved to the Edge Much smaller file to download and execute in the browser with less tasks to perform site.com/home script Standard CDN site.com/home script Performance Edge Experiments
  14. 14. 14 Why is it so fast? Less data Less code Faster machines ▪ Far smaller file loading over the wire ▪ Reduce download time ▪ Only code necessary for active variations ▪ Fraction of the execution time in browser ▪ Powerful Cloudflare Workers execute expensive tasks ▪ Located extremely close to end visitor
  15. 15. 15 Proper Measurement Make things smaller Performance Edge Summary ▪ User-focused Metrics ▪ Synthetic and real traffic ▪ Lighter download ▪ Shorter execution ▪ Quick implementation with any CDN ▪ Push logic to the Edge
  16. 16. 16 BiggerPockets Case Study
  17. 17. 17 BiggerPockets: Intro Alex Wald Growth Hacker Mike McKee Production Support Engineer
  18. 18. 18 BiggerPockets: Overview Company Size: 20-50 employees Location: Denver Mission: Help a million people become millionaires through real estate investing What we offer: Online resources for real estate investors (and aspiring real estate investors!) Education Interaction Investing Tools
  19. 19. 19 BiggerPockets: Testing Overview BiggerPockets User Life Cycle Testing focus for 2019: Drive free signup conversion rate from 1.2% to 2% Guest Visits website or listens to podcast Free Site Member User creates account on site Pro Site Member User upgrades to Pro Repeat customer User renews account after 1 year
  20. 20. 20 BiggerPockets: History w/Optimizely Minimal testing, not tracking our efforts Early 2018: BiggerPockets starts minimal copy testing with Google Optimize Free 2019: Partner with Optimizely, start ramping up testing velocity significantly and running more complicated A/B & MVT tests Late 2018: Dedicate more resources to testing, start running more complicated tests
  21. 21. 21 BiggerPockets: Approach to driving free membership sign ups Ways we can drive free membership sign ups Make it easy to sign up Make it more attractive Communicate value of free membership more effectively Add value to free membership (not pursuing) Most successful strategy
  22. 22. 22 Test: Reverse onboarding Control Challenger
  23. 23. 23 Control Winner: Challenger +8.2% Lift in Sign-up Rate (99% Stat Sig) +5% Lift in Sign-up Rate (Desktop) +13% Lift in Sign-up Rate (Mobile) Key takeaway: Users are more likely to sign up for a product when they feel like they are getting a personalized experience. Asking users questions about themselves that pertain to the product is a great way to do so. Test Results: Reverse onboarding
  24. 24. 24 Example test: Reverse onboarding 2 Control Test Normal Onboarding Signup w/email only Collect just email and then ask for name/passw ord at end Sign up w/email, pass & name Normal Onboarding
  25. 25. 25 Test Results: Reverse Onboarding 2 Control Winner: Challenger +4.9% Lift in Sign-up Rate (98% Stat Sig) +6% Lift in Sign-up Rate (Desktop) +4% Lift in Sign-up Rate (Mobile) Key takeaway: Users are more willing to give us their personal information if we get our foot in the door by asking questions about their investing strategy first
  26. 26. 26 Why we needed performance edge as we scaled Scaling testing drives business results ■ 100 experiments first 9 months ■ Improved conversion rate by 72% in 9 months ■ Doubled our testing velocity from 5 tests/month to 11 tests/month ■ Positive impacts down the business funnel BiggerPockets invests in Optimizely Downside: A growing point of internal tension… ■ More testing = slower page speed ■ More emphasis placed on performance at BiggerPockets
  27. 27. 27 How we implemented Performance Edge at BiggerPockets 1. Created a Worker on Cloudflare to connect with Optimizely Edge endpoint (just a few minutes) 1. Configured Worker with boilerplate code and added 1st party script tag and experiments starting showing on our staging server 1. Adjusted boilerplate CDN code to allow-list only cookies relevant to Optimizely 1. Integrated with Amplitude to send experiments metrics
  28. 28. 28 Speed improvements with Performance Edge Using Lighthouse, our Speed Index went from 4.2 seconds to 3.0 seconds. In our APM, New Relic, average front-end load times improved from 4.6 seconds to 3.9 seconds. The Edge snippet returned is significantly smaller in size than the traditional Web snippet. Performance Edge was a huge boost towards our goal of 3.5s load time by EOY!
  29. 29. 29 Cost of implementation: minimal ■ Minimal dev time ■ Existing experiments carry over from Optimizely Web to Optimizely Edge Hiccups in test velocity : none Trade-offs: minor w/work arounds ■ No preview mode > use QS parameters ■ Not loading Jquery > use Javascript (unless your whole site uses Jquery) ■ Some other minor feature limitations, consult with Optimizely Easy implementation leads to immediate benefits Our Costs of implementation Our Immediate Benefits -29% reduction in speed index Solves internal conflict -- testing no longer slows down site in a meaningful way Anticipating improvements in SEO rankings Most importantly: we can continue scaling our CRO program without slowing down our site!
  30. 30. 30 BiggerPockets: Testing Overview Aggressive Testing Across the Funnel Guest Visits website or listens to podcast Free Site Member User creates account on site Pro Site Member User upgrades to Pro Repeat customer User renews account after 1 year Testing focus for Q4 & 2020: Continue to drive free sign ups and expand testing to our free- to-paid conversion rate & paid member retention
  31. 31. 31 Questions? Ask them in Questions box in GotoWebinar

×