Whelan Boyd, Product Manager at Optimizely will be joining Alex Wald and Mike McKee from BiggerPockets’s Growth and Engineering teams to share how using Performance Edge by Optimizely can help teams scale experimentation with minimal performance impact to drive better conversion rates and less abandonment. In this webinar, you’ll learn: - Best practices for measuring website performance - Ways to experiment client-side without affecting site performance - How BiggerPockets saw improved site performance by adopting Performance Edge

1. 1
How BiggerPockets runs
experiments faster with
Performance Edge
+
Whelan Boyd, Alex Wald, Mike McKee

2. 2
Introductions
Alex Wald
Growth Hacker
Mike McKee
Production Support Engineer
Whelan Boyd
Sr. Product Manager

3. 3
Housekeeping
● We are recording this webinar
● You will be emailed the slides and
recording
● Ask your questions via the
GoToWebinar questions box
● We’ll answer all questions at end

4. 4
Agenda
1. Why performance matters
2. How to think about measuring performance
3. How experimentation affects performance
4. Performance Edge
5. Case study: BiggerPockets
6. Q&A

5. 5
Why
Performance
Matters

6. 6
The business impact of speed
13% increase in sales
when page load time
was cut by half
Found that 53% of mobile
visits were abandoned
if page took > 3 seconds
to load
7% increase in
conversion when page
load time reduced by
850ms
Source: https://developers.google.com/web/fundamentals/performance/why-performance-matters/

7. 7
How to
measure
performance

8. 8
Measure impact to user experience
Source: Google’s Web Fundamentals

9. 9
Best practices for measuring
Free Online Tools
■ Synthetic Traffic
■ Easy to run
■ Actionable Insights
Real User Monitoring
■ Production Traffic
■ Segment by Device, Location, etc
■ Robust Analytics

10. 10
How Website
Experimentation
Affects Performance

11. 11
Standard Web Experiments
Experiment A
variation 1: CTA: Get Started
variation 2: CTA: Try it out
Experiment B
variation 1: Promo Code
variation 2: No promo code
Standard
CDN
URL Targeting
Audience Targeting
Traffic Splitting
Apply Variations
Event Tracking
Task load:
1. Snippet size increases with more
experiments
1. For specific visitors, the
downloaded snippet includes all
project variations vs just active
variations
1. Larger snippet results in more time
to download and execute
1. Resource-constrained browsers
have to execute several tasks
site.com/home
script
Performance Challenges
for Web Experiments

12. 12
Introducing
Performance
Edge

13. 13
CTA: Get Started
Only returns visitor-specific
variations
Standard Web Experiments
Experiment A
variation 1: CTA: Get Started
variation 2: CTA: Try it out
Experiment B
variation 1: Promo Code
variation 2: No promo code
URL Targeting
Audience Targeting
Traffic Splitting
Apply Variations
Event Tracking
Task load:
URL Targeting
Audience Targeting
Traffic Splitting
Experiment
Delivery
Network
Apply Variations
Event Tracking
Key experiment decisions are
moved to the Edge
Much smaller file to download
and execute in the browser
with less tasks to perform
site.com/home
script
Standard
CDN
site.com/home
script
Performance Edge Experiments

14. 14
Why is it so fast?
Less data Less code Faster machines
▪ Far smaller file loading
over the wire
▪ Reduce download time
▪ Only code necessary for
active variations
▪ Fraction of the execution
time in browser
▪ Powerful Cloudflare
Workers execute
expensive tasks
▪ Located extremely close
to end visitor

15. 15
Proper Measurement Make things smaller Performance Edge
Summary
▪ User-focused Metrics
▪ Synthetic and real traffic
▪ Lighter download
▪ Shorter execution
▪ Quick implementation with
any CDN
▪ Push logic to the Edge

16. 16
BiggerPockets
Case Study

17. 17
BiggerPockets: Intro
Alex Wald
Growth Hacker
Mike McKee
Production Support Engineer

18. 18
BiggerPockets: Overview
Company Size: 20-50 employees
Location: Denver
Mission: Help a million people become millionaires through
real estate investing
What we offer: Online resources for real estate investors (and
aspiring real estate investors!)
Education Interaction Investing
Tools

19. 19
BiggerPockets: Testing Overview
BiggerPockets User Life Cycle
Testing focus for 2019: Drive
free signup conversion rate from
1.2% to 2%
Guest
Visits website or listens to podcast
Free Site Member
User creates account on site
Pro Site Member
User upgrades to Pro
Repeat
customer
User renews
account
after 1
year

20. 20
BiggerPockets: History w/Optimizely
Minimal testing, not
tracking our efforts
Early 2018: BiggerPockets
starts minimal copy
testing with Google
Optimize Free
2019: Partner with Optimizely, start
ramping up testing velocity significantly and
running more complicated A/B & MVT tests
Late 2018: Dedicate
more resources to
testing, start running
more complicated
tests

21. 21
BiggerPockets: Approach to driving free membership sign ups
Ways we can drive free
membership sign ups
Make it easy to
sign up
Make it more
attractive
Communicate value
of free membership
more effectively
Add value to free
membership (not
pursuing)
Most
successful
strategy

22. 22
Test: Reverse onboarding
Control Challenger

23. 23
Control Winner: Challenger
+8.2%
Lift in Sign-up Rate
(99% Stat Sig)
+5%
Lift in Sign-up Rate
(Desktop)
+13%
Lift in Sign-up Rate
(Mobile)
Key takeaway: Users are more likely to sign up for a product when they feel like they are getting a personalized
experience. Asking users questions about themselves that pertain to the product is a great way to do so.
Test Results: Reverse onboarding

24. 24
Example test: Reverse onboarding 2
Control
Test
Normal
Onboarding
Signup w/email only
Collect just
email and
then ask for
name/passw
ord at end
Sign up
w/email, pass
& name
Normal
Onboarding

25. 25
Test Results: Reverse Onboarding 2
Control Winner: Challenger
+4.9%
Lift in Sign-up Rate
(98% Stat Sig)
+6%
Lift in Sign-up Rate
(Desktop)
+4%
Lift in Sign-up Rate
(Mobile)
Key takeaway: Users are more willing to give us their personal information if we get our foot in the door by asking
questions about their investing strategy first

26. 26
Why we needed performance edge as we scaled
Scaling testing drives business results
■ 100 experiments first 9 months
■ Improved conversion rate by 72% in 9 months
■ Doubled our testing velocity from 5 tests/month to
11 tests/month
■ Positive impacts down the business funnel
BiggerPockets invests in Optimizely
Downside: A growing point of internal tension…
■ More testing = slower page speed
■ More emphasis placed on performance at
BiggerPockets

27. 27
How we implemented Performance Edge at
BiggerPockets
1. Created a Worker on Cloudflare to connect
with Optimizely Edge endpoint (just a few
minutes)
1. Configured Worker with boilerplate code
and added 1st party script tag and
experiments starting showing on our
staging server
1. Adjusted boilerplate CDN code to allow-list
only cookies relevant to Optimizely
1. Integrated with Amplitude to send
experiments metrics

28. 28
Speed improvements with Performance Edge
Using Lighthouse, our Speed Index went
from 4.2 seconds to 3.0 seconds.
In our APM, New Relic, average front-end
load times improved from 4.6 seconds to
3.9 seconds.
The Edge snippet returned is significantly
smaller in size than the traditional Web
snippet.
Performance Edge was a huge boost towards our goal of 3.5s load time by EOY!

29. 29
Cost of implementation: minimal
■ Minimal dev time
■ Existing experiments carry over from Optimizely
Web to Optimizely Edge
Hiccups in test velocity : none
Trade-offs: minor w/work arounds
■ No preview mode > use QS parameters
■ Not loading Jquery > use Javascript (unless your
whole site uses Jquery)
■ Some other minor feature limitations, consult with
Optimizely
Easy implementation leads to immediate benefits
Our Costs of implementation Our Immediate Benefits
-29% reduction in speed index
Solves internal conflict -- testing no longer slows
down site in a meaningful way
Anticipating improvements in SEO rankings
Most importantly: we can continue scaling our CRO program without slowing down our site!

30. 30
BiggerPockets: Testing Overview
Aggressive Testing Across the Funnel
Guest
Visits website or listens to podcast
Free Site Member
User creates account on site
Pro Site Member
User upgrades to Pro
Repeat
customer
User renews
account
after 1
year
Testing focus for Q4 & 2020:
Continue to drive free sign ups
and expand testing to our free-
to-paid conversion rate & paid
member retention

31. 31
Questions?
Ask them in Questions box in
GotoWebinar