The document discusses how to run SEO experiments to test changes on websites. It recommends bucketing pages into control and treatment groups to test elements like title tags, meta descriptions, and content. It provides steps for designing an experiment, waiting 2-4 weeks for results, analyzing traffic differences between groups, concluding if results are significant, and iterating on new experiments. It also includes an example of testing alternative title tags on a company's community pages.

1. Your Name | @Twitterhandle | #TechSEOBoost
SEO Experimentation
Brian Ta
Product Manager,
Coinbase

2. Brian Ta | @fanfavorite_bta | #TechSEOBoost
How to run SEO
experiments

3. Brian Ta | @fanfavorite_bta | #TechSEOBoost
About me

4. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Strava
Product Manager, Acquisition
Airbnb
SEO Lead
AngelList
Product Manager, Growth

5. Brian Ta | @fanfavorite_bta | #TechSEOBoost
what is SEO
experimentation?

6. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Assumptions:
1. You’re familiar with SEO
2. Organic traffic is already one of the
main drivers of your business
3. You have thousands of pages

7. Brian Ta | @fanfavorite_bta | #TechSEOBoost
it’s like any
normal a/b test
you’d run, but
cooler(and
different)
what can you test?
anything your heart
desires*
title tags
meta descriptions
image alt tags
content
any on-page element

8. Brian Ta | @fanfavorite_bta | #TechSEOBoost
it allows you to know,
with certainty, how
effective your changes
are

9. Brian Ta | @fanfavorite_bta | #TechSEOBoost
how is it useful?

10. Brian Ta | @fanfavorite_bta | #TechSEOBoost
it takes the
guesswork out
of SEO

11. Brian Ta | @fanfavorite_bta | #TechSEOBoost
You get to
learn and test
the boundaries
of SEO safely

12. Brian Ta | @fanfavorite_bta | #TechSEOBoost
how many of you knew this?
Airbnb SEO team has
known since 2017
this is a
competitive
advantage

13. Brian Ta | @fanfavorite_bta | #TechSEOBoost
when is it useful?

14. Brian Ta | @fanfavorite_bta | #TechSEOBoost
1. When you have something to
learn/test
2. If leadership wants you to
prove out your wins

15. Brian Ta | @fanfavorite_bta | #TechSEOBoost
experiment if
you have
something to
learn
Don’t test for the sake of testing
Ship often. Ship fast.
Run experiments as they’re needed,
not because you can.
Embody a growth mindset.
If you’re not learning, you’re failing.

16. Brian Ta | @fanfavorite_bta | #TechSEOBoost
opportunity
optimization
know the difference

17. Brian Ta | @fanfavorite_bta | #TechSEOBoost
How do we build
it?

18. Brian Ta | @fanfavorite_bta | #TechSEOBoost
By yourself. In-house.
→ The large majority of SEO experimentation
platforms use Javascript to do bucketing. 👎
→ Non-SEO specific experimentation platforms
don’t let you bucket by page.
→ The ones that don’t ask you to route through
their own CDN 👎

19. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Regular A/B Tests
→ They’re bucketed on the user level
→ Users will get the same experience on
the entire site

20. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Regular A/B Tests
control treatment
user
www.example.com/category/{pdp}

21. Brian Ta | @fanfavorite_bta | #TechSEOBoost
this obviously
doesn’t work
for seo
why doesn’t it?
Inconsistent
experience for Google
it’s not a consistent experience for
Google
If Google hits the same page multiple times, it’s going to get bucketed into control
or treatment, and will get an inconsistent experience.
experimentation platforms leverage
JS/cookies to do bucketing
This is not ideal for Google, as Google doesn’t execute JS everytime it crawls a
page and it doesn’t persist cookies across sessions

22. Brian Ta | @fanfavorite_bta | #TechSEOBoost
SEO A/B Tests
→ They’re bucketed on the page level
→ Users will get a different experience
based on the page they’re on
→ Google will get a consistent experience

23. Brian Ta | @fanfavorite_bta | #TechSEOBoost
SEO A/B Tests
control treatment
Googlebot
www.example.com/category/roses www.example.com/category/lilies

24. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Let’s get tactical

25. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Prerequisites
→ You’ll need the ability to bucket your
control and treatment on the page level
→ You’ll need the ability to track incoming
organic traffic to your site
→ You’ll need a decent amount of traffic to
your set of pages. ~5k organic traffic a day

26. Brian Ta | @fanfavorite_bta | #TechSEOBoost
bucketing your
control and
treatment on
the page level
how do I do this?
Programmatically or
manually
Programatically
Take the hash of the URL to decide whether it goes in control or treatment
Manually
If your site is small enough, you can manually define it yourself. Try your best to
make sure that organic traffic to both buckets is roughly the same

27. Brian Ta | @fanfavorite_bta | #TechSEOBoost
tracking
incoming
organic traffic
how do I do this?
Pulling from as close
to the data source as
possible
Don’t use GA
It’s sampled. I guess it’ll work in a pinch, but I wouldn’t trust GA data.
How’d we do it elsewhere?
Airbnb has a visitors table that logged every visitor and how they came into the
website. That’s what we used. Same with Strava

28. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Step 1: Design and Launch
→ Design, implement, and launch your
experiment.
→ Run one experiment at a time to start
→ Roll out your experiment to your entire
site

29. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Step 2: Wait
→ Let it run between 2-4 weeks
→ SEO experiments take longer than
normal A/B tests to show impact

30. Brian Ta | @fanfavorite_bta | #TechSEOBoost
seriously. wait.
why?
that’s just the nature of
running experiments. you
need time for the results to
settle.

31. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Step 3: Analysis
→ We’re going to look at the difference in
expected difference of traffic.
→ Compare the difference in traffic pre-
experiment, and the difference in traffic
post-experiment

32. Brian Ta | @fanfavorite_bta | #TechSEOBoost
measuring
difference in
difference
how do I do this?
Look at the difference
in traffic before
launch, and difference
after launch

33. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Step 4: Conclude
→ You can conclude if the experiment has
run for more than two weeks, and if your
results are statistically significant*
*if you have someone to tell you that. If not, just launch it if it looks good

34. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Let’s design an
experiment

35. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Setting the stage
We’re a Growth PM at Figma on the
Community team, and we want to grow
the organic traffic coming into the
Community pages

36. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Current title tag: {name of file | description of file that
gets truncated}
<title>Figma - iOS & iPadOS 14 UI Kit for Figma |
<p>Excited to share the latest iOS &amp; iPadOS 14 UI Kit
for Figma!</p><p><br></p><p>---</p><p><...</title>
Step 1: Designing the experiment

37. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Title tag to test: {name of file | tags}
<title>Figma - iOS & iPadOS 14 UI Kit for Figma | 14,
alert, apple, dark, emoji, ios, iphone, kit, light</title>
Step 1: Designing the experiment cont’d

38. Brian Ta | @fanfavorite_bta | #TechSEOBoost
We’ll do a straight A/B test with 50/50
treatment versus control to ALL
Community File pages.
Step 2: Rollout

39. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Title tag experiments are usually impactful. I’d
probably wait two weeks before running an
analysis.
Step 3: Wait

40. Brian Ta | @fanfavorite_bta | #TechSEOBoost
We’ll look at the expected difference in traffic if
you hadn’t run the experiment (control group)
and compare it against the difference that you
see now that you’ve made the change
(treatment). I’ll work with a data scientist to
make sure that we’re pulling data from our
visitors table.
Step 4: Analysis

41. Brian Ta | @fanfavorite_bta | #TechSEOBoost
If results are positive and are statistically
significant, I’d launch to 100%. If it’s neutral, or
inconclusive after a month, you can use your
own judgment on whether or not you want to
launch it.
Step 5: Launch

42. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Document your learnings, and start thinking about
other experiments you can do.
Step 6: Iterate

43. Brian Ta | @fanfavorite_bta | #TechSEOBoost
things to keep in
mind

44. Brian Ta | @fanfavorite_bta | #TechSEOBoost
impact of experiments
will decay. +5%
increase in traffic won’t
last forever

45. Brian Ta | @fanfavorite_bta | #TechSEOBoost
don’t be afraid to ship
neutral experiments.
experiments are to
inform your decision.

46. Brian Ta | @fanfavorite_bta | #TechSEOBoost
experimentation should
not be slowing you
down

47. Brian Ta | @fanfavorite_bta | #TechSEOBoost
have a good
growth/experiment
culture. ideate. iterate.
test. launch.

48. Brian Ta | @fanfavorite_bta | #TechSEOBoost
The Growth Process
Bottoms
Up Ideation
Grouping &
Prioritizing
Experiment
Scoping
Opportunity Scoping
Experimentation & Learning
Rapid
Prototyping
Experiment
Execution
Insights &
Sharing
Research
& Analysis

49. Brian Ta | @fanfavorite_bta | #TechSEOBoost
Thanks!
@fanfavorite_bta