SEO and SEM are over 20 years old, and lots has changed over the years. But the fundamental KPIs, analytical practices, and basic economics of Search Marketing are all the same. Learn/relearn some of the oldest tricks and maybe some new ones as we take a mathematical tour through a handful or two of entertaining and useful search scenarios. Originally presented at the Seattle Search Meetup on January 10, 2018

1. Back to SEO/SEM Basics
Search Marketing Math for Fun and Profit

2. David Langrock
● Search Marketing Leader
10+ years @
○ Amazon
○ Nordstrom
○ Groupon
● Software Engineer/Manager
25+ years
● Math / CS background
@davidlangrock
dlangrock@groupon.com

3. Agenda
● SEM Bidding Basics
● SEM Portfolio Optimization
● Increasing SEO Visibility
● How To Build Visibility Metrics
● Test Setup / Statistical Significance

4. SEM Bidding
Basics

5. SEM is Arbitrage - Not a Competition
● Bidding for position is almost
never right.
● Estimate/Evaluate the value of
the traffic and bid that
CPC
3.50
3.20
2.95
2.81
3.00

6. What Should go into a Bid, and What Shouldn’t
● Beware Google claims around the value of Brand Impressions
● Beware Category Managers and Marketing VPs that want to spend
“strategically”
● You Should Demand Data & Commit to Measuring
Spend Your Bid
Value (Easy) Transactional Revenue / Click +
(Hard) Customer Acquisition Value / Click +
(Beware) Strategic Value / Click +
(Beware) Branding Value / Impression / CTR +
< = 1=

7. Pos 1 Bidding: SEM inefficiency as CVR Problem
● Booking.com bids to 1st
Position for top keywords
● When ROAS goes below 1,
Sales/Product Teams get
tickets, not SEM team.
● Rare Case:
○ Hotels is a Narrow Vertical
○ Booking.com regularly has 75+%
of the Inventory

8. SEM Portfolio
Optimization

9. Tenets of Portfolio Optimization
● With sufficient data
○ A keyword lives or dies on its own
○ Does not subsidize other keywords
● Keyword Groups follow the same rules as Keywords
● Paying to Learn must be measured / monitored

10. Keyword Bidding Lifecycle
0 Clicks 1 to N clicks N+ clicks
Base Bid Algo ● Default Bid
● Predictive Bid by
Segment
● First Page Bid for
Keyword
● Performance
Bidding by
Segment
● Individual Bidding
Adjustments ● Sitewide Conversion
● Segment Conversion
● Time of Day / Day of Week
● Geotarget Adjust
● Device Type Adjust

11. Segmentation Drives Better Efficiency, Value
200
100
100

12. Bid Efficiency Curve and Marginal Efficiency
● Green: Value(Spend)
● Bid A: Breakeven -
Value ($84) = $84
● Bid B: Marginal
Breakeven -
Value ($19) = $58
● Inefficient to spend
more than $19: Spend
$65 to get $26
A
B

13. Increasing SEO
Visibility

14. Visibility - The Likelihood a User will find your property through SEO
Increasing Visibility
A. B.

15. Measuring Visibility
Visibility is a way to estimate the number of clicks a given
keyword will have.
Visibilitykw = MSVkw * Expected Click Thru Rate (Rankkw)

16. Example: Measuring Thrillist Things To Do SEO
● Rank Buckets
● Average Rank
● Weighted
Average Rank
● Visibility

17. Thrillist.com Data Set
● Keyword rank data for all keywords
○ matching “things to do in *”
○ Thrillist.com URL ranks 1-20
○ 3 Data points taken: Jan 2016, Jan 2017, Jan 2018
● Source: SEMRush - Details at the end
● SEO Keyword CTR pr Rank Model based on
https://moz.com/blog/google-organic-click-through-rates-in-
2014

18. Thrillist “Things To Do In” Rank Buckets
Bucket 2016 2017 2018
1-3 235 363 692
4-6 425 704 1165
7-10 441 1141 1352
11-20 838 1771 1953
Totals 1939 3979 5162

19. Thrillist “Things To Do In” Average Rank
2016 2017 2018
Num
Keywords
1939 3979 5162
Average
Rank
9.7 10.0 9.2

20. Thrillist “Things To Do In” Weighted Avg.
Rank
2016 2017 2018
Num
Keywords
1939 3979 5162
Average
Rank
8.4 9.9 7.9

21. Thrillist “Things To Do In” Overall Visibility
Bucket 2016 2017 2018
1-3 16458 7880 17180
4-6 30956 38096 51865
7-10 45303 76566 93661
11-20 20004 54983 24008
Total 112721 177524 186714

22. When/How to use Target Keyword Lists
● New SEO Project to improve a set of pages / keywords
○ Content Building Project
○ Link Building Project
● Index / Basket of Keywords
○ Head Keywords
○ Category Based
● Keep MSV Constant throughout
○ Corrects for Seasonality
○ Measures SEO not keyword trends

23. Thrillist Example of Target KW List
● Post hoc (take it with a grain of salt)
● Criteria:
○ KWS ranked in top 20 all three years
○ Chose MAX MSV over three years
○ >= 1000 MSV
● 155 keywords

24. Thrillist KW Basket Rank Buckets
Bucket 2016 2017 2018
1-3 18 10 33
4-6 45 45 53
7-10 44 68 72
11-20 58 47 17
Totals 165 170 175

25. Thrillist KW Basket Average Rank
2016 2017 2018
Num
Keywords
165 170 175
Average
Rank
8.9 8.6 6.6

26. Thrillist KW Basket Weighted Avg. Rank
2016 2017 2018
Num
Keywords
165 170 175
Average
Rank
7.8 9.4 7.5

27. Thrillist KW Basket Overall Visibility
Bucket 2016 2017 2018
1-3 17793 4237 13344
4-6 25976 23345 30292
7-10 15402 18585 30076
11-20 6244 14291 3317
Total 65415 60457 77029

28. How to Build
Visibility Metrics
using SEMRush and Excel

29. SEMRush Plans

30. Collect The Data Using SEMRush
1. Domain
Overview:
thrillist.com
2. Find Top Organic
Keywords and
click “view full
report”

31. SEMRush Continued
1. Click Advanced
Filters to select
your criteria
2. Click Export,
Choose your
criteria, click Save
to Excel.

32. Visibility Excel

33. Visibility Excel (Cont’d)
● SUMIFS and
AVERAGEIFS are
your friends
● Also use INDEX
to find CTR’s
from the CTR
table
● 100k’s of
VLOOKUPs is
very slow...

34. Running Tests

35. Most Failed SEO Question: How would you
test and validate your project?
● Where Candidates Go Wrong:
○ Test Setup - Not Random or Validated for Power / p-Value
○ Statistical Significance - No Mention of it

36. SEO Experiment Setup
● If your experiment intends to improve CX on the page, then
any standard A/B test suite can work
○ splits the customer base into two pools automatically
○ Control sees the current UX, Experiment sees the new UX
● If your experiment intends to improve rank and traffic, then
you must only treat some pages and not the others
○ Selection of the pages to treat should be RANDOM, ideally

37. SEO Rank/Traffic Experiment Setup
● Estimate how long it will take for Google to Crawl/Index
modified pages. (1 day, 2 weeks, more?)
● Estimate how much traffic it will take to reach Stat Sig
○ https://www.optimizely.com/sample-size-calculator/
○ https://abtestguide.com/calc/
● Learn what p-values and statistical power mean here:
○ http://blog.analytics-toolkit.com/2017/importance-
statistical-power-online-ab-tests/

38. Power and P-Value
Measurement of
Experiment Success
Experiment causes Hoped-for Results
FALSE
(your experiment doesn’t work)
TRUE
(your experiment works)
TRUE
(your measurements tell
you that your experiment
works)
Incorrectly Showed that the
Experiment Succeeded (p-value)
Correctly Showed that
Experiment Succeeded!
FALSE
(your measurements tell
you that your experiment
doesn’t work)
Correctly Showed that
Experiment Failed
Incorrectly Showed that the
Experiment Failed
(Power/Sensitivity)

39. What a StatSig test doesn’t look like:

40. What a StatSig Test Looks Like

41. Q & A