The document provides an overview of testing automation for large paid search (PPC) campaigns. It discusses: - The challenges of testing PPC automation approaches due to large campaign sizes and volumes. - Two alternative testing methods: splitting traffic between variants or splitting keywords, but these have limitations. - A proposed "A/B swap" method where keywords are split into two groups that are alternated between control and test variants each day, allowing tests to conclude faster. - How the testing approach enables large-scale PPC automation by providing a means to rigorously evaluate new bidding and optimization strategies on campaigns spending $200M+.

2. From Incremental to Exponential Mindset

3. The Evolution of an
Exponentially Growing Company
Kobi Salinas
Consumer Product Manager
Speeding up A/B Testing With the
Bayesian Approach
Omri Walton
Data Product Manager
How to Test Automation on
>$200M Campaigns
Matar Bar-Sheshet
Data Product Manager

4. Sooo...

5. Kobi Salinas, Consumer Product Manager
Who Am I?

6. The First Comparison Table
One chart
3 employees
PPC traffic
10 Brands

7. 10 Years Later...

8. 10 Years Later in Numbers
100K+
Monthly Purchase
Decisions Generated
10M+
Monthly
Unique Visitors
500+
Top Tier
Brands
300+
Sites
10
Years of
Expertise
400+
Talented
Employees

9. TOP 10
Mortgage Loans

10. THE ANATOMY OF A
TOP 10
SHORTLIST

11. What Kind of Traffic do We Deliver?
SEO/Direct

12. What Is AB Testing?

13. Innocent Drinks Story, 1998
Started at a music festival in London
Selling their shakes
Placing 2 bins...

14. Innocent Drinks Story, 1998
Started at a music festival in London
Selling their shakes
Placing 2 bins...

15. What’s the Difference?

16. 38% Increase in sales (!)
What’s the Difference?

17. How Do We Do It at NI?

18. Multi-Sites Testing
Bigger impact as
more users exposed
to the test
Faster decision Enhance our
Top10.com brand -
one look and feel
Why?

19. Example:
Chocolate goes great with wine, selling chocolate for people who love wine will increase
total revenues by 5% after 2 weeks
Every Test Starts with a Hypothesis
Hypothesis = [fact/the change] + [who you expect it to impact] + [how much] + [after how long]

20. How Do We Create Hypotheses?

21. Intuition
Competitors
Research
Data
How Do We Create Hypotheses?

22. ● Research - the jam experience - when less choice
means more sales
● If customers ask for more choice, don’t listen
Hypothesis - Research Inspiration
Increase our sales by 6.5%

23. Facts that leads us to a clear and clean design :
● Search more on mobile
● Buy less
● Bounce fast (~50% Bounce)
● Less time on mobile
Hypothesis - Research & Data Inspiration

24. Hypothesis - Research & Data Inspiration

25. A - Controlled B - Tested
Let’s Play a Crowd Sourcing Game

26. The Power of Trust
Increase our sales in 10%
A - Controlled B - Tested

27. A - Controlled B - Tested
The Power of Reviews

28. Let’s Play a Crowd Sourcing Game
Increase our sales in 3%
A - Controlled B - Tested

29. Use the KW - implement it on the H1, on the card, change images by food type, etc. personalization is everything
Increase our sales in 5%
How We Use Our Knowledge to Optimize Sites

30. A - Controlled
B - Tested

31. Significance level
Time
Interactions
Data Team! We Have a Problem!

32. ● Everything starts with a right date (or dating site)
● Multi-sites optimization as engine for growth
● Every AB test starts with hypothesis
● We need to move fast, run multi-tests, and get significant
● Data Data Data
Recap

33. Omri Walton, Data Product Manager
Who Am I?

34. 2 Statistical Approaches
vs.

35. Cookies

36. The Cookie Box Challenge (Classical vs. Bayesian)
A B
Which box has more pistachio cookies?
-100 cookies each
-Different kinds
-All look the same

37. The Cookie Box Challenge
A BClassical
Define Sample Size
Eat 20 cookies
from each box
Start Eat 40 Cookies Check Results
???
Not
Conclusive?
Great!
Conclusive
?

38. The Cookie Box Challenge
A BBayesian
Define Priors
Are the
chances really
equal?
Start Eat a Cookie Check Results
Not
Conclusive?
Great!
Conclusive
?
50%50% 60%40%

39. Same Goes for A/B Testing
Classical Define Sample Size
“Time Estimation”
Start Wait Check Results
Great!
Conclusive
?
Don’t pick at
the results!
Bayesian
Define Priors
(Optional)
Start Check Results
Not
Conclusive?
Great!
Conclusive
?
We have
options!
No waiting
required
Not
Conclusive?

40. How Is It Working For Us?
Too long
1
Can’t stop in the middle
2
Inconclusive
3
No tool to deal with inconclusive
4
Time to significance by site:

41. Bayesian - Proof-of-Concept
Same Bottom Line
50% Less Time!
No Major Risk
Analyzed
~400 AB tests
on both methods

42. So what can we do
about inconclusive
tests?

43. Don’t Just Test - Segment!

44. Winning Chances:

45. Bayesian - Summary
Enabler for running x2 more A/B tests
Empowers you to deal with inconclusive tests
Enabler for real-time optimization
using Multi-arm-bandit

46. Thank You!

47. Matar Bar-Sheshet, Data Product Manager
Who Am I?

48. What Kind of Traffic do We Deliver?
SEO/Direct
Remember
This?

49. If it ain’t broken, don’t fix it.
Spending $200M+ on user acquisition from PPC
Optimizing campaigns for 300+ sites
In 40+ different domains

50. If it ain’t broken, don’t fix it AUTOMATE IT.
Spending $200M+ on user acquisition from PPC
Optimizing campaigns for 300+ sites
In 40+ different domains

51. What Automations Are We Even Talking About?
Keyword bidding
How much should we pay for each keyword?
Ads optimization
Can we think of a better ad copy? What should we do with old ads?
Keywords coverage
what other keywords should we include in our campaigns?

52. Why Automate?

53. Growth / Scale
Why not use out-of-the-box tools?
Why Automate?

54. Growth / Scale
Matching know-how
Unique use-cases
Why Automate?

55. So…
How do we test
PPC Automation?

56. We A/B Test! (But Differently!)
VariantB
Test
VariantA
Control
KEYWORD BID
Shoes 1$
Boots 2$
Sandals 5$
... ...
TEST BID
1.4$
1.8$
3.7$
...
VS

57. The Challenges of Testing (PPC)
Or, requirements for a testing method
▢
▢
▢

58. Alternatives to Testing PPC Automation
▢
▢
▢KEYWORD
CONTROL
GROUP
TEST
GROUP
Shoes
Boots
Sandals
High Heels
METHOD 1
SPLITTING TRAFFIC

59. Alternatives to Testing PPC Automation
OR
METHOD 2
SPLITTING KEYWORDS
KEYWORD
CONTROL
GROUP
TEST
GROUP
Shoes
Boots
Sandals
High Heels
METHOD 1
SPLITTING TRAFFIC
KEYWORD
Shoes
CONTROL
GROUPBoots
Sandals
TEST
GROUPHigh Heels

60. Alternatives to Testing PPC Automation
▢
▢
▢
METHOD 2
SPLITTING KEYWORDS
KEYWORD
Shoes
CONTROL
GROUPBoots
Sandals
TEST
GROUPHigh Heels

61. BACK TO THE DRAWING BOARD

62. Method 3 - A/B Swap
SPLITTING AND SWAPPING variants
○ Split your keywords into two groups.
- Optimize your test group.
- Swap control/test groups.
○ Every day:
KEYWORD SUNDAY
Shoes
Boots
Sandals
High Heels
KEYWORD MONDAY
Shoes
Boots
Sandals
High Heels
KEYWORD TUESDAY
Shoes
Boots
Sandals
High Heels
So On...

63. Method 3 - A/B Swap
Not just splitting, but SWAPPING variants
AB SWAP CAN REDUCE OUR EXPERIMENTS
DURATION BY 50%-75%!
▢
▢
▢
▢
Group A: Group B:
Tuesday
And so on...
Group A: Group B:
Monday
Group A: Group B:
Sunday

64. Great hypotheses are awesome. Just make sure you can test them!
The testing platform is an enabler for automation.
A grand project requires an unusual solution.
Takeaways

65. THANK YOU!