1. Addressing variance
in AB tests
Interleaved evaluation of rankers

2. Wikimedia
Search
Platform
● 300 languages
● 900 wikis
● 80% of search: 20 wikis
● 700M wiki pages indexed
● 1000+ autocomplete qps
● 700+ full text qps
● 6 clusters in 2 DCs
● Team of 5 engineers

3. Chapelle, Joachims, Radlinski, Yue 2012
http://olivier.chapelle.cc/pub/interleaving.pdf
Large Scale Validation
and Analysis of
Interleaved Search
Evaluation

4. Evaluation
of
Search
Quality
Public Domain, US Government

5. Offline
Evaluation
● Metrics on scale of result
set changes
● Golden Set / Expert
Judgements
● Simulated AB tests

6. Online Evaluation

7. Goals
● Blind to the user
● Robust to biases
unrelated to ranker
quality.
● Not substantially alter
the search experience
● Lead to clicks that reflect
user preference
Joachims 2003

8. Absolute
Metrics
● Clicks@N
● Max Clicked Position
● Clicks per query
● Time to First Click
● Session Abandonment
● Reformulations
● Zero Result Rate
● Interactions on clicked
pages

9. So, Variance?
How far a set of numbers are spread out from
their average value.
https://en.wikipedia.org/wiki/Variance

10. CC by SA 4.0, Zachary McCunePublic Domain, JRBrown

11. Maximum
Clicked
Position
60k per bucket

12. Maximum
Clicked
Position
850k per bucket

13. Null Test
3M per bucket

14. Why so much
Variance?

16. Interleaving,
How Does it Work?

17. Balanced
Interleaving

18. INPUT RANKING BALANCED
Rank A B A FIRST B FIRST
1 z y z (A) y (B)
2 y v y (B) z (A)
3 x z v (B) v (B)
4 w u x (A) x (A)
w (A) u (B)
u (B) w (A)

19. INPUT RANKING BALANCED
Rank A B A FIRST B FIRST
1 z y z (A) y (B)
2 y v
3 x z
4 w u

20. INPUT RANKING BALANCED
Rank A B A FIRST B FIRST
1 z y z (A) y (B)
2 y v y (B) z (A)
3 x z
4 w u

21. INPUT RANKING BALANCED
Rank A B A FIRST B FIRST
1 z y z (A) y (B)
2 y v y (B) z (A)
3 x z v (B) v (B)
4 w u

22. INPUT RANKING BALANCED
Rank A B A FIRST B FIRST
1 z y z (A) y (B)
2 y v y (B) z (A)
3 x z v (B) v (B)
4 w u x (A) x (A)

23. INPUT RANKING BALANCED
Rank A B A FIRST B FIRST
1 z y z (A) y (B)
2 y v y (B) z (A)
3 x z v (B) v (B)
4 w u x (A) x (A)
w (A) u (B)

24. INPUT RANKING BALANCED
Rank A B A FIRST B FIRST
1 z y z (A) y (B)
2 y v y (B) z (A)
3 x z v (B) v (B)
4 w u x (A) x (A)
w (A) u (B)
u (B) w (A)

25. k_a, k_b, I = 0, 0, []
A = ['a', 'b', 'c', 'd']
B = ['b', 'e', 'a', 'f']
A_first = random.choice((True, False))
while k_a < len(A) and k_b < len(B):
if k_a < k_b or (k_a == k_b and A_first):
if A[k_a] not in I:
I.append(A[k_a])
k_a += 1
else:
if B[k_b] not in I:
I.append(B[k_b])
k_b += 1

26. INPUT RANKING BALANCED
Rank A B A FIRST B FIRST
1 z y z (A) y (B)
2 y x y (B) z (A)
3 x w x (B) x (B)
4 w z w (B) w (B)
Not Always So Balanced

27. Team Draft
Interleaving

28. INPUT RANKING TEAM DRAFT
Rank A B BBA ABA AAA
1 z y y (B) z (A) z (A)
2 y v z (A) y (B) y (B)
3 x z v (B) v (B) x (A)
4 w u x (A) x (A) v (B)
w (A) w (A) w (A)
u (B) u (B) u (B)

29. INPUT RANKING TEAM DRAFT
Rank A B BBA ABA AAA
1 z y y (B) z (A) z (A)
2 y v z (A) y (B) y (B)
3 x z
4 w u

30. INPUT RANKING TEAM DRAFT
Rank A B BBA ABA AAA
1 z y y (B) z (A) z (A)
2 y v z (A) y (B) y (B)
3 x z v (B) v (B) x (A)
4 w u x (A) x (A) v (B)

31. INPUT RANKING TEAM DRAFT
Rank A B BBA ABA AAA
1 z y y (B) z (A) z (A)
2 y v z (A) y (B) y (B)
3 x z v (B) v (B) x (A)
4 w u x (A) x (A) v (B)
w (A) w (A) w (A)
u (B) u (B) u (B)

32. Use Team Draft?

33. Finding the Preference

34. Confidence Intervals
Bootstrapping 101

35. How much
less data?
● Yahoo: 20x - 400x
● Netflix: > 100x
● Wikipedia: 10x - 100x

36. Is it Accurate?
Fair Use, Netflixhttps://medium.com/netflix-techblog/interleaving-in-online-experiments-at-netflix-a04ee392ec55

37. Running Shorter
Tests
But not too short.

38. Implementation

39. Backend
● _msearch
● interleave
● Include “owner” in
response

40. Frontend
● Log clicks with owner of
clicked link

41. Analysis
● Choose win/tie on
per-search basis
● Choose win/tie on
per-session basis
● Bootstrap confidence
intervals
● Pretty graphs

42. Thank
You!
(Camel of knowledge)