Talk from Yandex at LinkedIn Experimentation Meetup on Dec 13

2. Consistent Transformation of Ratio Metrics
for Efficient Online Controlled Experiments
Roman Budylin, Alexey Drutsa,
Ilya Katsev, Valeriya Tsoy

3. About Yandex
• The largest internet company in Europe
• ”Google of Russia”, “Amazon of Russia”, “Uber of Russia” and so on
• More than 50M DAU

4. A/B testing methodology

5. Group
A
Group
B
Split
them
randomly
trafficofusers

6. Group
A
Group
B
Split
them
randomly
trafficofusers
Variant for A
Variant for B
Expose to one of
two variants of the service
e.g., the current production version
e.g., an evaluated update

7. Group
A
Group
B
Split
them
randomly
trafficofusers
Variant for A
Variant for B
Expose to one of
two variants of the service
e.g., the current production version
e.g., an evaluated update
Calculate a key
measure for each user
X(uA1)
…
Calculate the OEC for each
group as the mean value
e.g., X(u) is the number
of sessions of the user u
X(uA2)
X(uA3)
X(uA4)
X(uA5)
X(uB1)
…
X(uB2)
X(uB3)
X(uB4)
X(uB5)
µA(X)=avgu in AX(u)
µB(X)=avgu in BX(u)
Overall Evaluation Criterion
(OEC) for the group B
Overall Evaluation Criterion
(OEC) for the group A

8. µA(X)=avgu in AX(u)
µB(X)=avgu in BX(u)
Overall Evaluation Criterion
(OEC) for the group B
Overall Evaluation Criterion
(OEC) for the group A
Calculate the OEC for each
group as the mean value
Δ(x) VS 0
Δ(X) = µB(X) – µA(X)
the evaluated update is
positive or negative
Statistical
significance test
the difference is caused by
a noise or
the treatment effect
(e.g., Student’s t-test)
Overall Evaluation
Criterion (OEC)
[Kohavi et al., DMKD’2009]
Overall Acceptance
Criterion (OAC)
[Drutsa et al., CIKM’2015]
Sensitivity
Directionality

9. OEC levels
1. User level metrics
2. Non-user level metric
For example ratio OEC:

10. Example – average length of session

11. Example – average length of session

12. Example – average length of session
> 30m

13. Example – average length of session

14. Example – average length of session
Let user u has sessions with lengths
Then the average of lengths of all sessions is equal to

15. Problem
For ratio OEC
t-test
sensitivity improvement techniques

16. Problem
For ratio OEC
t-test
sensitivity improvement techniques

17. Problem
For ratio OEC
t-test
sensitivity improvement techniques

18. Existing approaches
• Bootstrap test
Problem: computational expensiveness
• Delta method
Problem: does not allow to apply directly regression adjustment
techniques

19. Linearization transformation

20. Our paper, WSDM 2018
21
Consistent Transformation of Ratio Metrics for Efficient Online
Controlled Experiments
Roman Budylin, Alexey Drutsa, Ilya Katsev, Valeriya Tsoy
Data Quality metrics • OEC metrics • Guard rail metrics • Local feature/Diagnosticmetrics

21. Our paper, WSDM 2018
22
Consistent Transformation of Ratio Metrics for Efficient Online
Controlled Experiments
Roman Budylin, Alexey Drutsa, Ilya Katsev, Valeriya Tsoy
The best comment: «This is too good to be true!» (someone from
Facebook)
Data Quality metrics • OEC metrics • Guard rail metrics • Local feature/Diagnosticmetrics

22. We found a transformation such that
Our contribution

23. We found a transformation such that
Ratio OEC User level OEC
Our contribution

24. We found a transformation such that
Ratio OEC User level OEC
NB: Preserve directionality and significance level!
Our contribution

25. Let we have a ratio OEC:
Consider the next expression:
And let us use its average
as a metric:
Now we got a linearization OEC:
Linearization

26. Let A and B be the control and the experiment. Let us denote
Theorem 1: Let be positive. Then for any
the next is true:
Theorems: directionality

27. Theorem 2: Let be positive and . Let
be the t-statistic applied for the OEC and let be the be the asymt.
standard normal statistic of obtained via the Delta method.
If then then under the null hypothesis that
1. the t-statistics is asymptotically normal
2. converges to 1 by probability
Theorems: significance level

28. Approaches comparision

29. Experiments
• Internet search, 2013-2016, 390 experiments
• The transformation + regression adjustment
• +34% of sensitivity

30. Experiments

31. The code