/

1. The Specifics of the AB-test
for Unit Economics
with a Network Effect
20.11.2023 - 24.11.2023
Belgrade, Serbia
Alexei Chernobrovov, PhD

2. Alexei Chernobrovov is a multinational manufacturer
of confectionery, pet food, and
other food products.
Core expertise:
• Building DS teams
• Data monetization
• Implementation of
complex DS architecture
Data Monetization and Data Science Consultant
is one of the biggest
carmakers in the world.
is an online platform for food
delivery and the fastest growing
tech company in Qatar.
Phd in Mathematics
Worked with:
is one of the leading
banks in Austria, Central
and Eastern Europe.
Is largest bank in
Russia.
and others: …
https://chernobrovov.com/

3. 2
1
Plan
AB test
3 Conclusions
Network Effect in UE
4
Network Effect

4. AB test

5. AB testing (also known as a split testing or a bucket
testing) is a methodology for comparing two
versions (A and B) against each other to determine
which one performs better.
Usually, users are randomly divided into two groups
and allowed to interact with different versions
independently.
And groups are independency. This is known as the
Stable Unit Treatment Value Assumption (SUTVA).
AB test

6. AB tests have many pitfalls related to p-value and
other features of statistical hypothesis testing.
AB test

7. AB test for conversion rate

8. Network Effect

9. The network effect in the AB test is a situation when
behavior in one group affects the results in another
group (often it happens implicitly).
А В
Network effect

10. Two well-known cases:
1. A social network or dating in which users clearly
influence each other through actions.
2. Bounded supply (number of taxis or couriers) for
ride-hailing or delivery services.
Network effect

11. Why does network effect occur?
Users can influence each other explicitly.
For example, if you restrict users of one group in
some actions, and they have social media friends
that are users from another group, and then the
actions of users from one group affect the other
group.
Social network or datings

12. There are several ways to solve this problem and it
depends on the formulation of the problem in the
AB test.
But they are somehow related to the construction
of special graphs to minimize the network effect.
For example:
● Using Ego-Clusters (Linkedin)
● Commuting Zones experiment (Facebook)
How to solve?

13. Why does network effect occur?
Clients are interconnected implicitly through the
presence of a common supply.
For example, if taxi prices are decreased for one
group of users, they can use the whole supply.
Then fewer taxi drivers will be available to users
from another group, and because of this, the
conditions for them will be significantly worse due
to the delivery time and so on.
Ride-hailing

14. One of the most common approaches is switchback
tests.
The essence of this approach:
● Divide the map into areas A and B. Different
versions are used for A and B.
● After a short period of time, the areas change.
● The switching process continues throughout the
experiment.
Due to continuous switching, the network effect
practically disappears.
How to solve?

15. Network Effect in Unit Economics

16. What is Unit economics?
Unit economics are the direct revenues and costs
of a particular business measured on a per-unit
basis, where a unit can be any quantifiable item
that brings value to the business.

17. For manufacturer: item
For SaaS: server or license
For social network: user
For ride-hailing or delivery services: ride
For retail (e-commerce):
● client (user)
● order
Different units

18. Why is it so important?
It is a very simple indicator.
If UE<0 - more orders leads to more loss.
If UE>0 - business should be scaled. And it is
enough to have N (number of orders) such as:
UE·N – Other Costs*>0
Then the business will break even.
UE-per-orders
*Other Costs – office rent, salaries, etc.

19. Example of UE

20. We have 2 types of costs, calculating UE: variable
and fixed.
Variable – all variable costs, for example, marketing
costs, delivery of the order to the client, etc.
Fixed* – all fixed costs, such as warehouse rent,
highway mile.
Variable and fixed costs
*It is better not to consider fixed costs in UE at all, but in practice it can
be a significant part of the unit costs and it should be considered for a
fairer assessment of business.

21. Yet another network effect!

22. If we randomly divide users into groups A and B for
a test, then it will be enough for valid AB test on
revenue or conversion rate, etc.
But there is the paradox!
A random division by user may be incorrect for an
AB test for unit economics.
Because it violates the independence of groups!
One problem with AB tests in retail

23. 210 − 105 − 100
10
=
UE per order,
thousands
10
Orders, thousands
Revenue, mln
210
Var costs, mln
105
0.4
UE per order,
thousands
12
Orders, thousands
Revenue, mln
250
Var costs, mln
125
А
В
Model example of AB on UE
100
Fixed costs, mln
120
Fixed costs, mln
0.5
250 − 125 − 120
12
=

24. 210 − 105 − 100
10
=
UE per order,
thousands
10
Orders, thousands
Revenue, mln
210
Var costs, mln
105
0.4
UE per order,
thousands
12
Orders, thousands
Revenue, mln
250
Var costs, mln
125
А
В
Model example of AB on UE
100
Fixed costs, mln
120
Fixed costs, mln
0.5
250 − 125 − 120
12
=
Total fixed costs, mln
220
+
=

25. 210 − 105 − 110
10
=
UE per order,
thousands
10
Orders, thousands
Revenue, mln
210
Var costs, mln
105
-0.5
UE per order,
thousands
10
Orders, thousands
Revenue, mln
210
Var costs, mln
105
А
В
Model example of AB on UE
110
Fixed costs, mln
110
Fixed costs, mln
-0.5
210 − 105 − 110
10
=
А Total fixed costs, mln
220
+
=

26. 250 − 125 − 110
12
=
UE per order,
thousands
12
Orders, thousands
Revenue, mln
250
Var costs, mln
125
1.3
UE per order,
thousands
12
Orders, thousands
Revenue, mln
250
Var costs, mln
125
А
В
Model example of AB on UE
110
Fixed costs, mln
110
Fixed costs, mln
1.3
250 − 125 − 110
12
=
В

27. How to solve this problem?

28. There are 2 approaches to solving this problem:
1. Correct calculating of the AB test. It’s enough just
to see what happens if we apply the results. As it
was shown in the example above.
2. Building a predictive unit economics model that
considers nonlinearity from the number of orders
and other factors (maybe).
How to solve this problem?

29. The simplest way is to build a model that predicts
the change in fix costs per unit depending on the
number of orders (and possibly other parameters).
And when calculating tests, substitute the predicted
values.
In general, it is difficult to build such a model, but it
is quite sufficient to build a local model (within the
current level of business).
Predictive model for UE

30. All UE indicators (even variables) can depend on
the number of orders.
For example, advertising costs. Obviously, if you
want to attract more orders only through
advertising, it will likely become less effective per
unit.
Predictive model for UE

31. Calculate the total UE:
𝑈𝐸𝑡𝑜𝑡𝑎𝑙(𝑁) = 𝑅𝑒𝑣𝑒𝑛𝑢𝑒(𝑁) −
− 𝑉𝑎𝑟𝐶𝑜𝑠𝑡1 𝑁 − 𝑉𝑎𝑟𝐶𝑜𝑠𝑡2 𝑁 − ⋯
− 𝐹𝑖𝑥𝑐𝑜𝑠𝑡.
And calculate the total UE (per-unit):
𝑼𝑬 𝑵 =
𝑼𝑬𝒕𝒐𝒕𝒂𝒍 𝑵
𝑵
,
where 𝑉𝑎𝑟𝐶𝑜𝑠𝑡𝑖 𝑁 - simple model of each type of
costs. It can be a simple linear approximation.
𝑁 - number of orders.
Predictive model for UE

32. Conclusion

33. Conclusion
• You need to be very attentive to the network
effect and check for the presence of implicit
dependencies.
• Network effects occur in the most unexpected
cases. Any bound that links users can lead to
network effects. For example, fixed costs.
• Using a simple model to predict the local
dependency on the unit economy is a very
effective way to successfully conduct AB tests.

34. a@chernobrovov.com
fb.com/chernobrovov
linkedin.com/in/alex-chernobrovov/
@chernobrovov
Alexei Chernobrovov
Thanks for Attention!

35. Links
• Using Ego-Clusters to Measure Network Effects at LinkedIn
• Testing product changes with network effects (Meta)
• The pitfalls of A/B testing in social networks (OkCupid)
• Designing A/B tests in a collaboration network (Google
Cloud Platform)
• Switchback Tests and Randomized Experimentation Under
Network Effects at DoorDash
• Balancing Network Effects, Learning Effects, and Power in
Experiments (DoorDash)