This presentation by Gábor Koltay, Senior Economist at EU DGComp, was made during the discussion “Trials and experiments in competition and regulation” held at the 75th meeting of the OECD Working Party No. 2 on Competition and Regulation on 12 June 2023. This presentation was uploaded with the author’s consent.

1. Experiments in antitrust
The European Commission’s use of randomised control trials
in the Google Shopping, Amazon Marketplace and Buy Box cases
75th meeting of Working Party 2 on Competition and Regulation
12 June 2023, Paris, France
Gábor Koltay, Senior Economist
DG COMP, European Commission
The content of this presentation does not reflect the official opinion of the European Union. Responsibility for the information and
views expressed therein lies entirely with the presenter.

2. • Experimental data on consumer and firm behavior, as well as display and
algorithmic design choices
• A/B testing
• Units are split in two groups randomly and presented with two different versions of the
same product/service
• Standard in digital services
• Might still be expensive for smaller companies
• Their use in antitrust cases of the European Commission:
• Evidence on effects
• Effectiveness of commitments
Experiments in digital markets

3. Experimental evidence on the effects of abusive conducts

4. The Google Shopping case in a nutshell
Query
Shopping unit:
specialized
search results
Links to standalone
pages of specialised
search services
AdWords
("paid search")
Generic
search results
• Conduct: more favourable positioning
and display of Google Shopping
compared to comparison shopping
services
• Google Shopping always on top,
rich display (images, prices)
• Competitors ranked low, often do
not appear on the first page of
search results
• Conduct is a combination of
several business interventions:
• Product Universal
• Panda algorithm change
• Shopping Unit

5. A causal map for conducts and potential outcomes
Product
Universal
Consumer
decisions
Potential effect
on competitors
Panda update
to the ranking
algorithm
Shopping
Unit
Other
factors
influencing
consumers
Clicks to
competitors
Clicks to
Google
Change in
competitors’
and Google’s
display

6. • Testing part of the conduct: the display of the Shopping Unit
• Experiment submitted by Google in its defence
Google’s “ablation” experiment
Product
Universal
Consumer
decisions
Potential effect
on competitors
Panda update
to the ranking
algorithm
Shopping
Unit
Change in
competitors’
and Google’s
display
Other
factors
influencing
consumers
Clicks to
competitors
Clicks to
Google

7. • Testing part of the conduct: the display of the Shopping Unit
• By removing it for some of the users
Google’s “ablation” experiment
Product
Universal
Consumer
decisions
Potential effect
on competitors
Panda update
to the ranking
algorithm
Shopping
Unit
Change in
competitors’
and Google’s
display
Other
factors
influencing
consumers
Clicks to
competitors
Clicks to
Google

8. • Experimental design
• Tests only part of the conduct (Shopping Unit) and effects on competitors (not on Google)
• Randomisation over users: many queries do not show any comparison shopping result. Which are
the relevant queries?
• Imperfect compliance: search algorithm might not show the Shopping Unit for users neither in the
treatment nor in the control group
• In general it can be difficult to design experiments for complex events and equilibrium effects
• Results
• Generic traffic of competitors: “non-negligible decrease in traffic” (magnitude confidential)
• Ads traffic of competitors: 16 to 30% decrease (Judgement of the General Court)
• Importance: ads cannot substitute for generic traffic loss, because they themselves become
less effective
Experimental design and results

9. Experiments and the effectiveness of commitments

10. Amazon’s retail integration: marketplace platform + retail seller
• Amazon retail’s use of non-public third-party seller data
• Data on listings and transactions observed by Amazon marketplace
• Amazon retail uses this data in competition with third-party sellers
Structural competitive advantage for Amazon Retail + increased risks and costs of third-party
sellers + potential partial foreclosure.
• Amazon’s Buy Box and Featured Offer display:
• Amazon’s BuyBox and Featured Offer algorithms favour Amazon Retail and Prime
labelled offers
• Prime label and choice of delivery services
• Third-party sellers using Amazon’s fulfilment services are more likely to be selected as the
Featured Offer and are awarded the Prime label more easily
Amazon Marketplace and Buy-box cases

11. Commitments

12. Experimental testing of the Second Displayed Offer
• Amazon will report, based on the relevant experiments, the impact of the
implementation on
• Units sold for affected ASINs (including both Seller and Amazon Retail Offers)
• The frequency of display of the Second Displayed Offer
• Consumer interaction with the Second Displayed Offer
• For ASINs that had at least two Offers qualified for selection as the Featured Offer
across the Amazon Stores
• The European Commission can request reports on the results of all Experiments on
the implementation of the commitments
• run during the Implementation Period or
• in the twelve months following the end of the Implementation Period

13. Thank you

14. Back-up slides

15. AB testing in Google Shopping - “ Ablation
E xperim ent”
User
Box is
supressed
Business
as usual
Search
query
generic +Ad
Search
query
generic +Ad+ Box
generic +Box
generic +Ad
generic +Ad+ Box
generic +Box
CSS
No CSS
CSS
No CSS
Clicks
CSS
No CSS
CSS
No CSS
CSS
No CSS
CSS
No CSS
𝑊𝑊𝑖𝑖 = 0
𝑊𝑊𝑖𝑖 = 1

16. AB testing in Google Shopping - “ Ablation
E xperim ent”
User
Box is
supressed
Business
as usual
Search
query
generic +Ad
Search
query
generic +Ad+ Box
generic +Box
generic +Ad
generic +Ad+ Box
generic +Box
CSS
No CSS
CSS
No CSS
Clicks
CSS
No CSS
CSS
No CSS
CSS
No CSS
CSS
No CSS
No potential outcomes created
𝑊𝑊𝑖𝑖 = 0
𝑊𝑊𝑖𝑖 = 1