This presentation by John Davies (Independent competition economist) was made during the discussion “Trials and experiments in competition and regulation” held at the 75th meeting of the OECD Working Party No. 2 on Competition and Regulation on 12 June 2023. This presentation was uploaded with the author’s consent.

1. The use of
randomised control
trials in an abuse of
dominance complaint
JOHN DAVIES, JUNE 2023
Disclaimer: The material in this presentation relates to work the author carried out for Spotify while employed at
Compass Lexecon, a consultancy. The opinions expressed here are the author’s and should not be assumed to
represent Compass Lexecon, Spotify or any other organisation.

2. The smartphone/app ecosystem
App
developers
Consumers
Consumers Jail-breaking Side-loading
Pre-installing

3. Music streaming and Spotify
 Spotify operates a ‘freemium model’:
– Free (ad-funded) service with restrictions on
what users can do
– ‘Premium’ subscription service, no
restrictions
 ‘Conversion’ of users to Premium is
critical to success.
 Subscription is device-independent.
Can ‘convert’ anywhere that accepts
payment: in an app, on a web page,
even with a voucher etc.

4. Spotify’s 2019 complaint to the EC against
Apple (in part)
 To be downloaded from Apple’s iOS App Store, apps must follow Apple’s App Store rules.
“If you want to unlock features or functionality within your app, (by way of example:
subscriptions, in-game currencies, game levels, access to premium content, or unlocking a
full version), you must use in-app purchase.” App Store Rules, quoted in Epic vs Apple judgment 2021
 Spotify turned off in-app purchase on iOS, so the Spotify app on the i-Phone and i-Pad is
just a ‘reader’: it can access premium content, but users cannot pay in-app to convert
from free to premium. Whether in-app payment is enabled or not, the ‘anti-steering’
provisions of the App Store rules apply:
“Apps may not use their own mechanisms to unlock content or functionality, such as license
keys, augmented reality markers, QR codes, etc. Apps and their metadata may not include
buttons, external links, or other calls to action that direct customers to purchasing
mechanisms other than in-app purchase.” App Store Rules, quoted in Epic vs Apple judgment 2021
In a complaint to the EC in 2019, Spotify claimed the anti-steering provisions caused many
of its Free service users not to convert to Premium. Given that free users could pay on a
website, then access Premium in the app, how important was this?

5. A/B testing
Tech firms frequently run “A/B tests”
• Try out a new app (or new web page, or
new shape of text box) on some users
(A)
• Leave other users unaffected (B)
• Monitor the results and compare.
This is a ‘Randomised Control Trial’, an
experiment (like the clinical trial of a new
medical treatment). Group A gets the
‘treatment’ Group B is the ‘control’.
Source: xkcd.com

6. Spotify’s Android app could be modified to
act like its iOS app
Android ‘iPhone’
Crucially, Android did not apply anti-steering rules like Apple’s. On Android phones, Spotify
free users could convert by clicking on a button in the app (which opened a web page in the
app, allowing payment by credit card).

7. 60,000 Android users given experience like
an iOS app, results tracked over 6 months+
~60,000
Control (Android
experience)
Treatment
(iOS experience)
Randomly
assigned
Exposed
to
invitation
screen
New users on
Android in
Germany,
Spain,
France, UK,
Italy:
~600 000
~550,000 Then track
how many
from each
group
converted to
Premium

8. Spotify experiment 2: try out different
aspects of the iOS experience, on Android
New users on
Android in
Germany, Spain,
France, UK,
Italy, Australia,
Brazil, Mexico
and USA:
~1,000,000
Control:
Treatment 2: Walled-off
Treatment 1: Web page group
Treatment 3: Walled-off generic (iOS)

9.  Difference in conversion rates between
full iOS experience and normal Android
experience was significant (both
statistically speaking and, in Spotify’s
view, in business effects).
 Difference in conversion rates between
different options in the second
experiment showed relative importance
of aspects of the anti-steering rules and
can help design remedies.
Results of experiments
Source: Spotify

10. Suggestions and some questions
 Randomised control trials can provide evidence on price and other
elasticities of relevance to competition matters
 A/B tests in the digital sector can provide very strongly statistically
significant results because of huge sample sizes.
 Furthermore, even smaller well-designed experiments are likely to detect
even small effects reasonably robustly because the control vs treatment
method isolates the effect: little scope for alternative ways of analysing the
data (and ‘battles of the models’).

11. Suggestions and some questions
Source: xkcd.com

12. Suggestions and some questions
 Randomised control trials can provide evidence on price and other
elasticities of relevance to competition matters
 A/B tests in the digital sector can provide very strongly statistically
significant results because of huge sample sizes.
 Furthermore, even smaller well-designed experiments are likely to detect
even small effects reasonably robustly because the control vs treatment
method isolates the effect: little scope for alternative ways of analysing the
data (and ‘battles of the models’).
 Best practice for competition authorities: very similar to surveys?
– How much raw data do you want/need parties to a case to provide?
– Try to get companies to pre-commit to methodology?
– Try to be involved in the design?
 Require/encourage companies to carry out tests? Presumably not if to do
so could involve collusion or abuse of dominance!

13. Thank you
https://www.compasslexecon.com/the-analysis/testing-
testing/11-15-2021/
Further reading:
Testing, testing…
John Davies, Peter Bönisch, Rashid Muhamedrahimov
JohnDaviesEcon@gmail.com