Use of Economic Evidence in Cartel Cases – DAVIES – December 2023 OECD discussion

Economics evidence in cartel cases:
presenting to courts
John Davies
OECD Global Forum on Competition 2023
John Davies is a Member of the UK Competition AppealTribunal (CAT) and an Independent Consultant. This
presentation was prepared in a personal capacity and does not represent the views of the CAT.
As a consultant, I have been instructed for parties (under investigation/defendants or complainants/claimants) in
numerous competition matters in front of competition authorities and courts.

Topics
01. Intro: what should economists do and what do they do?
02. How to lie with statistics
03. Data analysis example: when economists disagree
04. How to tell the truth with statistics (or get someone else to do so)
05. Economists in the witness box – case examples
2

What should economists do?
“It should be a matter for specialists—like
dentistry. If economists could manage to get
themselves thought of as humble, competent
people on a level with dentists, that would be
splendid.”
J. M. Keynes, (1931) Essays in Persuasion.
“Give me a one-handed Economist. All my
economists say 'on hand...', then 'but on the
other...”
President Harry S. Truman (apocryphal)

What do economists do? (sometimes)
“Dr. Hall's opinion should not have been admitted because it did not incorporate all
aspects of the economic reality of the stern drive engine market and because it did not
separate lawful from unlawful conduct. Because of the deficiencies in the foundation of
the opinion, the expert’s resulting conclusions were mere speculation”
Judgment in Concord Boat Corp v Brunswick (207 F.3d 1039, 8th Cir., 2000
“By contrast, the evidence at trial showed that Professor Shapiro's model lacks both
"reliability and factual credibility," and thus fails to generate probative predictions of
future harm associated with the Government's increased-leverage theory.”
Judge Leon,ATT/TimeWarner judgment
“I have to say, I have never listened to evidence in any court for an hour and understood
so little of it as I have understood during the last hour. […] At the moment, I am firmly,
myself, of the school which says ‘this is all too difficult, we had better give up’. I simply
warn that - I am very sorry, it is all above my head.”
Mr. Justice Ferris, OFT vs F.A Premier League, 1999 (transcript)

Topics
5

Data analysis. Damn lies?
“There are three types of lies --
lies, damn lies, and statistics.”
Benjamin Disraeli
“Facts are stubborn things, but
statistics are pliable.”
MarkTwain
“Most people use statistics like
a drunk man uses a lamppost;
more for support than
illumination”
Andrew Lang
6

Be sceptical…
7
• Breast cancer rates higher among Chinese men
than women – because very few Chinese women
went to hospital (Statistical principle: selection bias)
• More stork nests on Dutch houses with more
children – because bigger houses associated with
more roof space and more children (Statistical
Principle: Common causal variable)

Be sceptical… but not cynical
8
‘The irreverence of these examples, which [Huff] also
used in testimony before the Senate Committee on
Commerce, provoked Senator Maurine Neuberger to
ask “Do you honestly think there is as casual a
relationship between statistics linking smoking with
disease as there is about storks and Chinese and so
on?”
They “seem to me the same,” Huff replied.’
Reinhart, A. (2014). Huff and puff. Significance, 11(4), 28-33.

Topics
9

Data analysis example:
cartel overcharge
Ideal: price before 2012 always EUR150,000; price after always EUR 100,000: overcharge=EUR50,000 per
project
Messy reality: prices of projects sold to claimants, EUR thousands
After a competition authority has found an illegal cartel, customers may seek damage awards from
courts. In England &Wales, such cases are usually heard by the CAT and:
• It is purely about the damages, it is not open to defendants to dispute the CA’s findings
• The case is decided on the balance of probabilities.
Often, economists will testify about the ‘overcharge’, comparing prices ‘during’ and ‘after’ the cartel.
Claimants: average price before dawn raid EUR 142,000; average price after EUR 91,000;
overcharge = EUR 51,000 per project.
2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
156 144 181 135 140 179 130 133 101 95 97 100 86 89 89 79
156 147 133 128 126 135 130 105 101 120 99 61 77
145 138 133 112 89 89 56 77
155 126 96 92
142 135 107 86
138

0
20
40
60
80
100
120
140
160
180
200
Dec 2003 Dec 2005 Dec 2007 Dec 2009 Dec 2011 Dec 2013 Dec 2015 Dec 2017 Dec 2019 Nov 2021
Power cable projects sold to Claimant (ideal, simple)
Example: cartel overcharge (ideal)
Overcharge
per-project
EUR 50,000

Example: cartel overcharge (reality)
0
20
40
60
80
100
120
140
160
180
200
Power cable projects sold to Claimant (messy reality)

0
20
40
60
80
100
120
140
160
180
200
Power cable projects sold to Claimant
Example: cartel overcharge (Claimant
economist initial estimate)
Overcharge
per-project
EUR 51,000
Average price
before
Average price after

Cartel overcharge: a
difference of expert opinion
• Claimant expert economist: average project prices EUR 51,000 higher during
cartel than after 2012
• Defendant expert economist: Claimant’s expert is trying to bamboozle the
Court:
• There is a downward trend in power cable prices, because of more
efficient production processes.
• Furthermore, some of the prices are weird and probably reflect
unrepresentative projects.
• Correct for those and there is zero change from 2012.

0
20
40
60
80
100
120
140
160
180
200
Example: cartel overcharge
(Defendant economist view)
Nothing
changed in
June 2012 –
overcharge
= zero
“Line of best fit” or
“Regression line”

0
20
40
60
80
100
120
140
160
180
200
(Defendant economist view +)
Nothing
changed in
June 2012 –
overcharge
= zero
“Line of best fit” or
“Regression line”
Outliers – exclude
them!
Outliers –
exclude them!

Cartel overcharge: a
difference of expert opinion
• Claimant expert economist:
• OK, I accept there may be a trend and I have now accounted for that in my
modelling. But there is a still a discontinuity (jump) in the series in 2012!
• I also accept that the very expensive projects in 2007 and 2010 were
supply of short lengths of spare cable. However, I see no reason to
exclude the low-priced projects post-cartel but I have corrected a project
for which the wrong currency was recorded.
• With these changes, I obtain an estimate of EUR 25,000 overcharge per
project (or EUR 21,000 without the corrected data point)

0
20
40
60
80
100
120
140
160
180
200
(Claimant economist revised view)
Outliers –
excluded
Data error
corrected
Overcharge
per-project
EUR 25,000
Nothing strange about
these – leave them in!

Which expert is right?
Examine it point by point
Two main areas of disagreement: outliers and trends
Outliers
• The experts disagree on whether some of the data points are representative or not.
• Defendant’s expert wants to exclude unusually high prices during the cartel, unusually low
prices after (favouring his client’s interests).
• Claimant’s expert is more balanced but disagrees that there is anything ‘unusual’ about the low
prices after.
Trends
• Is there an overall downward trend or is that just chance? Helps defendant but Claimant expert
accepts there is a downward trend.
• However, defendant expert says there is only a single downward trend, so nothing changed
when cartel ended.
• Claimant expert says data pattern is explained better by a downward trend with a one-off ‘jump’
down when the cartel ended – and identifies this ‘jump’ as the overcharge.

Which expert is right? Examine it
point by point: (1) outliers
The question of ‘outliers’:
• Obtain factual witness evidence.
Which projects were actually
unrepresentative? Why? Economists
are probably not expert in what
is ‘representative’ in this context!
Data errors too are matters of fact.
• If experts claim ‘outliers’ should
simply be excluded as a matter of
standard statistical practice (a) have
them challenge each other if possible
and (b) why did they not establish
their rules for ‘excluding outliers’
before looking at the data?
0
20
40
60
80
100
120
140
160
180
200

Which expert is right? Examine it
point by point: (2) trends
The question of trends and series
breaks:
1. Is there a downwards trend? Or
does the ‘downwards’ pattern we
see in the data just reflect some
random variation, plus a cartel effect?
2. Assuming there is a trend, is there
nonetheless a ‘break’ in 2012?
Both questions will be addressed by experts
using ‘statistical significance’. 0
20
40
60
80
100
120
140
160
180
200

Statistical significance: evaluating whether
we are seeing a pattern by chance?
Often expressed as ‘p-value’:What is the probability that we would see this by
chance even if there were no effect? Usual threshold is 5% or 1%.
Are these coins biased, in the sense that they always come up heads?
Toss one coin Toss three coins Toss ten coins
Probability of seeing this (if
coin is not biased):
<0.1%
50% 12.5%

0
20
40
60
80
100
120
140
160
180
200
1. Statistical significance: how likely is it that the
‘downward slopes’ are just chance variation?
Answer: very
unlikely: the
downward trend is
highly statistically
significant.
Probability < 1 in
1000 that we would
see a pattern like
this if there were
no trend.
The expert
economists agree
on this.

0
20
40
60
80
100
120
140
160
180
200
2. Statistical significance: how likely is it that the
‘jump’ in 2012 is just chance?
Answer: fairly
unlikely: the series
break is marginally
statistically significant.
Probability < 1 in 10
that we would see a
pattern like this if
there is no effect of
the cartel in 2012
The expert
economists agree on
this analysis, disagree
on interpretation

Statistical significance: important but
not the only test of good analysis
Often expressed as ‘p-value’:What is the probability that we would see this by
chance even if there were no effect? Usual threshold is 5% or 1%.
Economists often overstate the importance of statistical significance. It measures one
thing, not the overall reliability of the analysis. Even a very strong p-value (1% or
better) does NOT mean the economist’s conclusions are all correct.
• A 1% p-value does not mean that the finding is 99% likely to be true.
• Does not usually say anything about causation.Think about other factors: does
factual witness evidence suggest anything else happened in 2012 to lower prices?
• Selection of data sources, choice of methods can artificially improve the p-value
In short, can we say “My model is better than his because it has higher statistical
significance”? No!

How to ‘not quite lie’ with statistics:
“p-hacking”
Recall:‘p-value’:What is the probability that we would see this by chance even if there
were no effect? Usual threshold is 5% or 1%.
However, the probability in the data provided may not represent the overall probability.
Toss ten coins
<0.1%

“p-hacking”
Recall:‘p-value’:What is the probability that we would see this by chance even if there
were no effect? Usual threshold is 5% or 1%.
However, the probability in the data provided may not represent the overall probability.
23%
Toss ten coins
<0.1%
Toss sixteen coins (then throw six ‘tails’ away)

“p-hacking”
“The inappropriate manipulation of data analysis to enable a favoured result to be
presented as statistically significant.”
• Tossing a coin 16 times and only reporting 10 heads
• Tossing 10 coins 100 times and only reporting the time that 10 heads came up.
• Rolling 3 dice and getting the ‘unusual’ sequence “1, 2, 3”. What are the chances?
• Pretty low if you said in advance:“I wonder if I will get 1, 2, 3?”
• Much higher if you only said in advance:“I wonder if I will get an ‘interesting’
sequence?”. 1, 2, 3 or maybe 6, 6, 6 or maybe 3, 1, 4 (Pi!)
Huge problem in empirical/experimental social science. See https://datacolada.org/
Also: Stefan and Schönbrodt (2023) “Big little lies: a compendium and simulation of p-
hacking strategies”, https://royalsocietypublishing.org/doi/10.1098/rsos.220346

How to p-hack in competition
economics
There are all sorts of decisions/choices an economist makes when conducting
quantitative analysis. E.g.:
• What data to use
• How to measure the data, e.g. assigning values to non-quantified data, dealing with
exchange rates/inflation, how to split up continuous data, express values in logs or
not? etc etc…
• Excluding ‘outliers’ or not
• Choice of control variables – model design
If one final result is presented to the Court, it may be just one of many, many
possible results in the universe of different possible outcomes.
Economist will have a reason for each choice: but was it just an ex-post justification?

Topics
04. How to tell the truth with statistics (or get someone
else to do so)
30

How to tell the truth with statistics
(or get someone else to do so)? 1
Does the expert’s analytical process allow him/her to reach the favoured result?
• Has expert shown all results, from all modelling variants and explained choices?
• Have each expert’s analytical decisions worked in their clients’ favour?
Repeatedly?
• Did experts publicly state their methodologies before examining data?
Is expert’s behaviour credible?
• Does expert “change his/her opinion when the facts change” or double down?
• Conversely: has expert made outlandish claims from which he/she backs away
under challenge (before or during trial)?

How to tell the truth with statistics
(or get someone else to do so)? 2
Does analysis fit the facts? Corroborating factual evidence on same question: e.g.
how did salespeople actually go about setting the prices?
Testing to destruction (details will vary by legal system):
• Has everything that each expert examined been put to the court and/or the
other side?
• Can experts exchange views and criticise one another?
• Cross examination / hot tubbing. How strongly held is opinion?
A duty to the Court?
Court-appointed expert, or economist on Tribunal as or assisting judge?

Topics
33

Case examples:
Credibility of experts
It appeared to us that Dr B lacked the objectivity and balance required of an independent expert
witness. […] Moreover, whilst Dr B exhibited an impressive recall and command of the written
materials, in cross-examination we thought that he was intent on seeking to promote BT’s case
by advancing arguments, rather than simply answering the questions he had been asked.
Accordingly, we regret that we were not assisted by much of Dr B’s evidence.”
[Detailed example 1] In cross-examination, Dr B adhered to this opinion, which he expressed
without qualifying it in any way, volunteering that he thought that the contrary approach taken
by Ofcom’s witness, Ms. C, was “absurd”. However, it became apparent that Dr B’s argument
was unsupported by any authority or guidance and had never been adopted by any competition
authority or regulator.
[Detailed example 2] This entire text was pure argument and did not relate to any matters that
properly fell into the scope of an expert witness on economics. […] it is clearly inappropriate
for an expert witness to seek to “fight back” against a pleading and do so by reference to an
entirely unrelated dispute.
https://www.catribunal.org.uk/sites/cat/files/1260_BT_Judgment_CAT_25B_101117.pdf

Case examples (2):Theory consistent
with factual evidence?
[Accused of predatory pricing, NAPP Pharmaceutical argued that its low prices in one segment
encouraged doctors to prescribe its product in ‘follow on’ sales’]
On the basis of the various calculations carried out by Napp’s economic consultants, Nera,
Napp then argued that its hospital sales were profitable if one took into account this ‘follow-on
effect’.
However, […] there is simply no evidence that Napp ever took into account any follow-on
effect when setting or carrying out its pricing policy.
If its pricing policy had in fact been seen by Napp in the way that its economic consultants
suggest, we would have expected the company’s internal documents to demonstrate that.
While expert’s reports are often relevant and helpful to understanding the issues with which this
Tribunal has to deal, we find in this case that the idea of a ‘follow-on’ effect in the narrow or
mechanistic sense relied on by Napp flows not from any internal documents from Napp but
from the work done by Napp’s economic advisers for the purposes of the present case.
https://www.catribunal.org.uk/sites/cat/files/JdgNapp150102.pdf

Case examples (3): full disclosure of
economic analysis
Mr H’s [expert instructed by Claimant] initial intuition was that the standard demand controls
would be sufficient for the GFC effect. It is concerning that this only emerged at the hearing
while he was giving evidence on this area. He disclosed for the first time that he originally ran
his model with the standard demand controls in place and arrived at an Overcharge estimate of
between 1 and 2%. This result was not referred to in his Reports.
He then decided that the demand shift during the GFC was so profound that his demand
controls were not adequate in these years, so he made an ad hoc adjustment to his model to
include additional dummy variables in the three GFC years: 2008 to 2010. […] Having
implemented this change to his model, Mr H then found a higher Infringement effect of 6-14%.
Professor N criticised this ad hoc approach because [various technical reasons]
There is obvious appeal to this criticism, and it is inescapable that Mr H’s approach does appear
to have had the effect of shifting the goalposts ex post after his original model using the
standard demand controls reached an inconvenient result.
https://www.catribunal.org.uk/sites/cat/files/2023-02/2023.02.07_NON-
CONFIDENTIAL_Trucks_1284_90_Final.pdf

Case examples (4): limits of expert
evidence – and role of Instructions
Both Mr V and Dr N were, as we have said, expert economists. Neither of them is an expert in
the field of payment systems, whether generally or specifically in relation to the MasterCard
Scheme. […]
In these circumstances, it was incumbent upon the parties to ensure that the experts gave their
opinions based upon a common – and if possible, agreed – factual base. That did not occur in
this case: Mr V was confronted, in the course of his cross-examination, with material (albeit in
the public domain) which he had never seen before; Dr N, similarly, was confronted with
material which – buried in the 60 plus trial bundles – he had not considered.
Their expertise involved considering certain material, and providing their economic analysis in
relation to it. To the extent that this material was incomplete, or referred to them late, their
analysis was liable to be undermined.
Equally, neither Mr V nor Dr N are lawyers. […] Both economists would have benefitted from
having, in advance, a clear and agreed formulation of what legal principles they were to follow
and what assumptions arising out of these legal principles they were being required to make.
https://www.catribunal.org.uk/sites/cat/files/1241_Sainsburys_Judgment_CAT_11_140716.pdf

Thank you
JohnDaviesEcon@gmail.com
38

Use of Economic Evidence in Cartel Cases – DAVIES – December 2023 OECD discussion

Recommended

Recommended

More Related Content

Similar to Use of Economic Evidence in Cartel Cases – DAVIES – December 2023 OECD discussion

Similar to Use of Economic Evidence in Cartel Cases – DAVIES – December 2023 OECD discussion (20)

More from OECD Directorate for Financial and Enterprise Affairs

More from OECD Directorate for Financial and Enterprise Affairs (20)

Recently uploaded

Recently uploaded (20)

Use of Economic Evidence in Cartel Cases – DAVIES – December 2023 OECD discussion