Quantitative Methods for Lawyers - Class #13 - Power Laws, Hypothesis Testing & Statistical Significance - Professor Daniel Martin Katz

1. Quantitative
Methods
for
Lawyers
Power Laws, Hypothesis
Testing & Statistical
Significance
Class #14
professor daniel martin katz computationallegalstudies.com @ computational

2. Power Law Distribution
(Scale Free)
This is a Classic and Very Important Distribution
A power law is a special kind of mathematical relationship
between two quantities. When the frequency of an event varies
as a power of some attribute of that event (e.g. its size), the
frequency is said to follow a power law.

3. Power Law Distribution
(Scale Free)
Pareto distribution ( Wealth Distribution )
Zipf's law ( Natural Language Frequency )
Links on the Internet
Citations
Richardson's Law for the severity of violent conflicts (wars
and terrorism)
Population of cities
Etc.
Examples:

4. Power Laws Appear to be a
Common Feature of Legal Systems
Katz, et al (2011)
American Legal Academy
Katz & Stafford (2010)
American Federal Judges
Geist (2009)
Austrian Supreme Court
Smith (2007)
U.S. Supreme Court
Smith (2007)
U.S. Law Reviews
Post & Eisen (2000)
NY Ct of Appeals

5. Rare Events, Criticality
Power Laws
Rare Events
Criticality
Disorder
Induction

6. “ [T]here are known knowns; there are things we know we know.
We also know there are known unknowns; that is to say we know
there are some things we do not know.
But there are also unknown unknowns – there are things we do
not know we don't know. ”
United States Secretary of Defense
Donald Rumsfeld

7. Unknown, Unknowns and
Inductivist Reasoning
Philosophy of Science =
How do we Know What We Know?
Black Swan Problem
Even If We Observe White Swan after White Swan
cannot induce that all swans are white

8. Learning by Falsification
Popperian Perspective
Karl Popper Rejected Inductivist Reasoning
Science Advances Incrementally as Hypotheses
are Falsified

9. Learning by Falsification
Of Course, Certain Hypothesis cannot likely be falsified
on a Reasonable Time Scale
The problem of induction:
the sun has risen every day for as long as anyone can
remember.
what is the rational proof that it will rise tomorrow?
How can one rationally prove that past events will continue to
repeat in the future, just because they have repeated in the
past?

10. Learning by Falsification
Popper Solution to the Question:
No Need to Reject the Hypothesis of Sun Rising
Cannot Really Formulate a Theory that Can Prove
that the Sun Will Always Rise
Can Develop a Theory that It Rise which will be
falsified if the sun fails to rise

11. Hypothesis Testing
& Statistical Significance

12. The Null and
Alternative Hypothesis
Example from Criminal Law:
Criminal Trial Burden of Proof
Presumption of Innocence
Not Possible to Conclusively Prove a Lack of
Innocence (with zero doubt)
Must Be Overruled Beyond a Reasonable Doubt

13. The Null and
Alternative Hypothesis
Switch Now To a Scientific Inquiry:
Study is Typically Designed to Determine Whether
a Particular Hypothesis is Supported
Start with Presumption that Hypothesis is Not True
(Null Hypothesis)
Researcher Must Demonstrate That The
Presumption is Unlikely to Be True given the
Population

14. Example: Coin Flip
Nostradamus
Predicting Coin Flips -
Does you Friend Have the General Ability to Actually
Predict Coin Flips?
How Would You Evaluate This Proposition?
How Many Predictions Would Your Friend Have to Get
Right For You To Believe They Actually Have Real
Ability?

15. Example: Coin Flip
Nostradamus
Ho: Cannot Actually Predict Coin Flips
Ho is the Null Hypothesis
H1: Can Actually Predict Coin Flip
(i.e. do so at a rate greater than chance)
H1 is the Alternative Hypothesis

16. Reject the Null versus
Failing to Reject the Null
In the Coin Flip Example, We might have enough
evidence to reject the null
Remember the default (null) is that there is no
relationship
If We Fail to Reject the Null, we are left with the
assumption of no relationship
Although a Relationship might actually exist

17. Coin Flip Nostradamus:
Binomial Distribution
Here is the Formula for a binomial experiment consisting of
n trials and results in x successes. If the probability of
success on an individual trial is P, then the binomial
probability is:
b(x; n, P) = nCx * Px * (1 - P)n - x
What is the Probability Coin Flip Nostradamus Predicts
at least 3 of 4 Coin Tosses ?

18. Coin Flip Nostradamus:
Binomial Distribution
(
4!
)
3! (4-3)! (.53) (.54-3)
(.125) (.5)
( 24
Here is the Prob of
Getting Exactly 3 of
4 correct
6(1) ) = .25

19. Coin Flip Nostradamus:
Binomial Distribution
(
4!
)
3! (4-3)! (.53) (.54-3)
(.125) (.5)
( 24
Here is the Prob of
Getting Exactly 3 of
4 correct
6(1) ) = .25
We Want “At Least” Which Implies BOTH 3 and 4
= .3125
.25 + .0625
Exactly 3 Exactly 4 at least
3 of 4 Coin Tosses

20. Coin Flip Nostradamus:
Binomial Distribution
If Our Would Be Coin Flip Nostradamus were able to
get 3 out 4 Correct - we would not generally be
prepared to give him/her credit just yet
Namely, there is a 31.25% Probability that by
Chance he/she would be able to predict at least
3 out of 4

21. Coin Flip Nostradamus:
Binomial Distribution
Now We Can Calculate Probability Associated of
Prediction across some arbitrary number of trials
How Much Do We Need to Be Convinced that Our
Friend is Actually Coin Flip Nostradamus?
This is a Question of Type I and Type II
Error

22. Type I v. Type II Error

23. Type I
v.
Type II
Error

24. Type I v. Type II Error
Typical Convention is that a 5% Chance of Error is
Acceptable for Purposes of Statistical Significance
It is Depends Upon the Application
Social Science = 5%
Medicine with Serious Side Effects might Require
Greater Level of Significance 1% or even less

25. Back To
Coin Flip Nostradamus
Okay let say Our Coin Flip Nostradamus agrees to run
75 coins flips in order to demonstrate his/her true
powers
Predicts 43 out of 75 Correct
Is this Sufficient to Label Our Friend the
Coin Flip Nostradamus?

26. Binomial Probability Calculator
http://stattrek.com/tables/binomial.aspx

27. Binomial Probability Calculator
http://stattrek.com/tables/binomial.aspx
Enter
These
Three
Values
+
Hit Calculate

28. Binomial Probability Calculator
http://stattrek.com/tables/binomial.aspx
And
These are
the Results
Our P
value
Here is
12.4%

29. Coin Flip Nostradamus
Our P Value is the Probability of Observing this Data
Given the Null (i.e. that our friend does not have psychic
powers)
In this Case, the P Value is
Our Pvalue > 5% Statistical
Significance Threshold
“Fail to Reject” Our Null of No Psychic Powers
(We Do not Say Accept -- see the induction problem)

30. One Tailed -or-
Two Tailed Tests
There is a Difference Between a Directional and a Non-
Directional Hypothesis
In the Coin Flip Nostradamus Example it would be
amazing if our friend could actually fail to predict 75
consecutive events
Note:
These are
Symmetric

31. One Tailed -or-
Two Tailed Tests
We are Often Interested in a Non-
Directional Hypothesis
Stricter Crime Law and the Crime Rate
We are Interested in Whether there is
Deterrence and if there were to be higher
crime rates
New Drug and Health
We Want to Both if It Makes the Patient Better
and if the Patient’s condition get worse

32. One Tailed -or-
Two Tailed Tests
Two Tailed Test
One Tailed Test
(Positive direction)
One Tailed Test
(negative direction)

33. An Example of a
Hypothesis Test
Note: π is Prob
α is the Significance Level
https://onlinecourses.science.psu.edu/stat500/book/export/html/43

34. An Example of a
Hypothesis Test
Note: π is Prob
α is the Significance Level
Want to Make Sure
Sample is Large
Enough
https://onlinecourses.science.psu.edu/stat500/book/export/html/43

35. An Example of a
Hypothesis Test
Note: π is Prob
α is the Significance Level
Want to Make Sure
Sample is Large
Enough
If you Do Equal vs. Does
Not Equal --
Two Tail
https://onlinecourses.science.psu.edu/stat500/book/export/html/43

36. An Example of a
Hypothesis Test
z = (p - P) / σ
where p is our sample prov
P is theorized population prob
σ is our Standard Deviation
https://onlinecourses.science.psu.edu/stat500/book/export/html/43

37. An Example of a
Hypothesis Test
https://onlinecourses.science.psu.edu/stat500/book/export/html/43

38. Another Example Question
I roll a single die 1,000 times and
obtain a "6" on 204 rolls.
Is there significant evidence to
suggest that the die is not fair?

39. Another Example Question