1. P-values alone don't determine the importance of research findings; substantive significance is also important but subjective. 2. Formalizing how substantive significance is measured can help communicate results and inform decisions based on evidence. 3. The speaker proposes using a kinked linear utility function with a parameter gamma to quantify how much more or less someone values avoiding false positives versus false negatives. This allows substantive significance to be modeled numerically.

1. Nathan H. Danneman
c* -- A new way to
communicate importance
Esarey and Danneman (forthcoming). A Quantitative Method
for Substantive Robustness Assessment. Political Science
Research and Methods.

2. The Point:
1. p-values don’t matter
2. Substantive significance DOES matter
3. Substantive significance is subjective
4. Formalizing substantive significance can
help us communicate, and make decisions
from evidence

3. But…in school we learned p-values
are really really important!!!

4. Suppose my friend invented a magic
weight loss pill...

5. ...but, it *might* cause cancer:

6. How did you choose whether or not
to take the drug?
Ostensibly, you weighed:
1. The evidence that the drug causes cancer
2. How bad it would be to get cancer
3. The evidence that the drug is safe
4. How good it would be to lose weight

7. How did you choose whether or not
to take the drug?
Ostensibly, you weighed:
1. The evidence that the drug causes cancer
2. How bad it would be to get cancer
3. The evidence that the drug is safe
4. How good it would be to lose weight
Letting r denote the empirical change in cancer
probability, you solved this:

8. Some of the inputs to your choice
are subjective:
The evidence that the drug causes cancer
-objective
How bad it would be to get cancer
-subjective
The evidence that the drug is safe
-objective
How good it would be to lose weight
-subjective

9. Implication
It is totally unsatisfactory to say things like:
“As civil war becomes more prevalent in the state’
s proximity, threatening the state with
externalities that could make civil conflict more
likely, the state increases its level of repression.
Substantively, we find that moving from a conflict-
free neighborhood (e.g., Switzerland in 2000) to
one riddled with civil conflict (e.g., Swaziland in
1982) generates a change of four-tenths of a
[repression measure]-unit.”
Danneman and Ritter (2014).Contagious Rebellion and
Preemptive Repression

10. Two problems
Problem 1: the utility acting on a false positive
might be different than the utility of acting on a
false negative.
Problem 2: the relative utility of these decisions
may vary by person (that whole “subjective
thing)

11. Solution: Part 1
Pick a utility function that can capture the relative weighting
the u(false positive) versus u(false negative).
Our preferred choice: a kinked, linear utility function:
Utility=0

12. Solution: Part 2
The amount of “kink” or “bend” in the utility function is
governed by a single parameter. Let’s call it gamma, such
that it’s square root represents the number of times a person
values false positives versus false negatives.

13. Step 3: Plot gamma against EU

14. That assumes you are willing to get
off the couch for E[u]=0
If the minimum amount
of benefit you are willing
to get off the couch for
is, say, c, then we can
plot gamma against c*
that solves E[u] - c = 0.
Too much work?
See our R package,
{cstar} on CRAN.

15. Thanks.
Nathan Danneman
@ndanneman
ndanneman@gmail.com
Esarey and Danneman (forthcoming). A Quantitative
Method for Substantive Robustness Assessment. Political
Science Research and Methods.