3. Empirical design
Paths to take:
Secondary data & large sample analysis
Primary data & large sample analysis
Primary data & small sample - qualitative analysis
Combination of the above
4. Common student traps:
Developing many new pieces at the same time
(theory, data, method, measures).
Having beautiful and polished theory and starting to
collect data very late.
….and end up with theory which cannot be tested or cannot
be tested well.
Be “too” data/context-driven.
Hand-collected and novel/interesting but poor
dataset.
Suffering from mistakes made early.
Expending all energy in one area (e.g. identification
at the expense of theory).
5. How to avoid the traps?
Limit the number of novel components
Start data collection and analysis early (year 2-3) but
be open to restart the process.
Build/obtain a strong dataset!
Theory building and empirical testing is an iterative
process
The data will limit your degrees of freedom in the
theorizing
Match theory to data – measures need to reflect the
mechanisms
6. How to avoid the traps?
Measure twice cut once
Very path dependent process – choices that you make
early in the process, when you lack skills, may haunt you
later.
Be careful and spend a lot of time in the design stage.
Obtain feedback from different people.
Stay flexible. Be open to revising
theory/design/data/method after feedback (welcome to
the review process).
“Hourglass” approach
Start broadly (focus on a high level theoretical
relationship)
Narrow down to 1-2 key variables
Extensively analyze and obtain a set of well identified
baseline results
7. Methods: endogeneity and the “identification
revolution”
Do not despair!
Even though the identification is critically important, it
should not be at the expense of a theoretical
contribution.
Be explicit about the endogeneity concerns and
alternative explanations.
Understand the patterns in the data and be creative.
Good identification doesn’t necessarily mean an
instrument or a natural experiment.
8. …and feel free to violate any of these rules if you feel
that it is the right thing to do.
Because…
10. Which theory to test?
Tradeoffs in theory building
Verbal theorizing
Ease of
empirical
implementation
(Agent-based)
simulations
Analytical (math) equilibrium-
based models
Formalization
11. Which theory to test?
Tradeoffs in theory building
Ease of
empirical Methodological
implementation progress
Formalization
12. Which theory to test?
Tradeoffs in theory building
Ease of
empirical
implementation
Level of abstraction
Formalization
Editor's Notes
What I am going to tell you is not all based on large sample evidence (evidence based science). Some of these are conjectures based on observation.
Over focus on the context and data.Superb method but testing trivial relationships.
Sample size and selection, quality measures, all relevant controls, etc. Good measures.Very costly investments – want to leverage them across projects. Need to be done right!
Sample size and selection, quality measures, all relevant controls, etc. Good measures.Very costly investments – want to leverage them across projects. Need to be done right!
I focus on X because there is a good instrument. This will lead to finding a well identified trivial relationships.Supplement the empirics with ruling out alternative explanations with logic.I.e. there is endogeneity.
Sometimes referred to as rigor vs. relevance tradeoff.This is essentially a flexibility-constraints tradeoff.Less flexibility means better internal consistency that comes with formalization.Another tradeoff is in terms of specificity/generality (determines relevance).
ABM More degrees of freedom than formal models and fewer degrees of freedom than verbal theorizing. In turn, you gain rigor and internal consistency.
ABM More degrees of freedom than formal models and fewer degrees of freedom than verbal theorizing. In turn, you gain rigor and internal consistency.More distant from the phenomena – Higher abstraction – greater potential for theoretical contribution. And better generalizability.
