1. UNIVERSITY OF CASSINO AND SOUTHERN LAZIO
Department of Economics and Law
Master Degree in
GLOBAL ECONOMY AND BUSINESS
Thesis in Applied Statistics
Measuring the unobservable:
a counterfactual analysis of the
Italian Credit Guarantee Scheme
Supervisor Student
Prof. Giovanni Porzio Sara Fornabaio
2. Measuring the
unobservable
What would have happened
to the treated
without the treatment
• What we mean by
‘counterfactual’
• The potential outcome
model
• The selection bias
• Strategies to reduce the
selection bias
• A case study: the Italian
Credit Guarantee Fund
The main question in
evaluation is
3. The counterfactual
impact evaluation
Counterfactual methods are essential to quantify
the sign and size of the effect of the policy on
some outcome (Y) of interest
The ‘counterfactual logic’ is based on the
notion of causal effect of an intervention
(impact), defined as the difference between
the outcome observed after an intervention
has taken place
and
the outcome that would have occurred to
the same unit in the absence of the
intervention
An impact evaluation is a study which tackles
the issue of attribution by
identifying the counterfactual value of Yi (Y0i)
in a rigorous manner
• i = target unit
• Yi = outcome of interest
• Y1i= outcome after
intervention
• Y0i= outcome in absence of
intervention
• Di = treatment dummy
variable (0,1)
IMPACT = Y1i - Y0i
Y1i is observed
Y0i can only be estimated
4. It was developed by Rubin in the 70s
Impact estimated through comparison of
potential outcomes defined on the same
unit
• overall outcome =
• outcome for the control =
• outcome for the treated =
• Treatment Effect for unit i =
it can be estimated on average,
sometimes as
• Average Treatment Effect =
or as
• Average Treatment Effect on
Treated =
The potential outcome
model
Any empirical study of treatment effects
would typically start with simple
comparisons between a group of non-
treated ones (control group) and
a group of treated individuals
In order to make them comparable,
we need to remove the
selection bias
the differences between groups due to
unobservable pre-treatment characteristics
5. RANDOMIZATION
is the best strategy to avoid
selection bias
The selection bias
The observed difference between
treated and not treated is always
equal to the impact of the treatment
plus the ‘initial differences’
Observed difference = impact + bias
• Impact = E(Y1i|D=1) – E(Y0i|D=1)
factual – counterfactual
• Bias = E(Y0i|D=1) – E(Y0i|D=0)
E(Y0i|D=1)
E(Y1i|D=1)
Random assignment to treatment (intervention)
• accounts for unobserved initial differences
• makes the outcome of the control group a good
counterfactual for the treated group
NOT FEASIBLE IN OBSERVATIONAL STUDIES
INTENDED TO EVALUATE PUBLIC POLICIES
Selection bias has to be
removed (or reduced) to get
consistent estimators
6. Instrumental variables: exploits
situations that are similar to a
randomized experiment through an
instrumental variable z
Differences in differences: estimates the
impact (ATT) as difference in the change
of average outcome between treated and
control pre and post treatment
Propensity Score Matching: selects the
control group for the treated on the
Propensity Score, a balancing score of
observed pre-treatment characteristics
and estimates ATT
Strategies to reduce
the selection bias:
three estimators
OBSERVATIONAL STUDIES
• are usually ex-post evaluations
• data are already collected
• data need to be manipulated
COUNTERFACTUAL
ESTIMATORS
statistical and econometrical strategies to
mimic random assignment to treatment
• they strongly depend on available
data
• they can be combined
7. A case study: the
Italian Credit
Guarantee Fund
• Established within MISE in 1996
• Operational since 2000
• Targeted to SMEs (10-250 empl.)
• Grants partial public guarantee
on bank loans up to 90%
• Financed by public resources and
applicants’ fees
• 566.231 approved applications so
far
The counterfactual evaluation
The measured impact is the
financial additionality
the amount of extra bank debt borrowed
by guaranteed firms with respect
to non guaranteed ones
Treated firms
Firms who actually received the guarantee
Non treated firms
Firms whose application has been rejected
plus firms with similar characteristics who
never applied to the Fund
Period considered
From year 2000 to year 2006
The study has been developed during a
4 months internship at ISTAT
using the STATA software
8. The dataset: ‘no
causation without
manipulation’
Original datasets
• 471,050 applications submitted
to the Fund over the years
2000-2015
• Balance sheet data of 11,261
firms over the years 1999-2006
Tutorial datasets
• 9,421 applications (submitted by
7,207 firms) over the years
2000-2015
• Balance sheet data of 11,261
firms over the years 1999-2006
Step 4
Impact evaluation through 3 different
estimators
Step 1
Collapsing data from application level
to firm level
(‘one firm, one row’)
Step 3
Merging applications with financial data
Step 2
From cross-sectional
to panel data
10. Conclusions
• A rigorous evaluation requires hard work on data
• The counterfactual approach allows to measure the impact of
the intervention on beneficiaries
• The size of the impact should be a guideline for policy makers
• It is extremely difficult to communicate results when they are
not positive
• Impact evaluation can lead to efficiency oriented policies
