A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

1. H E A LTH RESU LTS IN NOVATION TRUS T FU N D
IE
and
HRITF
RBF
Workshop,
April
1,
2014
Milestones, Tools, Policy Requirements

2. Table
of
Content
• Milestones
– HRITF
EvaluaCon
PorEolio
– Milestones,
Funding
and
Deliverables
• Tools
and
Support
– Impact
EvaluaCon
Toolkit
– Annual
IE
Workshop
• Requirements
– World
Bank
clearance
of
publicaCons
and
papers
– Data
availability
– ReplicaCon
policy

3. HRITF
EvaluaCon
PorEolio
Type
and
Amount
per
Evalua2on
Number
of
CPG
Evalua2ons
Number
of
Stand-­‐alone
Evalua2ons
Total
Number
of
Evalua2ons
Impact
EvaluaCons
(up
to
$1.5
million)
26
6
32
Process
Assessments
(up
to
$400,000)
3
-­‐
3
Enhanced
Program
Assessments
(up
to
$400,000)
-­‐
1
1
QualitaCve
Assessments
(up
to
$250,000)
4
-­‐
4
Program
Assessments
(up
to
$75,000)
-­‐
5
5
Total
34
12
45
$51
million,
or
about
10%
of
the
HRITF
funding
is
commiXed
for
45
EvaluaCons,
which
includes
32
Impact
EvaluaCons
and
13
smaller
evaluaCon
studies.

4. Milestone,
Funding
and
Deliverables
• 5%
or
75,000
for
Concept
Note
preparaCon
Concept
Note
Approved
• 40%
or
$600,000
for
baseline
data
collecCon
Baseline
data
collected
and
uploaded
• 5%
or
$75,000
to
prepare
the
baseline
report
Baseline
report
delivered
• 45%
of
$675,000
for
end-­‐line
data
collecCon
and
report
End-­‐line
data
collected
and
uploaded
• 5%
or
75,000
final
payment
a]er
the
delivery
of
the
Final
report
Final
Report
delivered
Impact
Evalua2on
2meline
Total
funding:
$1.5
million
from
start
to
finish
Project
ImplementaCon
period
The
baseline
data
is
collected
before
project
start
The
end
line
data
is
collected
a]er
project
compleCon

5. Impact
EvaluaCon
Toolkit
on
the
Web

6. Sources
of
help
on
IE
Handbook:
why,
what
and
how?
Tools
of
the
trade

7. Where
does
this
toolkit
come
from?
• Developed
through
the
Health
Results
InnovaCon
Trust
Fund
(HRITF)
• SupporCng
results-­‐based
financing
approaches
• Focusing
on
health
MDGs
(1c,
4
and
5)
• Building
knowledge
on
results-­‐based
financing
– -­‐>
through
impact
evaluaCon
– -­‐>
through
other
types
of
evaluaCons

8. Impact
EvaluaCon
in
HRITF
• ProspecCve:
Built
into
program
design
• Robust:
Treatment
and
Control
groups
• Owned
by
country
teams
• Scaled:
36
Impact
EvaluaCons
at
different
stages
– 2
completed
– 17
ongoing
– 17
in
design
stage
(6
advance
&
11
early
design
stage)
• Toolkit
designed
to
support
teams
– But
useful
for
anyone
interested
in
impact
evaluaCon
8

9. RBF
specific
25%
Generic
75%
IE
toolkit
content

10. www.worldbank.org/health/
impactevaluaContoolkit

11. Module
2:
Building
the
Impact
EvaluaCon
Team
• “The
evaluaCon
manager
decided
that
surveys
do
not
deliver
any
credible
results.”
• “We
hired
local
consultants
(recommended
by
colleagues)
that
were
not
up
to
the
task.”
• “My
Research
Assistant
took
big
shortcuts.”

12. Module
3:
Designing
the
Impact
EvaluaCon
• “The
intervenCon
was
assigned
at
the
group
level,
but
the
data
were
collected
at
the
individual
level.
In
our
power
calculaCons,
we
forgot
to
take
into
account
this
clustering
of
individual
in
treated
group
–
we
ended
up
recruiCng
way
too
few
individuals
into
our
experiment.”

13. Module
6/7:
Storing,
Accessing,
Analyzing
Data
• What
happens
if
your
computer
crashes?
Do
you
loose
all
of
the
data?
• What
happens
if
your
RA
leaves?
Is
all
the
documentaCon
on
their
Macbook?
Will
anyone
be
able
to
replicate
her
analysis?

14. Module
8:
Monitoring
and
DocumenCng
RBF
Programs
• “We
had
a
randomized
design
but
didn't
keep
close
enough
communicaCon
with
the
government.
They
distributed
the
intervenCon
non-­‐randomly
without
our
knowledge.”
• “We
had
a
perfectly
good
(and
random!)
encouragement
design.
Then,
our
implemenCng
partner
started
bribing
the
parCcipants
to
take
up
the
treatment!”

16. This
toolkit
is
for
you..
Researchers
Survey
firms
Government
counterparts
&
In
country
IE
team
Project
leaders

17. A
living
tool
Toolkit
release
Use
of
guidelines
and
tools
User
feedback
Updates
to
guidelines
and/or
tools

18. Contact
us

19. Requirement: World Bank clearance of
publications and papers
• PublicaCons
(journal
submissions
and
working
papers)
need
to
be
submiXed
for
clearance
at
the
World
Bank.
• Clearance
by
WB
Country
director
if
paper
is
country-­‐specific
or
HNP
if
general.
• Applies
to
all
work
by
Bank
staff
or
consultants
+
any
paper
related
to
HRITF
funded
Impact
EvaluaCon
by
IE
team
members.

20. PublicaCons
• World
Bank
has
an
open
data
policy
encouraging
publicaCons
by
teams
linked
to
the
World
Bank
to
be
open
access.
• This
ensures
wide
disseminaCon
of
results.
• World
Bank
publicaCons
like
working
papers
are
also
widely
read
and
disseminated.

21. Requirement:
Data
availability
• All
data
from
the
Impact
EvaluaCon
will
need
to
be
uploaded
(part
of
the
milestones
process).
• Eventually,
the
data
should
be
made
available
to
the
outside
research
community.
• Time
frame
for
data
availability
is
2
years
a]er
data
collecCon
or
6
months
a]er
working
paper
is
out.
• Earlier
release
or
sharing
of
data
for
cross-­‐
country
analysis
is
encouraged.

22. Requirement:
ReplicaCon
policy
• With
their
data,
IE
teams
should
provide
complete
and
well-­‐documented
do-­‐files
allowing
replicaCon
of
their
results.
• The
do-­‐files
should
allow
going
from
the
raw
data
to
the
results
in
report.
• The
expectaCon
is
for
results
to
be
rouCnely
replicated,
for
example
by
HRITF
team.

23. QuesCons?
Comments?
Thank
you!