Presentation on Measuring Capacity Building by Tine Verdonck, Bénédicte Fonteneau en Bart Criel for DPH Day 2012

1. Measuring capacity building:
mission impossible?
Tine Verdonck, Bénédicte Fonteneau
(HIVA) and Bart Criel
1

2. The subject of our discussions…
• Academic trias
– Research
– Teaching
– Service Delivery
• Service delivery: multiform
– Consultancy work overseas
– Variety of services delivered to (Belgian) general public, civil society
organisations, public health authorities (cfr. “steunpuntfunctie” DGD)
– Capacity building via Institutional Collaborations (ICs)
– …
2

3. Purpose of session
• Share views on place & importance of
Capacity Building (CB) as academic activity
• Reflect on ways/methods how to measure its
effect(s).
Hypothesis: complexity of CB not properly captured
by instruments currently used
3

4. Programme
• Intro (Bart): 10’
• Some data on our ICs (Tine): 10’
– Quick reactions: 5’
• Outcome Mapping (Bénédicte): 20’
– Quick reactions: 5’
• Most Significant Change (Tine): 5’
• Discussion on sample of MSC stories collected (Bart
and Tine): 30’
• Final thoughts (Bart): 5’
4

5. Some history
ITM from 1990 till today
• 1990: a small ENOV / URESP • 2012: a big Public Health
– Training (ICHD): main project Department
• “Critical mass” – Training (MPH HSMP-MDC):
– Research: mainly action- still important, but less
research geared to central
change, few publications – Research: explosion in
– Service delivery (incl. capacity publications, action-research
building) central in culture of marginalized
unit: – Service delivery (incl. capacity
• “Societal relevance” building)
• Relevance for decision- • Has “objectively” strongly
makers increased
• But less prominent in
departmental culture and
policies than in past
5

6. A “contradiction” in current ITM policy &
discourse?
• De facto, we have never done as much CB as today
• But ITM discourse…
– vàv outside world: “ITM is different from other academic
institutions”
– Internally: service delivery/capacity buidling is not a
priority, not a core task, to be phased out, “we are not an
NGO”
6

7. Evaluation / recognition
academic work at ITM – as of today
• Research (publications) +++, Teaching +/++, SD/CB
+/-
• Our (departmental) “weaknesses”
– Is there consensus within department on importance of
SD/CB as academic activity à part entière ?
– Voluntaristic discourse, but no instrumentarium / set of
indicators to measure effect of SD/CB
7

8. Capacity Building
ITM 2020
Working Group CB
8

9. What is capacity building about?
Something to strive for
Interwoven with research & training
Long-term process
9

10. Dimensions of capacity building
Scientific Managerial Institutional
capacity capacity capacity
Capacity
Building
10
Adapted from Patrick Develtere, HIVA/KUL

11. Capacity building: now and in the
future
From rich to poor North
TODAY
From knowledgeable to less knowledgeable
A finite process South
South South
FUTURE
Alliances between equal partners
Mutually beneficial
North North Long lasting
11

12. Our message
Powerful justification of ITM involvement in capacity building
 Ethical reasons: solidarity
 Pragmatic / opportunistic reasons: ITM needs strong
partners & “fields” to do research
 Technical reasons: ITM’s expertise in capacity
building, health and health care (“excellence”?)
ITM and capacity building after 2020? Probably (still) the
case, here and there, but with highly variable outlook
depending on context
12

13. A framework to analyse Capacity
Building
13

14. Capacity Building
Potter C & Brough R (2004)
Systemic Capacity Building: a hierarchy of needs
Health Policy & Planning; 19(5): 336-345
Definition?
– “There is a need for CB” is an as unuseful
diagnosis as “the patient is feeling unwell”
– Consultant-speak
– Over-pompous synonym for training
–…
-> Lack of clarity
14

15. Capacity building (follow.)
Potter & Brough (2004)
“Enable programme execution independent of
changes of personalities, technologies, social
structures and resource crises”
“Develop sustainable and robust systems”
15

16. Systems Capacity:
a hierarchy of needs
Potter & Brough (2004)
Tools
enable Skills
effective require …
use of… Staff & Infrastructure
Structures, Systems and
Roles
Capacity Pyramid
16

17. Systems Capacity:
a hierarchy of needs
Potter & Brough (2004)
“Easier” and
more
technical Tools
Skills
Staff & Infrastructure
“Harder” and
more socio- Structures, Systems and Roles
cultural
Time to implement change
17

18. Indicators of the Framework Agreement

19. Project cycle management - indicators
• How do we verify success?
• Describe the objectives in operationally
measurable terms (quantity, quality, time)
• Basis of monitoring and evaluation system

20. Our indicators
• Framework Agreement
• Subprogramme: Institutional Collaboration
• Period: 2008-2010
• Total 381; objectives 57 and results 314
• Per project; range 10-61 and mean 25

21. Domain of FA indicators
Research
Training
Services/GRIPP/Networking
Management/Supporting
services
PI from Department Public Health PI from other Department

22. + output in numbers
People
• Master and PhD scholarships awarded and degrees obtained
• Short course scholarships awarded and certificates issued
• Exchanges and missions
• ...
Publications
• Publications in international peer-reviewed journals; other publications
• Theses (master and PhD)
• ...
Products
• CD-ROM, digital atlas, web-based course, online-modules, website
• Patent: application or obtained; test/method developed
• ...

23. Publications Framework Agreement
• Sources of references: reports of FA 1, 2, and 3
• Including only PubMed publications
• Removing duplicates
729 publications

24. Number of publications per year

25. Nr of publ. per year according to the
affiliation address of the first author

26. Tag cloud of key words

27. Questions raised
• Relevance in partner countries?
• Who determines research agenda? Who does
what?
• What happens if collaboration stops?
• Are publications a good indicator of capacity
building?

28. Level of FA indicators in hierarchy
Theoretical model Framework agreement (IC; FA3-I)
“Easier” and
more technical
Tools
Skills
Staff & Infrastructure
“Harder” and
more socio- Structures, Systems and Roles
cultural
Time to implement change

29. Formulation of FA indicators:
examples of “activity done”
• Quantité et qualité des ressources documentaires
acquises par le centre de documentation
• Routine testing for second line drugs standardised (2012)
• Gene mutations related to resistance investigated (2013)
• Work-based health systems fellowship programme fully
established
• One dissemination workshop/year for local MOH staff

30. Formulation of FA indicators:
examples of “changing influence”
• Publications describing the positive impact of measures
recommended by XXX to other national institutions
• Policy guidelines take into account results of studies
conducted at XXX
• XXX attracts a cadre of nationally renowned faculty
• Level of satisfaction of the health service managers with
alumni performance and initiatives has increased
• Le nombre de cadres gestionnaires formés à XXX et
exerçant des fonctions de gestion de haut niveau dans le
système de santé

31. Formulation of FA indicators
Activity done
Changing influence

33. Most significant change
An experiment

34. Steps
1. Collecting stories (done)
2. Selecting most significant of stories (now)
3. Feeding back results of selection (later)

35. Most Significant Change Technique
• What? Making sense of complex information;
participatory; qualitative
• When to use? Complex participatory
programmes; emergent outcomes; social change
• When not to use? Accountability reports, public
relations, to capture expected change
• Characteristics? Identifies values; bridges
cultures; analysis > data collection; rich picture

36. Best illustration of what we want to achieve?
1. PhD student learned to analyse data
2. Invitations to participate in consultancy & reflection
3. Launch of new training programme
4. PhD student involved in grant writing
5. Positive feedback on newsletter
6. PhD student submitted PhD proposal
7. Organization of new training programme
8. Inauguration alumni association
9. We got access to data on births and deaths
10. PhD student – discussion about draft manuscript

37. MSC: group discussion
• Reading out of 10 “stories” recently collected
• Discussion
– Which story is a good / the best illustration of what we
wish to achieve with CB?
– Why?
– Is this currently captured by the current evaluation
instrumentarium that is used?
37

38. 38

39. Final thoughts
39

40. Systems Capacity:
a hierarchy of needs
Other
PCM/
methods
“Easier” and Logframes
more (OM, MSC)
technical Tools
quant
Skills
Staff & Infrastructure
“Harder” and
more socio- Structures, Systems and Roles qualit
cultural
Time to implement change
40

41. Way forward?
• Evaluation questions: measure not only what we do
(=production), but also what we induce (=change in
processes)
• More room for “soft” things: values, trust, affect, self-
confidence,…
• As a department, familiarise with other methods that
complement PCM/Logframe
• Introduce this enriched evaluation framework in forthcoming
FA4
41

42. 42

43. Statements put forward to Director
• 5 statements on Service Delivery / Capacity
Building
• Scaling 1 (strongly disagree) to 5 (strongly
agree) by “A” & “B”
43

44. A: 3B: 3
• Stelling 1. Dienstverlening aan ontwikkelingslanden -
waaronder onze activiteiten mbt institutionele
samenwerking en steunpuntfunctie - zit ingebakken
in het DNA van het ITG en haar personeel. Het was/is
bepalend in de keuze van velen om op het ITG te
komen werken
44

45. A: 1B: 1
• Stelling 2. Gegeven stelling 1, zou het van slecht
personeelsbeleid getuigen om deze activiteiten niet
als volwaardig te erkennen en te valoriseren.
45

46. A: 4B: 4
• Stelling 3. De gangbare methodieken om
dienstverlening/institutionele samenwerking te
meten / evalueren slagen er onvoldoende in om de
complexiteit en de (eventuele) impact van deze
activiteit te capteren.
46

47. A: 4B: 5
• Stelling 4. Het ITG moet als ambitie hebben om naast
een "Centre of Excellence" in onderzoek (en
onderwijs), ook een “Centre of Excellence” te zijn in
het domein van de dienstverlening/ institutionele
samenwerking.
47

48. A: 2B: 1
• Stelling 5. De huidige druk in onze samenleving op
academische instellingen om steeds meer en sneller
meetbare outputs (“productie”) te bereiken in hun
onderzoek, gaat ten koste van de maatschappelijke
dienstverlenende rol van deze instellingen.
48