SlideShare a Scribd company logo
1 of 22
Download to read offline
EU Joint Programme - Neurodegenerative
       Diseases Research (JPND)

      Developing a framework for monitoring and evaluation

Joint Programming Conference 2013

Dublin, 28 February & 1 March 2013, Parallel Session 8
Edvard Beem, Co-director ZonMw, member of the Executive Board of JPND
This presentation

•   Rationales and objectives of JPND

•   Development of a monitoring and evaluation framework
    • Logical Framework Analysis as a systemic tool
    • Overview of the defined set of indicators of performance

•   Execution of monitoring to date

•   Key issues, challenges and lessons learned
Alzheimer’s Disease in Europe

                  40
                                                                     Number of Societal costs (€)
                                                                        cases
                  30                 Dementia                 2010    6,000,000    72,000,000,000
    Cases / 100




                  20                                          2040   12,000,000   144,000,000,000
                                           Stroke
                  10
                                          Parkinson's

                   0

                                                               A major societal challenge
                       60   70       80        90       100
                                 Age (years)

                                                                 for the coming years


From Rocca W et al, Eurodem
A major societal
« Grand Challenge »


   Energy

            Climate Change

                             Food & Health

                                             Neurodegeneration
Scope of JPND
                              Scientific
                              Scientific
                            • Animal models            Medical
                                                       Medical
                              Animal models
                            • Biobanks
                            • Biobanks               • Early diagnosis
                                                       Early diagnosis
  Focus on                                           • Prevention
                            • Cohorts/registries
                            • Cohorts/registries     • Prevention
Three Domains               • Disease pathology
                            • Disease pathology      • Clinical trials
                                                     • Clinical trials


                                          Social
                                          Social
                                        • Health care delivery
                                          Health care delivery
                                        • Home automation
                                        • Home automation
                                        • Health economics
                                        • Health economics
                                        • Ethics
                                        • Ethics
Diseases Targeted
 Alzheimer’s Disease and other    Parkinson’s Disease & related disorders
 dementias                        Huntington’s Disease
 Motor Neurone Disease            Spinocerebellar Ataxia (SCA)
 Prion Disease                    Spinal Muscular atrophy (SMA)
What are the JPND goals?

•   Increase coordinated investment in ND research aimed
    at finding causes of disease, developing cures, and
    identifying appropriate ways to care for those with ND

•   Our immediate goals:
    • Launch activities that add value to national research efforts
    • Leverage additional resources through partnerships
    • Link and support national plans with relevance for ND
Why JPI?

•   Awareness
    • common challenge, common vision
•   Mapping
    • Inventory running national programmes (database)
    • Overlaps and gaps, opportunities for joint actions
    • Baseline for monitoring
•   SRA (>10 years)
    • Roadmap for EU-wide investment
•   Implementation plan phase 1 2012-14
    • Joint actions: calls, enabling activities, action groups
CALLS
Total commitment 2011-2014
•   Pilot Joint Transnational Call 2011 = €15M

•   CoEN program call 2011 (€ 4M) and calls 2012-2014 (€12M)

•   Joint Transnational Call 2012
    •   “identification of genetic, epigenetic and environmental risk and protective
        factors” = €19M
    •   “evaluation of health care policy, strategies and interventions = € 10M

•   Estimate commitment 2013 = €25M
•   Estimate commitment 2014 = €25M

•   Total projected investment: over €100M in cash contribution
                                 (no in-kind resources included)
Enabling activities
JPND will execute the SRA within the next ten years through a combination
of:
 • Knowing our research                      • Capacity
   capability                                  building
 • Opportunities for                         • Education &
   infrastructure and platforms
                                               training
 • Working in partnership
   with industry                             • Connection to
 • Working with the regulators                 policy makers
 • Global partnership (third                 • Communication
   country collaboration)                      and outreach
Action groups
•   Action Groups to determine transnational research needs and
    opportunities in specific scientific areas
    • Longitudinal Cohort Studies (disease- & population-based)
    • Animal and Cell Models
    • Assisted Living Technologies
•   Action Groups to promote engagement, commitment and
    partnerships
    • Engagement and Partnership with Industry
    • Engagement and Partnership with the EC and beyond
    • User and Public Involvement in ND Research
    • Alignment of National Plans and activities
Why monitoring and evaluation?

Critical questions

•   Does a co-ordinated EU-wide approach makes a difference
    achieving the outcomes?

•   Will the outcomes of a concerted approach be more than the
    sum of the outcomes of national approaches?

•   Can we reduce fragmentation?

•   Can we recognize and learn from successes and failures?
Monitoring and evaluation framework
•   Analysis of the links between (societal) challenges,
    objectives and activities of a programme or initiative
    related to the expected effects and impacts
•   By mapping the intervention logic it will be easier to
    assess whether the objectives have been met and what
    the success criteria are
•   However it is “not as easy as it looks”
    • Different objectives and effects
    • Changing objectives
    • Unexpected effects
Logical Framework Analysis (LFA)

The logic:
•   A public action is undertaken for a reason (rationale)
•   It has objectives which address (societal) needs
•   It provides inputs which lead to activities
•   These activities create (direct) outputs
•   Which on their lead to outcomes/results and finally to
    (long-term) impacts
Intervention logic
                                                                            Mission
                                            Long-term
                                             Impacts

                                                                 Specific
                    Result                 Outcomes or            goals
                    chain                    results
        Programme
           logic
                                                         Operational
                                             Outputs       goals
Interventio
     n
   logic
                    Inputs                  Activities


                             Rationale or
                              reasons


                              (Societal)
                             challenges
Logical Framework of JPND
Indicators of performance

•   Based on the Logical Framework Analyse, indicators of
    performance have been developed:
    • Type A indicators monitor the effect of JPND on (European)
      research programming, research policy and funding (the concept of
      Joint Programming)
    • Type B indicators monitor the scientific and societal impact of
      research on neurodegenerative diseases conducted in the context
      of JPND
•   Distinction is made between:
    • Input indicators
    • Output indicators
    • Outcome indicators
    • Impact indicators
Development of indicators
Examples of type A indicators
•   Input indicators
    • Attitude towards JPND goals and objectives
    • The entry or drop out of countries
•   Output indicators
    • The national research (funding) priorities based on JPND’s Research
      Strategy
•   Outcome indicators
    • The amount of JPND common research funding for neurodegenerative
      diseases as share of total EU research funding
•   Impact indicators
    • The number of publications in high impact journals of European
      researchers in neurodegenerative diseases research
Examples of type B indicators
•   Input indicators
    • The number of collaborative research projects funded through JPND
    • The number of non-project funded activities
•   Output indicators
    • The number of Europe-wide population-based studies with contribution
      of JPND
•   Outcome indicators
    • Regular interactions between JPND and stakeholder groups
•   Impact indicators
    • Exchange of practices across different types of research (basic,
      clinical and healthcare)
Execution of monitoring to date
1.       Monitoring reports based on indicators
     •     Quantitative indicators based on desk study, information made
           available by other work packages
     •     Qualitative indicators obtained through a questionnaire
           investigating the attitudes towards JPND:
           •   Expectations regarding JPND
           •   Opinions on current JPND activities
           •   Views on the future of JPND

1.       External review of experts
     •     Review focusing on JPND’s governance and and processes of
           strategic policy programming (strategic direction) that have led
           to the development of the Research Strategy (SRA) and the
           implementation plan
Key issues, challenges and lessons
learned
•   Special attention should be paid to monitoring and evaluation as an
    important element to show progress and results of the initiative to
    various audiences

•   Development of monitoring and evaluation framework should be
    aligned with the development of the Research Strategy

•   Monitoring and evaluation is NOT just listing a set of indicators and
    start measuring, but takes some time and effort to develop according
    to intended objectives and achievements of the initiative

•   The systematic use of a Logical Framework Analysis approach is
    considered to be very helpful in (re)defining
    the objectives of JPND
More information

•   The full version of the monitoring and evaluation framework
    to be found on the website of JPND:

         http://www.jpnd.eu

The team responsible for monitoring and
  evaluation of JPND:

•   Edvard Beem       beem@zonmw.nl
•   Bastian Mostert   bastian.mostert@technopolis-group.com

More Related Content

Viewers also liked (8)

EPDA 20th anniversay conference (sep 2012) Research workshop slides
EPDA 20th anniversay conference (sep 2012)   Research workshop slidesEPDA 20th anniversay conference (sep 2012)   Research workshop slides
EPDA 20th anniversay conference (sep 2012) Research workshop slides
 
Adriana Maggi Month of the Brain conference II
Adriana Maggi   Month of the Brain conference IIAdriana Maggi   Month of the Brain conference II
Adriana Maggi Month of the Brain conference II
 
Philippe Amouyel, JPND Milan Nov. 2013
Philippe Amouyel, JPND Milan Nov. 2013Philippe Amouyel, JPND Milan Nov. 2013
Philippe Amouyel, JPND Milan Nov. 2013
 
Enda Connolly Slides, Washington DC, Nov. 2012
Enda Connolly Slides, Washington DC, Nov. 2012Enda Connolly Slides, Washington DC, Nov. 2012
Enda Connolly Slides, Washington DC, Nov. 2012
 
JPND Presentation P. Amouyel
JPND Presentation  P. AmouyelJPND Presentation  P. Amouyel
JPND Presentation P. Amouyel
 
Martin Rossor, Oxford UK, June 20th 2013
Martin Rossor, Oxford UK, June 20th 2013Martin Rossor, Oxford UK, June 20th 2013
Martin Rossor, Oxford UK, June 20th 2013
 
Philippe Amouyel slides - Washington DC, Nov. 2012
Philippe Amouyel slides - Washington DC, Nov. 2012Philippe Amouyel slides - Washington DC, Nov. 2012
Philippe Amouyel slides - Washington DC, Nov. 2012
 
JPND Research Strategy presentation T. Gasser
JPND Research Strategy presentation  T. GasserJPND Research Strategy presentation  T. Gasser
JPND Research Strategy presentation T. Gasser
 

Similar to JPI Conference Dublin - Edvard Beem - Evaluation and Monitoring Framework

Implementation of an Evaluation Model for Evaluating Complex Health Research ...
Implementation of an Evaluation Model for Evaluating Complex Health Research ...Implementation of an Evaluation Model for Evaluating Complex Health Research ...
Implementation of an Evaluation Model for Evaluating Complex Health Research ...
aihs
 
Master Economic & Consumer Psychology
Master Economic & Consumer PsychologyMaster Economic & Consumer Psychology
Master Economic & Consumer Psychology
LeidenUniversity
 
OECD, victor van rij, sept 2008 national horizons
OECD, victor van rij, sept 2008 national horizonsOECD, victor van rij, sept 2008 national horizons
OECD, victor van rij, sept 2008 national horizons
Victor Van Rij
 
Phd defence-pvalente
Phd defence-pvalentePhd defence-pvalente
Phd defence-pvalente
Pedro Valente
 
Performance Indicators in the Public Sector: Case Study
Performance Indicators in the Public Sector: Case StudyPerformance Indicators in the Public Sector: Case Study
Performance Indicators in the Public Sector: Case Study
Natalia Katasonova
 

Similar to JPI Conference Dublin - Edvard Beem - Evaluation and Monitoring Framework (20)

More Years, Better Lives JPI February 2015
More Years, Better Lives JPI February 2015More Years, Better Lives JPI February 2015
More Years, Better Lives JPI February 2015
 
Implementation of an Evaluation Model for Evaluating Complex Health Research ...
Implementation of an Evaluation Model for Evaluating Complex Health Research ...Implementation of an Evaluation Model for Evaluating Complex Health Research ...
Implementation of an Evaluation Model for Evaluating Complex Health Research ...
 
Utility and Added Value of Classifications in Health Information Systems
Utility and Added Value of Classifications in Health Information SystemsUtility and Added Value of Classifications in Health Information Systems
Utility and Added Value of Classifications in Health Information Systems
 
Master Economic & Consumer Psychology
Master Economic & Consumer PsychologyMaster Economic & Consumer Psychology
Master Economic & Consumer Psychology
 
OECD, victor van rij, sept 2008 national horizons
OECD, victor van rij, sept 2008 national horizonsOECD, victor van rij, sept 2008 national horizons
OECD, victor van rij, sept 2008 national horizons
 
Phd defence-pvalente
Phd defence-pvalentePhd defence-pvalente
Phd defence-pvalente
 
Philippe Amouyel - G7 Global Dementia Legacy Workshop, Tokyo, Nov 6th, 2014
Philippe Amouyel - G7 Global Dementia Legacy Workshop, Tokyo, Nov 6th, 2014Philippe Amouyel - G7 Global Dementia Legacy Workshop, Tokyo, Nov 6th, 2014
Philippe Amouyel - G7 Global Dementia Legacy Workshop, Tokyo, Nov 6th, 2014
 
Tackling societal challenges through digital transformation
Tackling societal challenges through digital transformationTackling societal challenges through digital transformation
Tackling societal challenges through digital transformation
 
European Chiropractor's Union - 2014 - Dublin
European Chiropractor's Union - 2014 - DublinEuropean Chiropractor's Union - 2014 - Dublin
European Chiropractor's Union - 2014 - Dublin
 
Riskilaste konverents 2012: Willy Tore Morch: Integrated services in primary...
Riskilaste konverents 2012: Willy Tore Morch:  Integrated services in primary...Riskilaste konverents 2012: Willy Tore Morch:  Integrated services in primary...
Riskilaste konverents 2012: Willy Tore Morch: Integrated services in primary...
 
Innovation og værdiskabelse i it-projekter
Innovation og værdiskabelse i it-projekterInnovation og værdiskabelse i it-projekter
Innovation og værdiskabelse i it-projekter
 
judith dyson collaborative launch
judith dyson collaborative launchjudith dyson collaborative launch
judith dyson collaborative launch
 
Providing Behavioral Resources to the Military Community: Mobile and Web-bas...
Providing Behavioral Resources to the Military Community: Mobile and Web-bas...Providing Behavioral Resources to the Military Community: Mobile and Web-bas...
Providing Behavioral Resources to the Military Community: Mobile and Web-bas...
 
World Health Organization on Health Information
World Health Organization on Health InformationWorld Health Organization on Health Information
World Health Organization on Health Information
 
Ken Guy, OECD: Science and Societal Challenges, Moving Ahead
Ken Guy, OECD: Science and Societal Challenges, Moving AheadKen Guy, OECD: Science and Societal Challenges, Moving Ahead
Ken Guy, OECD: Science and Societal Challenges, Moving Ahead
 
La théorie de la diffusion des innovations de Rogers et son application au c...
La théorie de la diffusion des innovations de Rogers et son application  au c...La théorie de la diffusion des innovations de Rogers et son application  au c...
La théorie de la diffusion des innovations de Rogers et son application au c...
 
mHealth Hub
mHealth HubmHealth Hub
mHealth Hub
 
Performance Indicators in the Public Sector: Case Study
Performance Indicators in the Public Sector: Case StudyPerformance Indicators in the Public Sector: Case Study
Performance Indicators in the Public Sector: Case Study
 
GIA Singapore - Introduction (Mulgan)
GIA Singapore - Introduction (Mulgan)GIA Singapore - Introduction (Mulgan)
GIA Singapore - Introduction (Mulgan)
 
Earma Presentation - Cora Hors (Brasil)
Earma Presentation - Cora Hors (Brasil)Earma Presentation - Cora Hors (Brasil)
Earma Presentation - Cora Hors (Brasil)
 

JPI Conference Dublin - Edvard Beem - Evaluation and Monitoring Framework

  • 1. EU Joint Programme - Neurodegenerative Diseases Research (JPND) Developing a framework for monitoring and evaluation Joint Programming Conference 2013 Dublin, 28 February & 1 March 2013, Parallel Session 8 Edvard Beem, Co-director ZonMw, member of the Executive Board of JPND
  • 2. This presentation • Rationales and objectives of JPND • Development of a monitoring and evaluation framework • Logical Framework Analysis as a systemic tool • Overview of the defined set of indicators of performance • Execution of monitoring to date • Key issues, challenges and lessons learned
  • 3. Alzheimer’s Disease in Europe 40 Number of Societal costs (€) cases 30 Dementia 2010 6,000,000 72,000,000,000 Cases / 100 20 2040 12,000,000 144,000,000,000 Stroke 10 Parkinson's 0 A major societal challenge 60 70 80 90 100 Age (years) for the coming years From Rocca W et al, Eurodem
  • 4. A major societal « Grand Challenge » Energy Climate Change Food & Health Neurodegeneration
  • 5. Scope of JPND Scientific Scientific • Animal models Medical Medical Animal models • Biobanks • Biobanks • Early diagnosis Early diagnosis Focus on • Prevention • Cohorts/registries • Cohorts/registries • Prevention Three Domains • Disease pathology • Disease pathology • Clinical trials • Clinical trials Social Social • Health care delivery Health care delivery • Home automation • Home automation • Health economics • Health economics • Ethics • Ethics Diseases Targeted Alzheimer’s Disease and other Parkinson’s Disease & related disorders dementias Huntington’s Disease Motor Neurone Disease Spinocerebellar Ataxia (SCA) Prion Disease Spinal Muscular atrophy (SMA)
  • 6. What are the JPND goals? • Increase coordinated investment in ND research aimed at finding causes of disease, developing cures, and identifying appropriate ways to care for those with ND • Our immediate goals: • Launch activities that add value to national research efforts • Leverage additional resources through partnerships • Link and support national plans with relevance for ND
  • 7. Why JPI? • Awareness • common challenge, common vision • Mapping • Inventory running national programmes (database) • Overlaps and gaps, opportunities for joint actions • Baseline for monitoring • SRA (>10 years) • Roadmap for EU-wide investment • Implementation plan phase 1 2012-14 • Joint actions: calls, enabling activities, action groups
  • 8. CALLS Total commitment 2011-2014 • Pilot Joint Transnational Call 2011 = €15M • CoEN program call 2011 (€ 4M) and calls 2012-2014 (€12M) • Joint Transnational Call 2012 • “identification of genetic, epigenetic and environmental risk and protective factors” = €19M • “evaluation of health care policy, strategies and interventions = € 10M • Estimate commitment 2013 = €25M • Estimate commitment 2014 = €25M • Total projected investment: over €100M in cash contribution (no in-kind resources included)
  • 9. Enabling activities JPND will execute the SRA within the next ten years through a combination of: • Knowing our research • Capacity capability building • Opportunities for • Education & infrastructure and platforms training • Working in partnership with industry • Connection to • Working with the regulators policy makers • Global partnership (third • Communication country collaboration) and outreach
  • 10. Action groups • Action Groups to determine transnational research needs and opportunities in specific scientific areas • Longitudinal Cohort Studies (disease- & population-based) • Animal and Cell Models • Assisted Living Technologies • Action Groups to promote engagement, commitment and partnerships • Engagement and Partnership with Industry • Engagement and Partnership with the EC and beyond • User and Public Involvement in ND Research • Alignment of National Plans and activities
  • 11. Why monitoring and evaluation? Critical questions • Does a co-ordinated EU-wide approach makes a difference achieving the outcomes? • Will the outcomes of a concerted approach be more than the sum of the outcomes of national approaches? • Can we reduce fragmentation? • Can we recognize and learn from successes and failures?
  • 12. Monitoring and evaluation framework • Analysis of the links between (societal) challenges, objectives and activities of a programme or initiative related to the expected effects and impacts • By mapping the intervention logic it will be easier to assess whether the objectives have been met and what the success criteria are • However it is “not as easy as it looks” • Different objectives and effects • Changing objectives • Unexpected effects
  • 13. Logical Framework Analysis (LFA) The logic: • A public action is undertaken for a reason (rationale) • It has objectives which address (societal) needs • It provides inputs which lead to activities • These activities create (direct) outputs • Which on their lead to outcomes/results and finally to (long-term) impacts
  • 14. Intervention logic Mission Long-term Impacts Specific Result Outcomes or goals chain results Programme logic Operational Outputs goals Interventio n logic Inputs Activities Rationale or reasons (Societal) challenges
  • 16. Indicators of performance • Based on the Logical Framework Analyse, indicators of performance have been developed: • Type A indicators monitor the effect of JPND on (European) research programming, research policy and funding (the concept of Joint Programming) • Type B indicators monitor the scientific and societal impact of research on neurodegenerative diseases conducted in the context of JPND • Distinction is made between: • Input indicators • Output indicators • Outcome indicators • Impact indicators
  • 18. Examples of type A indicators • Input indicators • Attitude towards JPND goals and objectives • The entry or drop out of countries • Output indicators • The national research (funding) priorities based on JPND’s Research Strategy • Outcome indicators • The amount of JPND common research funding for neurodegenerative diseases as share of total EU research funding • Impact indicators • The number of publications in high impact journals of European researchers in neurodegenerative diseases research
  • 19. Examples of type B indicators • Input indicators • The number of collaborative research projects funded through JPND • The number of non-project funded activities • Output indicators • The number of Europe-wide population-based studies with contribution of JPND • Outcome indicators • Regular interactions between JPND and stakeholder groups • Impact indicators • Exchange of practices across different types of research (basic, clinical and healthcare)
  • 20. Execution of monitoring to date 1. Monitoring reports based on indicators • Quantitative indicators based on desk study, information made available by other work packages • Qualitative indicators obtained through a questionnaire investigating the attitudes towards JPND: • Expectations regarding JPND • Opinions on current JPND activities • Views on the future of JPND 1. External review of experts • Review focusing on JPND’s governance and and processes of strategic policy programming (strategic direction) that have led to the development of the Research Strategy (SRA) and the implementation plan
  • 21. Key issues, challenges and lessons learned • Special attention should be paid to monitoring and evaluation as an important element to show progress and results of the initiative to various audiences • Development of monitoring and evaluation framework should be aligned with the development of the Research Strategy • Monitoring and evaluation is NOT just listing a set of indicators and start measuring, but takes some time and effort to develop according to intended objectives and achievements of the initiative • The systematic use of a Logical Framework Analysis approach is considered to be very helpful in (re)defining the objectives of JPND
  • 22. More information • The full version of the monitoring and evaluation framework to be found on the website of JPND: http://www.jpnd.eu The team responsible for monitoring and evaluation of JPND: • Edvard Beem beem@zonmw.nl • Bastian Mostert bastian.mostert@technopolis-group.com