Evaluation of Integrated Care: From methods to governance and applications - Economics of eHealth

5,528 views
5,420 views

Published on

Recent Developments and Future Challenges of Integrated Care in Europe and Northern America

The International Network of Integrated Care, The Julius Center of the University Medical Center Utrecht and the University of Southern Denmark - 11th International Network of Integrated Care

Evaluation of Integrated Care: From methods to governance and applications

To understand the real impact of ICT within health care adopting a single analytical approach is inadvisable and that insight into the overall effects of ICT is best gained from consideration of a mix of study types

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
5,528
On SlideShare
0
From Embeds
0
Number of Embeds
3,876
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Although relatively few studies have been done to date, the field is challenging from a methodological point of view, offering analysts a wealth of opportunities. Guidance to realise these opportunities is provided by the general principles for economic evaluation, which can be tailored to the requirements of this particular field.
  • Case studies reported by OECD (2010) stated that ICT implementation benefits could be grouped according to four inter-related categories of objectives: (1) Increasing quality of care and efficiency; (2) Reducing operating costs of clinical services; (3) Reducing administrative costs and (4) Enabling entirely new modes of care. Within this context OECD (2010) claimed that Governments could provide motivation for high-performing projects through targeted incentives and also occupied a central position as initiator, funding provider, project facilitator, and neutral convener, playing a special role to encourage the utilization of standards to reach a common goal. Furthermore, the main findings of this study could be summarised as follow: (1) Establish robust and coherent privacy protection; (2) Align incentives with health system priorities; (3) Accelerate and steer interoperability efforts; and (4) Strengthen monitoring and evaluation.
  • Brynjolfsson, E., L. Hitt, and Sloan School of Management. Center for Information Systems Research., Computers and economic growth : firm-level evidence . CISR Working Paper. 1994, Cambridge, Mass: Alfred P. Sloan School of Management, Massachusetts Institute of Technology. 43 p. 44. Brynjolfsson, E., et al., Is information systems spending productive ? : new evidence and new results . Sloan Working Paper No.3571-93. 1993, Cambridge, Massachusetts: Center for Information Systems Research, Sloan School of Management, Massachusetts Institute of Technology. 32p.
  • , even though the centrality of information exchange in the care process and its usefulness in management, accountability, research and financial transaction (Street, 2007).
  • A particular problem in health sector is that there is no measure of performance analogous to profits from private sector firms, and health care organisations tend to pursue multiple objectives. Furthermore, ICT implementation may have effects that are multidimensional and often uncertain in their reach and scope, and difficult to control. In addition, the realisation of benefits from ICT implementation strongly depends on contextual conditions (Street, 2007). On the one hand, these difficulties are further exacerbated by data limitations, definitional problems and lack of appropriate sets of indicators on adoption and use of ICT comparison. On the other hand, dimensions related with measurement errors, time lag, redistribution and mismanagement of ICT are being pointed out within the application of “productivity paradox” into health care. E. Brynjolfsson, The productivity paradox of information technology: review and assessment, Commun. ACM 36 (12) (1993) 67–77. E. Brynjolfsson, L.M. Hitt, Beyond the productivity paradox, Commun. ACM 41 (8) (1998) 49–55.
  • To understand the real impact of ICT within health care adopting a single analytical approach is inadvisable and that insight into the overall effects of ICT is best gained from consideration of a mix of study types (Street, 2007). Empirical studies into the impact of ICT could be grouped into four broad categories: (1) Aggregate analyses that take a macro perspective by looking at the economy as a whole; (2) Industry or sectoral level analyses that focus on specific industries or sectors within the economy; (3) Firm or organisational-level analyses and (5) Case studies that focus on specific examples of ICT (Street, 2007).   Firm-level analyses could be grouped twofold. On the one hand, standard economic theory of the firm and regressions (econometric) models and, on the other hand, studies attempt to estimate the relative efficiency of organisations. These studies can be sub-divided into those that use the regression based technique called Stochastic Frontier Analysis (SFA) and those that employ a linear programming technique know as Data Envelopment Analysis (DEA).
  • Although relatively few studies have been done to date, the field is challenging from a methodological point of view, offering analysts a wealth of opportunities. Guidance to realise these opportunities is provided by the general principles for economic evaluation, which can be tailored to the requirements of this particular field.
  • Evaluation of Integrated Care: From methods to governance and applications - Economics of eHealth

    1. 1. The views expressed in this presentation are those of the author and do not (necessarily) reflect the position of the European Commission. Neither the Commission nor any person acting on behalf of the Commission can be hold responsible for the use which is made of this presentation Evaluation of Integrated Care: From methods to governance and applications Economics of eHealth 11th International Conference on Integrated Care (INIC11) April 1, 2011 Odense, Denmark Dr. Francisco Lupiáñez-Villanueva http://www.ictconsequences.net/ JRC – IPTS – IS Unit
    2. 2. Institute for Prospective Technological Studies Francisco Lupiáñez-Villanueva, PhD http://www.ictconsequences.net Information Society Unit European Commission, DG JRC Institute for Prospective Technological Studies (IPTS) Edificio Expo - Calle Inca Garcilaso, s/n E-41092 Seville - Spain http://ipts.jrc.ec.europa.eu [email_address] Phone +34 954 488 206 (direct) Fax +34 954 488 208 IPTS: Part of DG JRC of the EC: 7 Research Institutes across EU Mission: “to provide customer-driven support to the EU policy-making process by researching science-based responses to policy challenges that have both a socio-economic as well as a scientific / technological dimension” Modus operandi: desk research, expert groups, modelling, centres of Expertise, foresight
    3. 3. Economic evaluation of integrated care “ It is generally assumed that integrated care results in increased effectiveness and quality of care, while being cost-effective or even cost-saving at the same” “ There is a need for well-designed economic evaluation studies of integrated care arrangements, in particular in order to support decision making on the long-term financing of these programmes” “ A skilled health economist should be consulted in the early phase of the design of the study. Ideally, the health economist would be part of the research team from start to finish of the project”
    4. 4. Evidences Limitations: Available quantitative research was limited and was done by a small number of institutions. Systems were heterogeneous and sometimes incompletely described. Available financial and contextual data were limited. As this paper demonstrates, few economic evaluations of telemedicine can be trusted to provide reliable information for decision-making. The majority of the evaluations reviewed were not in accordance with standard evaluation techniques and still have a long way to go before decision-makers can rely on them to produce valid and reliable cost-effectiveness data.
    5. 5. “ Productivity Paradox” While information technology (IT) investments in healthcare are higher than ever, there are contradictions in the literature regarding their ability to improve productivity, quality of care and/or healthcare system efficiency.
    6. 6. ICT challenges within health systems <ul><li>ICT implementation benefits could be grouped according to four inter-related categories of objectives: </li></ul><ul><li>Increasing quality of care and efficiency; </li></ul><ul><li>(2) Reducing operating costs of clinical services; </li></ul><ul><li>(3) Reducing administrative costs and </li></ul><ul><li>(4) Enabling entirely new modes of care. </li></ul><ul><li>Main findings of this study could be summarized as follow: </li></ul><ul><li>Establish robust and coherent privacy protection; </li></ul><ul><li>(2) Align incentives with health system priorities; </li></ul><ul><li>(3) Accelerate and steer interoperability efforts; and </li></ul><ul><li>(4) Strengthen monitoring and evaluation. </li></ul>However, it is worth pointing out that this study mentioned an absence, in general, of independent, robust monitoring and evaluation of programmes and projects to determine the actual payoff from the adoption and use of ICT
    7. 7. Lack of consolidated evidence is main barrier for RMT
    8. 8. <ul><li>What if </li></ul>The Best is the enemy of the good Towards Randomize Control Trials But…
    9. 9. ICT as General Purpose Technology The term “general-purpose technology”, or GPT, has seen extensive use in recent treatments of the role of technology in economic growth, and is usually reserved for changes that transform both household life and the ways in which firms conduct business. Steam, electricity, internal combustion, and information technology (IT) are often classified as GPTs for this reason. They affected the whole economy.
    10. 10. A schematic of Health Economics
    11. 11. ICT characteristics Robert Solow famous quip that “You can see the computer age everywhere but in the productivity statistics “ was later systematised into the so called Productivity Paradox <ul><li>Subsequent research partially reverse the paradox as after the 1990 productivity resurgence was attributed also to ICT in macro-economic models, and micro-economics studies showed that ICT does increase the productivity of firms especially when occurred together with </li></ul><ul><li>re-organisation, </li></ul><ul><li>change management, </li></ul><ul><li>re-training of employees. </li></ul>Firms started to really leverage IT when they were fully capable also to capture and mine customers’ data and to use ICT to integrate the value chain both upstream (supply chain) and downstream (delivery), as well as to better connect with inter-organisational networks of cooperation
    12. 12. ICT characteristics <ul><li>SUBTITUTION EFFECT </li></ul><ul><li>COMPLEMENTARY EFFECT </li></ul><ul><li>LEARNING CURVE </li></ul><ul><li>ECONOMIES OF SCALE </li></ul><ul><li>ECONOMIES OF SCOPE </li></ul><ul><li>EXTERNALITIES </li></ul>
    13. 13. ICT complexity <ul><li>Due to the special characteristics of ICT market, the main barriers to ICT adoption in health sector: </li></ul><ul><li>low product differentiation, </li></ul><ul><li>high switching costs in replacing technologies, </li></ul><ul><li>lack of technical compatibility </li></ul><ul><li>explained why it lags behind other sectors in ICT adoption </li></ul>
    14. 14. ICT complexity A particular problem in health sector is that there is no measure of performance analogous to profits from private sector firms, and health care organisations tend to pursue multiple objectives. ICT implementation may have effects that are multidimensional and often uncertain in their reach and scope, and difficult to control. In addition, the realisation of benefits from ICT implementation strongly depends on contextual conditions These difficulties are further exacerbated by data limitations, definitional problems and lack of appropriate sets of indicators on adoption and use of ICT comparison. Dimensions related with measurement errors, time lag, redistribution and mismanagement of ICT are being pointed out within the application of “productivity paradox” into health care
    15. 15. Stakeholders STRATEGIC AND OPERATIONAL OBJECTIVES INPUT OUTPUT (treatment) OUTCOMES / IMPACTS EFFICIENCY (INPUT/OUTPUT) EFFECTIVENESS (COST/OUTCOMES) CODAGNONE (2011) ICT complexity Intervening variables, or confounders, covariates, etc (demographic, socio-economic, attitudinal, relational, etc) plus unobservable and/or ommitted variables Fundamental problem of causal inference and impact evaluation: observing me taking and not taking an aspirin: is not possible!
    16. 16. Canada Health Infoway Benefits Evaluation Indicators-Technical Report (2006) ICT complexity
    17. 17. HEALTH ICT: Indicators for international comparisons of health ICT adoption and use. (OECD, 2008) ICT complexity
    18. 18. To understand the real impact of ICT within health care adopting a single analytical approach is inadvisable and that insight into the overall effects of ICT is best gained from consideration of a mix of study types (Street, 2007). ICT complexity
    19. 19. Codagnote et al. adapted from Drummond et all, 2005 Economic evaluation
    20. 20. Mapping questions to approaches Codagnone, 2011 Question Approach Did we do what we planned? To what extent? Where we did it and where we did not? Evaluation as control, implementation effectiveness = expected output minus actual output (if zero then full effectiveness). Within the domain of Monitoring & Operational evaluation (M&OE). Mostly about data and indicators. Could we produce the same output spending less? Who spent less and who spent more and why? Evaluation as control, implementation efficiency= output/input. If intervention on multiple sites data from M&OE can be used for statistical multivariate analysis of input efficiency (i.e. Data Envelopment Analysis) Did we produce an effect contributing to reach the target outcome that can be attributed solely to our intervention? Impact Evaluation with counterfactual approach, outcome effectiveness = proved treatment effect Was the produced effect worth the costs? Impact Evaluation with counterfactual approach, outcome cost-effectiveness = proved treatment effect net of cost (or estimate of unit costs obtained as a net effect of intervention through counterfactual approach. For instance cost-per – QALY produced threshold used in the UK by NICE)
    21. 21. Important distinction <ul><li>Monitoring and Operational Evaluation (M&OE): </li></ul><ul><ul><li>Monitoring systems set up goals, indicators, and target outcomes for policy interventions </li></ul></ul><ul><ul><li>Operational evaluation, fed by the monitoring system and by additional data gathering activities (i.e. interviews with beneficiaries and with those in charge of implementation), seeks to understand whether implementation of an intervention (treatment) unfolded as planned </li></ul></ul><ul><ul><li>M&OE aim at improving policy design and implementation, by looking mostly at output and implementation processes </li></ul></ul><ul><ul><li>As part of M&OE it is possible also to compare the target outcomes with the actual outcomes after the intervention to identify gaps and inform policy making, but no scientific causal attribution of the realised outcomes to the intervention is possible as part of M&OE </li></ul></ul><ul><li>Impact Evaluation (IE) strictu sensu (I also refer to it as scientific evaluation) </li></ul><ul><ul><li>Is the systematic and scientific attempt to prove that changes in target outcomes (effects) are due only to the specific intervention being evaluated and not to some other causes </li></ul></ul><ul><ul><li>M&OE and IE are two very distinct activities with very different requirements, although they should ideally feed each other </li></ul></ul><ul><ul><li>Yet, IE is much more demanding, should be applied selectively and, at any rate, in many cases is not feasible for the conditions do not ensure scientifically robust and unbiased estimates of effects </li></ul></ul><ul><li>(*) See for instance Khander SR, et al, Handbook on Impact Evaluation: Quantitative Methods and Practice , The World Bank, Washington D.C, 2010, pp. 7-22 </li></ul>Codagnone, 2011
    22. 22. Economic evaluation of integrated care “ A skilled health economist should be consulted in the early phase of the design of the study. Ideally, the health economist would be part of the research team from start to finish of the project” REMEMBER
    23. 23. Thank you for your attention Francisco Lupiáñez-Villanueva, PhD http://www.ictconsequences.net/ JRC – IPTS – IS Unit

    ×