1.
Assessing the potential and progress of web-based feedback for quality improvement: an evaluation using Patient Opinion as a case study Deborah Davidson, University of Birmingham Olivia Joyner, Tavistock Institute Joe Cullen, Tavistock Institute

2.
What we will cover <ul><li>Focus of the evaluation </li></ul><ul><li>Methodology </li></ul><ul><li>Analytical framework </li></ul><ul><li>Theory of change modelling </li></ul><ul><li>Findings </li></ul>

3.
Context <ul><li>High Quality Care for All (DH 2008a) highlighted the importance of using service user experience for both monitoring and improving the quality of health care; </li></ul><ul><li>Number of reviews highlight the NHS complaints procedure as flawed, serving neither patients nor providers; </li></ul><ul><li>Compelling evidence to show that NHS complaints often become escalated, and positions entrenched as a result of poor initial handling of complaints by front-line staff or managers; </li></ul><ul><li>This leads to a protracted process and a dynamic of contestation and defensiveness, instead of learning and resolution; </li></ul><ul><li>The proactive use of feedback to inform and drive quality improvement is far from being achieved in the NHS. </li></ul>

4.
The assumption <ul><li>Other sectors and industries have developed vastly more sophisticated approaches to capturing and actively using patient feedback. A far more diverse set of platforms are used, e.g. greater use of real-time and near-real-time methods and proactive seeking of views; </li></ul><ul><li>The emergence of web-based platforms for patient feedback, such as the independent Patient Opinion, I Want Great Care and Healthtalkonline sites and the “official” NHS Choices feedback facility, appears significant, since the evidence from other sectors would suggest that such platforms may have the potential to overcome some of the key limitations of traditional feedback and complaints mechanisms and to lead more directly to tangible improvements in the quality of care. </li></ul>

5.
The commission: 5 research questions <ul><li>In what ways do current NHS complaints and feedback mechanisms fall short? </li></ul><ul><li>How might web-based approaches offer an opportunity to overcome these limitations? </li></ul><ul><li>In practice, how far are web-based systems actually overcoming existing limitations </li></ul><ul><li>What factors need to be present in a healthcare organisation for platforms to allow both user objectives and quality improvement objectives to be met? </li></ul><ul><li>Are there particular advantages or disadvantages to an independent as compared with an NHS-run feedback platform? </li></ul>

6.
Dual focus Individual facing: access and utilization (use by patients and public) System facing: leading to impact on quality (organisation subscribers)

7.
Research design <ul><li>SCOPING: </li></ul><ul><li>Reference group </li></ul><ul><li>Data audit </li></ul><ul><li>Literature review </li></ul><ul><li>Stakeholder </li></ul><ul><li>interviews </li></ul><ul><li>Methodology and </li></ul><ul><li>evaluation toolkit </li></ul><ul><li>BENCHMARKING: </li></ul><ul><li>Comparisons </li></ul><ul><li>analysis </li></ul><ul><li>Analytical </li></ul><ul><li>framework </li></ul><ul><li>System audit </li></ul><ul><li>FIELD WORK </li></ul><ul><li>On-line survey and individual Interviews with service users/patients </li></ul><ul><li>Impact evaluation through 5 case study sites tracking feedback through to implementation </li></ul>SYNTHESES & REPORTING <ul><li>DEVELOPMENT </li></ul><ul><li>Feedback </li></ul><ul><li>Developmental </li></ul><ul><li>workshop </li></ul>FINAL REPORT

8.
Socio-technical health systems <ul><li>Technical coding (Feenberg, 1992) </li></ul><ul><li>‘ those features of technologies that reflect the hegemonic values and beliefs that prevail in the design process’ </li></ul><ul><li>Contest of meanings </li></ul><ul><li>‘ Value embedded action systems’ (Cullen and Cohen, 2007) </li></ul><ul><li>‘ Immanence’ of health technologies </li></ul><ul><li>Questions established notions of ‘ownership’ and ‘empowerment’ </li></ul><ul><li>Need to capture ‘functional’ and ‘discursive’ attributes of the systems </li></ul>

9.
Functional Attributes <ul><li>Based on ‘instrumental’ properties of systems </li></ul><ul><li>Example: FDA classification framework for medical devices </li></ul><ul><ul><li>The service configuration provided (including tools used for collaboration) </li></ul></ul><ul><ul><li>The delivery platform (the infrastructure used to deliver health systems and services) </li></ul></ul><ul><ul><li>The key ‘knowledge domain’ covered </li></ul></ul><ul><ul><li>The revenue model adopted </li></ul></ul><ul><ul><li>The ‘knowledge production’ model adopted (as reflected in the nature of interactivity between stakeholders) </li></ul></ul><ul><ul><li>The ‘scenarios of use’ implemented (how the information/support is transmitted) </li></ul></ul><ul><ul><li>The ‘techniques’ used to promote interaction between actors (i.e. how feedback operates) </li></ul></ul><ul><ul><li>Holding techniques used to secure and retain audience interest </li></ul></ul>

10.
Discursive Attributes <ul><li>Decontextualisation – embededness of tools and functions in life world </li></ul><ul><li>Reductionism – user-centeredness and usability of functionalities </li></ul><ul><li>Autonomisation - availability for feedback and knowledge co-production functions </li></ul><ul><li>Positioning – degree to which users experiences are applied to technical functions </li></ul>

11.
Analytical Framework Research Questions Research Criteria Accessibility, Usability & choice Responsiveness Redress Independence Accountability Standards Change processes M&E Competence development Benchmarks Technical platform Service model Timeliness Governance Best practice Performance Functional attributes Response time User satisfaction SLA’s Training policy Discursive attributes Open-ness Knowledge co-production Participatory culture Values

12.
Implementing the Framework <ul><li>Triangulation </li></ul><ul><ul><li>Representing stakeholder voices </li></ul></ul><ul><ul><li>Multi-methodological (Surveys; Interviews; Focus Groups; Content/Discourse Analysis’ System Audit; Case Studies) </li></ul></ul><ul><li>Theory of change modelling (Weiss, 1995) </li></ul><ul><ul><li>Intervening variables </li></ul></ul><ul><ul><li>Evolving and immanent technologies </li></ul></ul><ul><ul><li>Discursive attributes </li></ul></ul><ul><ul><li>Involves embedding a theory of change and looking for causal pathways </li></ul></ul>

13.
Using the theory of change <ul><li>Theory of change maps: </li></ul><ul><ul><li>help to lay out the issues or problems a project or programme is hoping to address, the actions being taken, and how these will lead to the final objectives that it is set up to achieve; </li></ul></ul><ul><ul><li>provides a road map as to how the aims and objectives will be achieved </li></ul></ul><ul><ul><li>Useful to start with the initial issues to be addressed, then look at the hoped for impacts and work backwards from there through the other stages. </li></ul></ul><ul><ul><li>One important thing to think about is that each stage is a ‘prerequisite’ for the next. ‘What are the requirements necessary to achieve that goal’. </li></ul></ul>

15.
In what ways do current NHS complaints and feedback mechanisms fall short? <ul><li>System not centred on patients’ needs – lack of a customer focus, accessibility, flexibility and transparency </li></ul><ul><li>Most frequently raised issue was how the complaint was initially handled by the healthcare provider; </li></ul><ul><li>Poor communication and insufficient information; </li></ul><ul><li>Poor attitude of staff; </li></ul><ul><li>Perceived lack of fairness (independence) </li></ul><ul><li>Timescales and process very protracted </li></ul>

16.
<ul><li>Patients don’t want to complain, because this takes them into a quasi-judicial process that brings with it, contention. Instead, people prefer to give feedback to the service about their experience, to ensure that their bad experience doesn’t happen to others in the future </li></ul><ul><li>(Bark, et al, 1994; Vincent et al, 1994; Friele and Sluijs, 2006). </li></ul>

17.
How might web-based approaches offer an opportunity to overcome these limitations? <ul><li>Accessible, quick, easy to access and give feedback without having to go through a formal process </li></ul><ul><li>Experienced as responsive and perceived as independent </li></ul><ul><li>Potential for generating high volumes of traffic and data from people on a specific topic or theme; </li></ul><ul><li>Use can increase response rates; </li></ul><ul><li>Useful for gathering ongoing feedback over longer periods of time. </li></ul>

18.
Comparisons analysis <ul><li>Customer discussion forums </li></ul><ul><li>Market research communities </li></ul><ul><li>Democracy and transparency websites </li></ul><ul><li>Review and purchasing/booking websites </li></ul><ul><li>Content submission and rating </li></ul><ul><li>Multichannel contact centres </li></ul><ul><li>Social tools and services </li></ul><ul><li>Joining other forums </li></ul><ul><li>Elements looked at: business model; theory of change; drivers </li></ul><ul><li>for change; service model; awareness raising and engagement </li></ul><ul><li>approach: independence; traffic levels; and key success factors </li></ul>

19.
In practice, how far are web-based systems actually overcoming existing limitations?

20.
Impacts? Different aims and agendas? Individual facing: Wanting a voice, help others, access and utilization (use by patients and public) System facing: leading to impact on quality (organisation subscribers) Feedback platform: wanting to give service users a voice through technology- lead to impact on quality

21.
Web-based platform for patient feedback Rationale If there is sufficient real time feedback traffic in the public domain it will lead to change. (reputational systems theory) Issue it seeks to address Lack of opportunities for service users to relate experience/to be heard. Need for public, independent, transparent feedback platform. Activities Feedback website Local Delivery Subscribers Outputs Outcomes Impacts Website- provides opportunity to publicly tell a story. Power of web to share positive and negative experiences. Engage NHS organisations as subscribers. Provide training and guidance, access to posting feedback. Subscribers inform service users of the web based platform and encourage its use. Subscribers (and non subscribers ) receive email alerts when feedback has been given. Organisation responds to feedback in a timely and considered manner. Organisation passes on feedback to services. Improvements in quality of care take place. Praise leads staff to be open to learning when later feedback is negative.

22.
Individual Service User Level : actual experience Rationale Service users with voice = enhanced experience and satisfaction. Help others. Issue it seeks to address Lack of opportunities for service users to relate experience/to be heard Activities Feedback website Local Delivery Subscribers Outputs Outcomes Impacts Website- provides opportunity to write/tell story Local champion with commitment becomes a subscriber. Services inform service users of the web based feedback mechanism Or/and User finds mechanism on internet Individual stories/ feedback posted Feedback appears on public platform. User may receive a response from health organisation. Service user has satisfaction of voice being ‘heard’. May hear if changes have occurred.

23.
Subscriber level: Theory of Change map Rationale Services improved by having better quality assurance and feedback mechanisms More feedback mechanism choice for service users (greater accessibility) Need to fulfil patient experience and choice agendas. ‘Tick boxes’ Issue it seeks to address Inadequacy of complaints procedures re: encouraging learning rather than encouraging contestation, defensiveness and resistance. Little choice of independent feedback mechanisms. Greater patient choice.

24.
Activities Feedback Website Local Delivery Embedding quality Improvement Outputs Outcomes Impacts Director of Nursing receives email alert of feedback from website provider. Director of Nursing emails the post to the relevant nurses. Director of Nursing and Head of Midwifery are the only people who can respond. If negative it is sent onto Clinical Lead. No holding responses. But a Reply to all posts within a week. Champion for website. <ul><li>Records if feedback </li></ul><ul><li>was positive or </li></ul><ul><li>negative for each </li></ul><ul><li>service at </li></ul><ul><li>Management Group. </li></ul><ul><li>Not in Feedback </li></ul><ul><li>Handbook. Website </li></ul><ul><li>logo on their </li></ul><ul><li>website but no </li></ul><ul><li>description apart </li></ul><ul><li>from in giving </li></ul><ul><li>positive feedback. </li></ul><ul><li>lack of effective </li></ul><ul><li>and clear promotion. </li></ul>Does not feed into other feed back mechanisms or patient experience/ complaint reports. “ Doesn’t go any where”. Encourage Matrons to discuss feedback at Band 7 Meetings. If changes could be made the Director of Nursing will email staff asking for the change and checks on progress. Not recorded? If negative posts- patient will be asked to attend a meeting. Positive posts increase staff morale: “ professional pride and feeling of competence” Small changes will be made- Lack of recording of change and pathways to the Board.

25.
What factors need to be present in a healthcare organisation for platforms to allow both user objectives and quality improvement objectives to be met? Are there particular advantages or disadvantages to an independent as compared with an NHS-run feedback platform?

26.
Key findings <ul><li>Patient Opinion web-based feedback mechanism clearly provides for patients’ needs that were lacking in complaints systems (access, responsiveness and independence): </li></ul><ul><ul><li>Customer focussed </li></ul></ul><ul><ul><li>Accessible and flexible </li></ul></ul><ul><ul><li>Longevity to being public and transparent </li></ul></ul><ul><ul><li>Good initial handling by web based site </li></ul></ul><ul><ul><li>Good communication and information giving </li></ul></ul><ul><ul><li>Perceived to be fair and independent </li></ul></ul><ul><ul><li>Near real-time responses and straight forward process </li></ul></ul>

27.
Meeting service user needs <ul><li>Importantly, it enables service users to give feedback to the service about their experience (stories of experience) </li></ul><ul><li>Web based platform provides opportunity for others to learn about others’ experiences </li></ul><ul><li>Redress not within the boundary (or gift) of the individual-facing system to provide just remedies or ‘compensation’ </li></ul>

28.
Embedding quality improvement <ul><li>Accountability </li></ul><ul><li>Organisational learning </li></ul><ul><li>Standards </li></ul><ul><li>Change processes </li></ul><ul><li>Monitoring and evaluation </li></ul><ul><li>Training and development </li></ul>

29.
Key findings continued <ul><li>Little evidence that this mechanism leads to an embedded and systematic approach to improvement in quality of service and care, though it has the potential to lead to improvement in the quality of service and care; </li></ul><ul><li>Progress is slow </li></ul><ul><li>Love affair with technology may be more of a driver than responding to individual and system needs; </li></ul><ul><li>Mechanisms of control – less enabling and empowering than in comparator services </li></ul>

30.
<ul><li>Service model vulnerable as predicated on three ‘dependent variables’: </li></ul><ul><ul><li>Traffic (PO) </li></ul></ul><ul><ul><li>Subscriptions (PO) </li></ul></ul><ul><ul><li>Champions (subscribers) </li></ul></ul><ul><li>Vulnerable if even one of these isn’t present </li></ul>

31.
Embedding quality: processes <ul><li>Management and governance: transparent and involved; </li></ul><ul><li>Established and systematised information and processes; </li></ul><ul><li>Customer focused behaviours: promoted and actively used; </li></ul><ul><li>Capture and route feedback: channelled to appropriate person for action; </li></ul><ul><li>Acting on feedback is seen as ‘business as usual’; </li></ul><ul><li>Manage and maintain good customer relations with clients. </li></ul>

32.
Quality improvement in practice <ul><li>“ the real challenge is equipping organisations to be able to embed feedback into practice to improve services.” </li></ul>