• Save
Assessing the potential and progress of web-based feedback for quality improvement: an evaluation using Patient Opinion as a case study
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Assessing the potential and progress of web-based feedback for quality improvement: an evaluation using Patient Opinion as a case study

on

  • 1,432 views

TIHR (Olivia Joyner + Joe Cullen) and HSMC (Deborah Davidson) presented emerging findings from the service evaluation on accessing the potential and progress of web-based feedback for quality ...

TIHR (Olivia Joyner + Joe Cullen) and HSMC (Deborah Davidson) presented emerging findings from the service evaluation on accessing the potential and progress of web-based feedback for quality improvement in the Health Service at the prestigious 7th Biennial Conference in Organisational Behaviour in Health Care in April 2010.

Statistics

Views

Total Views
1,432
Views on SlideShare
958
Embed Views
474

Actions

Likes
0
Downloads
0
Comments
0

5 Embeds 474

http://www.tavinstitute.org 432
http://tav.danielcrabbe.com 22
http://elmde.wordpress.com 17
http://translate.googleusercontent.com 2
http://www.tavinstitute.org.ej.t.hpi.transer.com 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • At a time when there is an increasing policy emphasis on patient experience, there is also increasing evidence that much of the NHS lacks effective systems for either gathering or acting upon patient feedback. This is due to a number of reasons including public cynicism about the impact of involvement (Smith, 2005); certain groups being significantly under-represented and the need for encouragement of Black and Ethnic Minority communities (Sheldon and Rasul, 2006); the need to use a range of methods as not all people have the same preferences (Ipsos MORI 2008, Entwistle et al 2003); and information developed for patients and the public is written at an above average reading ability, making it difficult for many people to understand (Coulter and Ellins, 2006). In addition, research suggests that the existing channels do not produce a satisfactory experience for those who wish to offer feedback or make a complaint. Since 1996, numerous reviews of the NHS complaints procedure have taken place, each highlighting problems and identifying improvements, for example ( Being Heard , 1994; NHS Complaints Procedure National Evaluation, 2001; Spotlight on Complaints, 2007; Is anyone listening? A report on complaints handling in the NHS, 2007; Feeding back? Learning from complaints handling in health and social care , 2008; Making Experiences Count , 2008; and more recently government policy has shifted its focus to a customer relations approach ( Listening, Improving, Responding: A guide to better customer care , 2009 ) . All of these reviews highlight the NHS complaints procedure as flawed, serving neither patients nor providers. Furthermore, they provide compelling evidence to show that in the NHS, complaints often become escalated, and positions entrenched as a result of poor initial handling of complaints by front-line staff or managers. This leads to a protracted process and a dynamic of contestation and defensiveness, instead of learning and resolution, and has led to rising litigation costs for the NHS ( Making amends. A consultation paper setting out proposals for reforming the clinical negligence system , 2005). Lastly, it seems to be clear that the proactive use of feedback to inform and drive quality improvement is far from being achieved in the NHS not just because of ineffective feedback mechanisms (Vingerhoets, 2001; Wensing, 2003; Davies and Cleary, 2005; Rogut and Hudson, 1995; and Tasa et al, 1996).
  • Five research questions were posed: On the basis of existing research, in what ways do current NHS complaints and feedback mechanisms fall short in meeting the needs of users, and in realising the potential of user feedback to drive quality improvement in the NHS?   In principle, how might web-based approaches offer an opportunity to overcome the limitations of existing systems for feedback? In particular, what might be the capacity of web-based feedback mechanisms to lead directly to improvements in the quality of care?   In practice, how far are web-based systems actually overcoming existing limitations and helping users to give feedback when, how and if they choose to do so and to secure any action by the NHS organisation where that was their motivation for providing feedback, and being used by NHS as a contributing or indeed a driving factor in their work to improve the quality of care? If this is happening, what is the mechanism by which it is happening within those organisations? (process evaluation)     What factors need to be present in a healthcare organisation for platforms like Patient Opinion to allow both user objectives and quality improvement objectives to be met? How far is the impact of such a platform within any particular healthcare organisation dependent on potentially vulnerable factors, such as key individuals or a favourable organisational context? How far is the causal chain between receipt of feedback and steps taken to improve the quality of care sustainable, strong and replicable across the constituent parts of each NHS organisation and between NHS organisations?   Is there any evidence that there are particular advantages or disadvantages to an independent as compared with an NHS-run feedback platform? How significant are the underlying values and positioning of the platforms in determining the value of different feedback sites to patients and NHS stakeholders? Is there any evidence of a differential impact on feedback as a driver of quality improvement?
  • Reflects a range of different evaluation ‘purposes’, balancing the need to demonstrate outcomes and impacts with the need to capture and apply learning in order to support the future development of patient review and service user feedback initiatives Incorporates a ‘theory of change’ model, aimed at identifying the explicit and implicit ‘visions’ of change that underpin these initiatives and reflecting these in evaluation methods and instruments uses a context-sensitive approach. An important challenge for the evaluation is to explore the importance of context – for example how different service user environments, involving different types of user – shape the use of such systems and their value Uses triangulation. This allows for the synthesis of evidence of different types and from different sources, and representing different stakeholder positions, in order to arrive at research conclusions. In practice, this means: multiple sources of data, including secondary data (drawn from existing studies and databases, and primary data (acquired, for instance, through case studies implemented by the evaluation); a multi-methodological approach - the research will include an element of quantitative analysis, together with qualitative data. Has an evolutionary focus and is aimed at applying the learning from the evaluation to supporting operational improvements and future strategic development
  • \\
  • 1. Identifying the assumptions behind them 2. Writing a narrative to explain the logic of the initiative. Connect the preconditions or requirements necessary to achieve that goal. i.e. without taking the actions you wouldn’t have the outputs, and without these outputs you would be able to achieve the outcomes or impacts.
  • TOC articulates the assumptions the feedback organisation, health organisation and the individual service user has in relation to giving or managing feedback and its impacts. External factors: policy agendas, funding, other feedback mechanisms- PALS, Complaints, NHS Choices. How they impact?
  • Cowan and Anthony (2008) suggest that there are two interlinked elements that lead to a decision to complain: The reason (what gave rise to the complaint) The motivation (the emotional response and expectations about what will be achieved) Reasons The HCC report suggests that issues have remained broadly consistent, with main concerns being the “fundamentals of healthcare, such as safe care and clinical treatment, and the essential elements of a considerate, customer-focused approach to complaints – such as communication and the way that the complaint was handled overall” (HCC, 2009:45) Motivation The key motivation for making a complaint is that people want to make sure that their bad experience doesn’t happen to others in the future i.e. making it better for the next person (Bark, et al, 1994; Vincent et al, 1994; Friele and Sluijs, 2006). The literature in general shows that a formal complaint is usually something that people do when they have exhausted other possibilities for redress - it is not a default option but a last resort (ref). This suggests that complaints are not being adequately well dealt with (e.g. by PALS, services) when initially raised , and this is leading them to escalate into something formal One third of trusts deal with complaints without assessing the expectations of the complainants Process April 2003: NHS complaints reform, making things right (DH, 2003) . The changes – although not radically different - set out their intention that the procedure be: open and easy to access fair and independent responsive provide an opportunity for learning and developing Friele and Sluijs (ibid) concluded that the dimensions of communication and outcome (change) are equally, if not more, important than the complaint itself.
  • Use of websites to provide feedback is quick, easy and convenient, though this method may be more appropriate for certain types of feedback such as general comments and opinions rather than concerns or specific issues.   Online communities are useful for generating large volumes of qualitative data from people on a specific topic or theme. They are quick and convenient and their use can increase response rates. This method is also useful for gathering ongoing feedback over longer periods of time.   These communities require a significant amount of management and maintenance however. Recruiting the right members will be an important issue for organisations to address. Samples cannot be representative as internet coverage is not universal.
  • Elements: business model; theory of change; drivers for change; service model; awareness raising and engagement approach: independence; traffic levels; and key success factors Customer discussion forums: Cisco systems -telecommunications supplier; Alzheimers Associaiton (USA) Market research communities: Institute of Directors Democracy and transparency websites: Mysociety.org; fixmystreet.com; writetothem.com; whatdotheyknow.com; theyworkforyou.com; planningalerts.com; Review and purchasing/booking websites: toptable.com Content submission and rating: threadless.com Multichannel contact centres: Rotherham Streetpride; irobot.com; Black & Decker Social tools and services: BBC; Topeka and Shawnee County Public Library Joining other forums: Topeka and Shawnee County Public Library
  • Example for today: The model of using an external and independent web based platform for patient feedback requires the organisation to have a dual facing function: Individual facing: access, having a voice and utilization System/ NHS organisation facing: impact on quality In fact 3 way facing: 3. Has own agenda- using web 2.0 technology to give voice to services users (primary?) leading to improvements in quality. Most important is citizens, giving voice I will display The 3 TOC models, which have levels of complementarity but also some tensions. TOC helps us to evaluate if the case study site (web feedback platform) is able to facilitate both sets of needs/goals or if they have a preference/ leaning towards the individual or organisations needs?
  • Overall Aims: wanted to make the wisdom and insights of patients, available to the NHS through “Web 2.0 technology. to enable citizens to engage with, and perhaps improve, their health services, by providing feedback Rationale: That the mechanism will improve quality of care because it is independent, public and transparent. referring to Reputation capital - which relies on critical mass of voice to . 1. improve services by organisations listening to what users want. 2. provided legitimacy to services through involvement of users. Local delivery: automated email alerts, tagging of relevant information i.e service and type of feedback-concern, suggestion, thank you, responding, reporting and analysis of the feedback information. Subscribers : their responsibility to promote the web based feedback platform. Outputs: after moderation of the post (based on editorial policy- remove names, offensive language). Outcomes: web based feedback platform provides training on how to respond (tone, no jargon, personal, informal) and to do so as soon as possible- Improved flow of communication/dialogue between NHS and its users. Impacts: ·     Feedback helps organisations: To plan how to develop and improve services; ·  To give doctors, nurses and managers a feel for what patients are saying about the service they manage.
  • From interviewing service users and from online survey: Rationale: around 50% of postings are positive, 25 % are critical and the rest mixed. Motivation varies- gratitude, anxiety, anger and altruism (PO report). Range of Impacts: weight off my mind, feel I may have helped others, to stop it happening again Main Barrier- FEAR of reprisal
  • Great variance between case study organisations
  • Local delivery- Director of Nursing. bespoke and friendly responses Champion- is on Board so has authority when replying. If they went then the role would be given to someone less senior. Does not want to devolve the responsibility of replying and managing posts- only him and Head of Midwifery. Web based feedback platform aimed for as many as possible front line staff to receive emails and to respond. Outputs: If a complaint it does not go into complaint report. “doesn’t want people to start soliciting responses”. Outcomes: when issues was resolved at meeting it was not put on website- journey was not in public anymore. Changes were not seen on website. End of public dialogue. Impacts: Quality improvement is embedded in the organisation- monthly meetings. But how embedded is the feedback mechanism in the organisations quality improvement process? Not a main tool: For improvements be sustained? Need staff engagement. Knowledge of mechanism (subscribers role) and links with other feedback tools. Service users generally trust the website but do staff trust it? Issue of trust of website- due to a negative post which had the staff name in it (issue of moderation)- felt it was untrue. But Head of Nursing (PPE?) said that “staff do trust the feedback mechanism”. Tensions between front line staff and champion of web based feedback mechanisms. Need for more posts to make real changes- need more evidence before making a change. (+ to be value for money)- also need critical mass for Reputation capital to work (feedback mechanism aim)
  • For external organisations, there is a dual focus – so the influencing of the healthcare organisation is difficult – key to effectiveness is proactive customer relations management
  • “ the real challenge appears not to be in identifying the feedback approaches to be used, but in equipping organisations to be able to put that feedback into practice to improve services. In other words – how are different sources of feedback brought together to give organisations an overall view of the patient experience, how is feedback reported upwards (to SMTs, boards etc) and downwards (to wards, teams etc), who is responsible for actioning feedback, how do organisations monitor what they do with feedback once its gathered etc.”

Assessing the potential and progress of web-based feedback for quality improvement: an evaluation using Patient Opinion as a case study Presentation Transcript

  • 1. Assessing the potential and progress of web-based feedback for quality improvement: an evaluation using Patient Opinion as a case study Deborah Davidson, University of Birmingham Olivia Joyner, Tavistock Institute Joe Cullen, Tavistock Institute
  • 2. What we will cover
    • Focus of the evaluation
    • Methodology
    • Analytical framework
    • Theory of change modelling
    • Findings
  • 3. Context
    • High Quality Care for All (DH 2008a) highlighted the importance of using service user experience for both monitoring and improving the quality of health care;
    • Number of reviews highlight the NHS complaints procedure as flawed, serving neither patients nor providers;
    • Compelling evidence to show that NHS complaints often become escalated, and positions entrenched as a result of poor initial handling of complaints by front-line staff or managers;
    • This leads to a protracted process and a dynamic of contestation and defensiveness, instead of learning and resolution;
    • The proactive use of feedback to inform and drive quality improvement is far from being achieved in the NHS.
  • 4. The assumption
    • Other sectors and industries have developed vastly more sophisticated approaches to capturing and actively using patient feedback. A far more diverse set of platforms are used, e.g. greater use of real-time and near-real-time methods and proactive seeking of views;
    • The emergence of web-based platforms for patient feedback, such as the independent Patient Opinion, I Want Great Care and Healthtalkonline sites and the “official” NHS Choices feedback facility, appears significant, since the evidence from other sectors would suggest that such platforms may have the potential to overcome some of the key limitations of traditional feedback and complaints mechanisms and to lead more directly to tangible improvements in the quality of care.
  • 5. The commission: 5 research questions
    • In what ways do current NHS complaints and feedback mechanisms fall short?
    • How might web-based approaches offer an opportunity to overcome these limitations?
    • In practice, how far are web-based systems actually overcoming existing limitations
    • What factors need to be present in a healthcare organisation for platforms to allow both user objectives and quality improvement objectives to be met?
    • Are there particular advantages or disadvantages to an independent as compared with an NHS-run feedback platform?
  • 6. Dual focus Individual facing: access and utilization (use by patients and public) System facing: leading to impact on quality (organisation subscribers)
  • 7. Research design
    • SCOPING:
    • Reference group
    • Data audit
    • Literature review
    • Stakeholder
    • interviews
    • Methodology and
    • evaluation toolkit
    • BENCHMARKING:
    • Comparisons
    • analysis
    • Analytical
    • framework
    • System audit
    • FIELD WORK
    • On-line survey and individual Interviews with service users/patients
    • Impact evaluation through 5 case study sites tracking feedback through to implementation
    SYNTHESES & REPORTING
    • DEVELOPMENT
    • Feedback
    • Developmental
    • workshop
    FINAL REPORT
  • 8. Socio-technical health systems
    • Technical coding (Feenberg, 1992)
    • ‘ those features of technologies that reflect the hegemonic values and beliefs that prevail in the design process’
    • Contest of meanings
    • ‘ Value embedded action systems’ (Cullen and Cohen, 2007)
    • ‘ Immanence’ of health technologies
    • Questions established notions of ‘ownership’ and ‘empowerment’
    • Need to capture ‘functional’ and ‘discursive’ attributes of the systems
  • 9. Functional Attributes
    • Based on ‘instrumental’ properties of systems
    • Example: FDA classification framework for medical devices
      • The service configuration provided (including tools used for collaboration)
      • The delivery platform (the infrastructure used to deliver health systems and services)
      • The key ‘knowledge domain’ covered
      • The revenue model adopted
      • The ‘knowledge production’ model adopted (as reflected in the nature of interactivity between stakeholders)
      • The ‘scenarios of use’ implemented (how the information/support is transmitted)
      • The ‘techniques’ used to promote interaction between actors (i.e. how feedback operates)
      • Holding techniques used to secure and retain audience interest
  • 10. Discursive Attributes
    • Decontextualisation – embededness of tools and functions in life world
    • Reductionism – user-centeredness and usability of functionalities
    • Autonomisation - availability for feedback and knowledge co-production functions
    • Positioning – degree to which users experiences are applied to technical functions
  • 11. Analytical Framework Research Questions Research Criteria Accessibility, Usability & choice Responsiveness Redress Independence Accountability Standards Change processes M&E Competence development Benchmarks Technical platform Service model Timeliness Governance Best practice Performance Functional attributes Response time User satisfaction SLA’s Training policy Discursive attributes Open-ness Knowledge co-production Participatory culture Values
  • 12. Implementing the Framework
    • Triangulation
      • Representing stakeholder voices
      • Multi-methodological (Surveys; Interviews; Focus Groups; Content/Discourse Analysis’ System Audit; Case Studies)
    • Theory of change modelling (Weiss, 1995)
      • Intervening variables
      • Evolving and immanent technologies
      • Discursive attributes
      • Involves embedding a theory of change and looking for causal pathways
  • 13. Using the theory of change
    • Theory of change maps:
      • help to lay out the issues or problems a project or programme is hoping to address, the actions being taken, and how these will lead to the final objectives that it is set up to achieve;
      • provides a road map as to how the aims and objectives will be achieved
      • Useful to start with the initial issues to be addressed, then look at the hoped for impacts and work backwards from there through the other stages.
      • One important thing to think about is that each stage is a ‘prerequisite’ for the next. ‘What are the requirements necessary to achieve that goal’.
  • 14.  
  • 15. In what ways do current NHS complaints and feedback mechanisms fall short?
    • System not centred on patients’ needs – lack of a customer focus, accessibility, flexibility and transparency
    • Most frequently raised issue was how the complaint was initially handled by the healthcare provider;
    • Poor communication and insufficient information;
    • Poor attitude of staff;
    • Perceived lack of fairness (independence)
    • Timescales and process very protracted
  • 16.
    • Patients don’t want to complain, because this takes them into a quasi-judicial process that brings with it, contention. Instead, people prefer to give feedback to the service about their experience, to ensure that their bad experience doesn’t happen to others in the future
    • (Bark, et al, 1994; Vincent et al, 1994; Friele and Sluijs, 2006).
  • 17. How might web-based approaches offer an opportunity to overcome these limitations?
    • Accessible, quick, easy to access and give feedback without having to go through a formal process
    • Experienced as responsive and perceived as independent
    • Potential for generating high volumes of traffic and data from people on a specific topic or theme;
    • Use can increase response rates;
    • Useful for gathering ongoing feedback over longer periods of time.
  • 18. Comparisons analysis
    • Customer discussion forums
    • Market research communities
    • Democracy and transparency websites
    • Review and purchasing/booking websites
    • Content submission and rating
    • Multichannel contact centres
    • Social tools and services
    • Joining other forums
    • Elements looked at: business model; theory of change; drivers
    • for change; service model; awareness raising and engagement
    • approach: independence; traffic levels; and key success factors
  • 19. In practice, how far are web-based systems actually overcoming existing limitations?
  • 20. Impacts? Different aims and agendas? Individual facing: Wanting a voice, help others, access and utilization (use by patients and public) System facing: leading to impact on quality (organisation subscribers) Feedback platform: wanting to give service users a voice through technology- lead to impact on quality
  • 21. Web-based platform for patient feedback Rationale If there is sufficient real time feedback traffic in the public domain it will lead to change. (reputational systems theory) Issue it seeks to address Lack of opportunities for service users to relate experience/to be heard. Need for public, independent, transparent feedback platform. Activities Feedback website Local Delivery Subscribers   Outputs Outcomes Impacts Website- provides opportunity to publicly tell a story. Power of web to share positive and negative experiences. Engage NHS organisations as subscribers. Provide training and guidance, access to posting feedback. Subscribers inform service users of the web based platform and encourage its use.   Subscribers (and non subscribers ) receive email alerts when feedback has been given. Organisation responds to feedback in a timely and considered manner. Organisation passes on feedback to services. Improvements in quality of care take place. Praise leads staff to be open to learning when later feedback is negative.
  • 22. Individual Service User Level : actual experience Rationale Service users with voice = enhanced experience and satisfaction. Help others. Issue it seeks to address Lack of opportunities for service users to relate experience/to be heard Activities Feedback website Local Delivery Subscribers   Outputs Outcomes Impacts Website- provides opportunity to write/tell story Local champion with commitment becomes a subscriber. Services inform service users of the web based feedback mechanism Or/and User finds mechanism on internet Individual stories/ feedback posted Feedback appears on public platform. User may receive a response from health organisation. Service user has satisfaction of voice being ‘heard’. May hear if changes have occurred.
  • 23. Subscriber level: Theory of Change map Rationale Services improved by having better quality assurance and feedback mechanisms More feedback mechanism choice for service users (greater accessibility) Need to fulfil patient experience and choice agendas. ‘Tick boxes’ Issue it seeks to address Inadequacy of complaints procedures re: encouraging learning rather than encouraging contestation, defensiveness and resistance. Little choice of independent feedback mechanisms. Greater patient choice.
  • 24. Activities Feedback Website Local Delivery Embedding quality Improvement   Outputs Outcomes Impacts Director of Nursing receives email alert of feedback from website provider. Director of Nursing emails the post to the relevant nurses. Director of Nursing and Head of Midwifery are the only people who can respond. If negative it is sent onto Clinical Lead. No holding responses. But a Reply to all posts within a week. Champion for website.
    • Records if feedback
    • was positive or
    • negative for each
    • service at
    • Management Group.
    • Not in Feedback
    • Handbook. Website
    • logo on their
    • website but no
    • description apart
    • from in giving
    • positive feedback.
    • lack of effective
    • and clear promotion.
    Does not feed into other feed back mechanisms or patient experience/ complaint reports. “ Doesn’t go any where”. Encourage Matrons to discuss feedback at Band 7 Meetings. If changes could be made the Director of Nursing will email staff asking for the change and checks on progress. Not recorded? If negative posts- patient will be asked to attend a meeting. Positive posts increase staff morale: “ professional pride and feeling of competence” Small changes will be made- Lack of recording of change and pathways to the Board.
  • 25. What factors need to be present in a healthcare organisation for platforms to allow both user objectives and quality improvement objectives to be met? Are there particular advantages or disadvantages to an independent as compared with an NHS-run feedback platform?
  • 26. Key findings
    • Patient Opinion web-based feedback mechanism clearly provides for patients’ needs that were lacking in complaints systems (access, responsiveness and independence):
      • Customer focussed
      • Accessible and flexible
      • Longevity to being public and transparent
      • Good initial handling by web based site
      • Good communication and information giving
      • Perceived to be fair and independent
      • Near real-time responses and straight forward process
  • 27. Meeting service user needs
    • Importantly, it enables service users to give feedback to the service about their experience (stories of experience)
    • Web based platform provides opportunity for others to learn about others’ experiences
    • Redress not within the boundary (or gift) of the individual-facing system to provide just remedies or ‘compensation’
  • 28. Embedding quality improvement
    • Accountability
    • Organisational learning
    • Standards
    • Change processes
    • Monitoring and evaluation
    • Training and development
  • 29. Key findings continued
    • Little evidence that this mechanism leads to an embedded and systematic approach to improvement in quality of service and care, though it has the potential to lead to improvement in the quality of service and care;
    • Progress is slow
    • Love affair with technology may be more of a driver than responding to individual and system needs;
    • Mechanisms of control – less enabling and empowering than in comparator services
  • 30.
    • Service model vulnerable as predicated on three ‘dependent variables’:
      • Traffic (PO)
      • Subscriptions (PO)
      • Champions (subscribers)
    • Vulnerable if even one of these isn’t present
  • 31. Embedding quality: processes
    • Management and governance: transparent and involved;
    • Established and systematised information and processes;
    • Customer focused behaviours: promoted and actively used;
    • Capture and route feedback: channelled to appropriate person for action;
    • Acting on feedback is seen as ‘business as usual’;
    • Manage and maintain good customer relations with clients.
  • 32. Quality improvement in practice
    • “ the real challenge is equipping organisations to be able to embed feedback into practice to improve services.”