This document discusses multisource feedback (MSF) and its use in physician assessment and revalidation. It provides evidence from various studies that MSF can reliably and validly assess physician competencies. However, it also notes limitations in terms of its ability to consistently change physician behavior and the high costs associated with MSF programs. Overall, the document presents both sides of the debate around using MSF as an essential component of physician revalidation.

1. Multisource
feedback & its
utility
A N E P I C D E B A T E
D R . J O A N S A R G E A N T
D R . K A R E N M A Z U R E K

5. Resolution
"Be it resolved that based on the evidence that MSF can lead to
changes in physicians' behaviour, that MSF should be an essential
component of any system of revalidation"

6. MSF – The
Scientific
Evidence
J O A N SA RG EA N T P H D
P RO F ES S O R A N D H EA D, D I V I S I O N O F M E D I C A L E D U C AT I O N
DA L H O U S I E U N I V E RS I T Y, H A L I FA X , N S , C A N A DA
I A M R A M O N T R EA L O C TO B E R 2 0 1 5

7. In Canada, PAR - Continued
Development of Suites of
Instruments
Anesthesia
Surgery,
Medicine, Pediatrics, Psychiatry
Episodic Care
Laboratory Medicine
Radiology
Family Medicine (Revised)
Anesthesia (In process of being revised)
7

8. Evidence in support of PAR
Each suite was examined for psychometric
properties -
◦ Descriptive data on each item
◦ Range, means (s.d.), unable to assess
◦ Reliability
◦ G-study/ Cronbach’s alpha
◦ Factor analysis
◦ Association between scores and
◦ Sociodemographic variables
◦ Rater familiarity
About 20 scientific publications
◦ Violato/Lockyer/Fidler and others

9. Validity
There is a body of evidence that is coherent (hangs together)
and that supports the use of the results of an assessment for
a particular purpose.
Items and scales are developed through focus groups with
targeted stakeholders
Items correlate in intended ways producing fairly robust
factors (scales)
Scores increased on 2nd iteration for colleague and co-
worker

10. Reliability
The results of the assessment would be the same if repeated under
similar circumstances.
◦ Stability of the instrument
Factor analysis/ Cronbach’s alpha for instruments and scales >0.8
G-studies: Ep2 approximately 0.7 for most instruments with 8 medical
colleagues & co-workers (15-40 items) and 25 patients with 15-40 items
Note - High stakes Ep2 > 0..8
10

11. Utility: Changes to practice
Changes contemplated or initiated based on feedback
(Violato et al, Acad Med, 1997; Hall et al, CMAJ 1999; Fidler
et al, Acad Med 1999; Sargeant 2003; Overeem 2007)
◦ About 50% of MDs report making changes in practice
◦ Changes: specific aspects of care; communication with
patients, colleagues and co-workers; stress management
◦ Changes requiring support or expenditures less likely
◦ Change is influenced by multiple factors; e.g, perceptions
of data credibility; fairness of process; specificity of data;
beliefs about change or ability to effect change
◦ Limitations: self-report
1
1

12. So wouldn’t you agree…..
Lots of evidence from PAR?

13. More evidence - Systematic review of
MSF studies (Violato et al 2014)
A systematic literature review for English-language
studies (1975 to 2012) was conducted.
Search parameters: multisource-feedback, 360-degree
evaluation, assessment of medical professionalism
48 studies (Canada, US, UK, NL, China and elsewhere
met the inclusion criteria)

14. Domains Assessed by MSF
•Professionalism
•Clinical competence
•Communication with patients and families
•Case and office management
•Collegial relationships
•Best for humanistic domains that can’t be assessed
in other ways (Archer et al 2005, 2006; Overeem
2010)

15. Conclusions
MSF has evidence of reliability, validity and feasibility and can
◦ assess core competencies of physicians
◦ identify strengths and weaknesses within competencies
◦ provide feedback for professional development

16. Validity-Recruiting reviewers
1. Ramsey e tal, JAMA 1993 – ratings similar for physician and
supervisor selected reviewers
2. Archer and McAvoy, Medical Education 2011 – ratings were higher
for physician-selected reviewers than referring-body selected
reviewers (high-stakes assessment)
1
6

17. Validity- factors influencing reviewers’
MSF assessments
Campbell et all. BMJ, 2011 -
◦ Country primary medical qualification
◦ Locum status
◦ Clinical specialty
◦ Contractual role
◦ Familiarity (% reporting ‘daily’ or ‘weekly’ contact ↑)
Sargeant et al, Acad Med 2003, Med Ed 2007 –
◦ Familiarity ↑
◦ Those less familiar selected “unable to assess” more frequently
◦ “ Raters default to ‘3’ (average) if they don’t know you”
◦ “need to be able to observe someone to assess them”
1
7

18. Factors influencing feedback
acceptance and use
Physicians are more satisfied with feedback that includes
narrative comments (Overeem et al., 2010)
Physicians need specific feedback to guide them in
improvement (Sargeant et al 2007, 2008)
◦ “being a ‘4’ doesn’t tell you what to do to improve”
◦ need narrative for specific guidance on “how” to improve
1
8

19. Factors influencing feedback
acceptance and use
Agreement with MSF report is positively correlated with
scores (Sargeant 2003, 2008)
Impacts of this -
◦ Low scores can cause distress and result in demotivation
(Sargeant 2008, Overeem 2010)
◦ Facilitated reflective feedback can enhance feedback
acceptance (Denisi, Kluger 2000; Goodstone, Diamante 1998;
Sargeant et al 2015)
1
9

20. What are you doing with THIS evidence??

21. MSF- The
Practical Issues
KAREN MAZUREK
DEPUTY REGISTRAR
COLLEGE OF PHYSICIANS AND SURGEONS OF ALBERTA

22. Why PAR?
believe that all physicians can benefit from continuous
quality improvement
Wanted to build a culture of CQI
Did not want a hunt for “bad apples”
Public believed it was already being done
22

23. How does PAR work?
Every physician, once every 5 years
25 patients, 8 colleagues, 8 co-workers, self-assessment
Report gives results and comparisons; how to use results; how
to claim CPD credits
3-month follow-up feedback
23

24. PAR Survey
Self-Analysis
Response
PAR
Committee
Review
Individualized
Assessment
Advice to
Physician
Feedback
Office Peer
Review
Competence
Assessment
Diagram 1
The College’s PAR Program

25. Participation in PAR
1999-2014 totals
GP/family physicians = 5,546
Surgeons = 1,600
Anesthesiologists = 571
Medical specialists = 1,940
Psychiatrists = 754
Pediatricians = 563
Episodic care = 1,122
Diagnostic Imaging = 494
Lab Medicine = 256
Total to date = 12,846
25

26. Raw scores
Communication
Humanistic
Office mgt.
Telephone
Staff
4.2 4.4 4.6 4.8 5.0
PAR Results
communication
humanistic
office mgt.
telephone
staff

28. T-Score Format

30. Presentation of feedback
Sargeant et al, Academic Medicine 2003
◦ Physicians who receive high scores agree with colleague feedback; those who receive low
scores are neutral or disagree with results.
◦ Facilitation of feedback should be considered
Eva KW, Regehr G. Commentary. CMAJ, 2013
◦ “Cognitive dissonance is the discomfort created by trying to maintain 2 conflicting beliefs
at the same time”
◦ “Easier to question the data than to question oneself, this pair of conflicting beliefs will
often be resolved by discounting the feedback rather than altering one’s sense of self*”
*Eva KW, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: on the
interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ Theory
Pract 2012;17:15–26.)

31. Self selection of raters
Ramsey et al, JAMA 1993
– ratings similar for physician and supervisor selected reviewers
Archer and McAvoy, Medical Education 2011
– ratings were higher for physician selected reviewers than referring-
body selected reviewers (high-stakes assessment); self selection of
raters should end

32. Outcome
Violato, C, Lockyer, JM, Fidler H, British Medical Journal 2003
◦ “Many surgeons in this study used the feedback to
contemplate or initiate changes to their practice”.
Lockyer, J, Violato C, Fidler H, Teaching and Learning in Medicine
2003
◦ “Surgeons made few changes in practice in response to feedback data”
Ferguson et al, BMC Medical Education 2014
◦ systematic review found 16 studies in the World literature looking at MSF
and change in physician behavior - only one found a measurable
improvement.

33. PAR Feedback
% of respondents who report making at least one practice change
51% Jan to June 2013
48% July to Dec 2013
54% Jan to June 2014
62% July to Dec 2014
56% Jan to June 2015

34. Cost of PAR
$200 per physician; $40 per year
2013/2014 financial data - $600 to 700 per physician; $120 to $140 per
year
3x the cost!!!

35. Alberta survey - 2015
Sent surveys to all physicians/surgeons in Alberta (not provisional
licenses)
How good is existing program (PAR)?
2215 responses out of 9021 = 25% response rate
Previous CPSA record was Physician survey 2014 with RR of 14%.
Typically <10%
99% probability that these results are within 2.4% of actual true value

36. Please rate how successful the existing Physician Achievement Review (PAR)
Program is in assessing the following dimensions. (1-10 where 1=not at all,
5=fairly, and 10= extremely)

37. 303 Text responses: Substantially negative – we grouped into 8
key themes
“Too easy to cheat. Probably little use to 90% of profession”
“To me the PAR is an incredible waste of time and money. I do not know one MD
who actually pays it much attention, either filling it out or implementing changes”
“…Patients uniformly complain only about waiting times in parking and really have
little useful to say about the quality of care that is delivered. Comments from peers
as well as other physicians in the community are uniformly positive and there is
never anything in the way of constructive criticism or useful feedback…”
Irrelevant
Subjective, biased
Easy to cheat
Discriminatory
Confusing
Time-consuming
Repetitive
Waste of money

38. Why Canadian regulators would
consider abandoning Par
The impact on the physician is not what we hoped
After 15 years we don’t know that it improves care for patients
It costs too much money
Physicians do not like it

39. Questions
whose side are you on?
are you using MSF now?
what do you see as its strengths and weaknesses?
how do you see its role in assessment and revalidation?