Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Nice Model, But What Is The Evidence?

1,157 views

Published on

Key Note of the EHMA 2016 Annual Conference in Porto

In this key note, Rob Briner and Eric Barends from the Center for Evidence Based Management will discuss the basic principles of EBMgt and consider why while most people agree with the principles of EBMgt, few organisations are able to take advantage of its potential benefits. Utilising interactive social media tools
Rob and Eric will demonstrate how EBMgt can be used to separate the wheat from the chaff.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Nice Model, But What Is The Evidence?

  1. 1. Nice model, but what is the evidence? EHMA Annual Meeting, Porto, June 15th, 2016 Evidence Based Practice, basic principles and practical examples
  2. 2. Postgraduate Course
  3. 3. Our Fellows
  4. 4. CEBMa: what we do  Promote (seminars, papers, blogs, tweets)  Educate (universities & business schools)  Train & coach (companies > projects)  Support / REAs (companies)  Support / 2nd opinion (BS detector)
  5. 5. 15 min Management models 25 min EBP: What is it and why do you need it? 20 min A practical example … min Discussion Our mission for today To disturb To reassure
  6. 6. Dangers of believing things that aren’t true  It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so. (Mark Twain)  The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge. (Stephen Hawking)
  7. 7. Some examples What’s happened to average job tenure in the UK and US over the past 10 years?  Up?  Down?  Stayed the same?
  8. 8. Some examples: Job satisfaction What’s happened to job satisfaction in the US over the past 10 years?  Up?  Down?  Stayed same?
  9. 9. Some examples: VUCA Is the world moving ever faster – more VUCA (volatility, uncertainty, complexity and ambiguity)?  Faster?  Slower?  Same?
  10. 10.  The idea that time is speeding up is clearly popular. It is also plausible. There is just one problem. It is very hard to prove that it is actually happening.  Hard evidence of a great acceleration is hard to come by. The Economist has considered a variety of measures by which the speed of business in America can be quantified. A few do show some acceleration. But a lot do not.
  11. 11. UK Patents
  12. 12. Some examples: Generational differences Are there generational differences in work attitudes?  Big differences?  Small differences?  No differences?
  13. 13. So what?  There are many taken-for-granted assumptions and ‘truths’ in management that turn out to be wrong or at least overstated  It’s important to examine these assumptions because if they are wrong will lead to poor decisions  Possibly also in the case of Management Models
  14. 14. Nice pictures
  15. 15. Model Taken-for-granted assumptions? Fad? Evidence? The question is …
  16. 16. Model Most of these models are developed within the domain of management
  17. 17. The problem with managers They love fads!
  18. 18. 38  Scientific Management/Taylorism  Business Process Reengineering  Total Quality Management  Learning Organizations  Knowledge Management The nearly forgotten fads
  19. 19. 39
  20. 20. 40  Talent management  Emotional intelligence  Employee engagement  Lean / six sigma / theory of constraints  Autonomous teams (it’s back!)  Agile The fads that haven’t been forgotten (yet)
  21. 21. 41 Fads seem to be attractive, compelling and irresistible  Promise to deliver a lot and fast  Appear simple  Will make everything alright and help contain anxieties around intractable problems  Help user feel effective and cutting edge  Bits of some fads may work in some context
  22. 22. 43
  23. 23. 44
  24. 24. 1. What tells you a model is a fad? 2. Can you give an example in healthcare management? Discuss with your neighbour (2 min) Discussion 60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210
  25. 25. 1. Simple, straightforward: Easy to communicate, comprehend, and reduce to a small number of factors or dimensions. 2. Promising results: Fad auteurs are confidently didactic. There is no false humility or hedging. 3. Universal: Solutions for everyone everywhere. 4. Step-down capability: Can be implemented in ritualistic and superficial ways. How to spot a management fad
  26. 26. 5. In tune with zeitgeist: Resonate with trends or perceived business problems of the day. 6. Novel, not radical: Novelty is not so much new discovery as a rediscovery and repackaging of older ideas. 7. Lively, entertaining: Articulate, bold, memorable and upbeat. Filled with labels, buzzwords, lists, acronyms and fun anecdotes > your system 1 loves it! 8. Legitimacy via gurus and star examples: Supported by stories of excellent companies and the status of gurus, not by solid evidence. How to spot a management fad
  27. 27. Model So, we need a meta-model to check whether the model is evidence based
  28. 28. Evidence based practice: What is it?
  29. 29. Evidence based medicine?  Never heard off  Heard off  Heard off and can explain
  30. 30. Evidence based management?  Never heard off  Heard off  Heard off and can explain
  31. 31. Evidence-based practice Central Premise: Decisions should be based on the ‘best available evidence‘.
  32. 32. Evidence? information, facts or data supporting (or contradicting) a claim, assumption or hypothesis
  33. 33. Evidence? outcome of scientific research, organizational facts & figures, benchmarking, best practices, personal experience
  34. 34. All managers and leaders base their decisions on ‘evidence’
  35. 35. But…many managers pay little or no attention to the quality of the evidence they base their decisions on and use too few sources of evidence
  36. 36. Trust me, 20 years of management experience
  37. 37. Sources of evidence problem solution Practitioners professional expertise Organization internal data Stakeholders values and concerns Scientific literature empirical studies Ask Acquire Appraise Aggregate Apply Assess
  38. 38. Professional experience and judgment Multiple sources?
  39. 39. Multiple sources? Organizational data, facts and figures
  40. 40. Multiple sources? Organizational data, facts and figures
  41. 41. Multiple sources? Scientific research outcomes
  42. 42. Evidence based practice: Where does it come from?
  43. 43. McMaster University Medical School, Canada Medicine: Founding fathers David Sackett Gordon Guyatt
  44. 44. How it all started
  45. 45. 1. Ask: translate a practical issue into an answerable question 2. Acquire: systematically search for and retrieve the evidence 3. Appraise: critically judge the trustworthiness of the evidence 4. Apply: incorporate the evidence into the decision-making process 5. Assess: evaluate the outcome of the decision taken 5 steps of EBmed
  46. 46. Evidence-Based Practice 1991 Medicine 1998 Education 2000 Social care, public policy Nursing, Criminal justice, Policing, Architecture, Conservation 2010 Management
  47. 47. Evidence-Based Practice
  48. 48. Evidence-Based Practice
  49. 49. Evidence-based … whatever = the use of evidence from multiple sources to increase the likelihood of a favourable outcome Focus on the decision making process Think in terms of probability
  50. 50. Evidence-Based Decision-Making Why do we need it?
  51. 51.  It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so. (Mark Twain)  The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge. (Stephen Hawking)
  52. 52. Advice: lie babies down to sleep on their belly (unanimous support through to the 1990s) Example: medicine
  53. 53. Nr of cot deaths (Holland)
  54. 54. Example: economy / finance Nov 15, 2005 "With respect to their safety, derivatives, for the most part, are traded among very sophisticated financial institutions and individuals who have considerable incentive to understand them and to use them properly.” Jan 10, 2008 ”The Federal Reserve is not currently forecasting a recession”
  55. 55. Collateralized Debt Obligations > AAA 0.12% (about 1 chance in 850) default in 5 years Example: economy / finance
  56. 56. Forecasted Actual Forecasted and actual 5-year default rates for AAA-rated CDO tranches 0.12% 28%
  57. 57.  5 Trillion dollars in pension money, real estate value, savings and bonds disappeared  8 million people lost their job  6 million people lost their houses Example: economy / finance (and that was only in the USA)
  58. 58. Scared straight Example: policy / prevention
  59. 59. Example: HR management
  60. 60. 1. Incompetent people benefit more from feedback than highly competent people. 2. Task conflict improves work group performance while relational conflict harms it. 3. Encouraging employees to participate in decision making is more effective for improving organizational performance than setting performance goals. Likely or unlikely?
  61. 61. How evidence-based are HR managers?  959 (US) + 626 (Dutch) HR professionals  35 statements, based on an extensive body of evidence  true / false / uncertain HR Professionals' beliefs about effective human resource practices: correspondence between research and practice, (Rynes et al, 2002, Sanders et al 2008)
  62. 62. Outcome: not better than random chance
  63. 63. Example: management models
  64. 64. George BuckleyJames McNerney
  65. 65. So where does the evidence-based stuff come in?
  66. 66. Relying on only 1 source: bad idea! problem solution Practitioners professional expertise Organization internal data Stakeholders values and concerns Scientific literature empirical studies
  67. 67. Discuss with your neighbor (1 min) Over a 5 year period, why is an orthopedic surgeon's experience, as a rule, more trustworthy than a change manager’s experience? 60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210
  68. 68. Developing expertise 1. A sufficiently regular, predictable environment 2. Numerous opportunities to practice 3. Receive accurate (objective) feedback The management domain is not highly favorable to expertise!
  69. 69. Bounded rationality
  70. 70. How your brain works System 1  Fast  Intuitive, associative  heuristics & biases  emotional System 2  Lazy  Slow  Deliberate  Rational
  71. 71. System 1: short cuts
  72. 72. System 1 or system 2? 10 seconds
  73. 73. System 1 or system 2?  A bat and a ball cost $1.10 in total.  The bat costs $1 more than the ball  How much does the ball cost? 109876543210
  74. 74. System 1: necessary to survive 95%
  75. 75.  Pattern recognition  Overconfidence bias  Halo effect  False consensus effect  Group think  Self serving attribution bias  Sunk cost fallacy  Cognitive dissonance reduction System 1: prone to cognitive errors  Confirmation bias  Authority bias  Small numbers fallacy  In-group bias  Recall bias  Anchoring bias  Availability bias
  76. 76. We are predisposed to see order, pattern and causal relations in the world. Patternicity: The tendency to find meaningful patterns in both meaningful and meaningless noise. pattern recognition
  77. 77. We are pattern seeking primates: association learning pattern recognition
  78. 78. Points of impact of V-1 bombs in London
  79. 79. Points of impact of V-1 bombs in London
  80. 80. Pattern recognition
  81. 81. Pattern recognition
  82. 82.  A Type I error or a false positive, is believing a pattern is real when it is not (finding a non existent pattern)  A Type II error or a false negative, is not believing a pattern is real when it is (not recognizing a real pattern) Dr. Michael Shermer (Director of the Skeptics Society) Pattern recognition
  83. 83.  A Type I error or a false positive: believe that the rustle in the grass is a dangerous predator when it is just the wind (low cost) Pattern recognition
  84. 84.  A Type II error or a false negative: believe that the rustle in the grass is just the wind when it is a dangerous predator (high cost) Pattern recognition
  85. 85.  A Type I error or a false positive: believe that the rustle in the grass is a dangerous predator when it is just the wind (low cost)  A Type II error or a false negative: believe that the rustle in the grass is just the wind when it is a dangerous predator (high cost) Pattern recognition
  86. 86. Doctors and managers hold many erroneous beliefs, not because they are ignorant or stupid, but because they seem to be the most sensible conclusion consistent with their own professional experience!
  87. 87. “I’ve been studying judgment for 45 years, and I’m no better than when I started. I make extreme predictions. I’m over- confident. I fall for every one of the biases.”
  88. 88. Practitioners professional expertise Organization internal data Stakeholders values and concerns Scientific literature empirical studies Four sources of evidence (not only 1)
  89. 89. The performance of knowledge workers A Practical Example
  90. 90.  550 beds  3300 employees  210 medical specialists  225,000 admissions  Top Clinical & Teaching hospital Organization
  91. 91. The model
  92. 92. 2015: 7.2 2016: 6.3
  93. 93. A happy & engaged employee is a productive employee Fundamental assumption
  94. 94. Let’s have a look at the evidence Professional experience and judgment Organizational data, facts and figures Stakeholders’ values and concerns Scientific research outcomes Ask Acquire Appraise Apply Assess problem solution
  95. 95. GREAT! NOW WHAT?
  96. 96. Evidence-based approach, step 1: ASK problem solution Practitioners professional expertise Organization internal data Stakeholders values and concerns Scientific literature empirical studies Ask Acquire Appraise Aggregate Apply Assess
  97. 97. Population? Knowledge workers Whether nurses, lawyers, engineers, managers, or staff members, nowadays most workers in organizations are highly dependent on information and communication technology and are involved in work that involves a high level of cognitive activity.
  98. 98. Question “Which of the factors that are related to the performance of knowledge workers are most widely studied and what is known of their effect?”
  99. 99. I Don’t Know (but I know how to find out) The 3 hardest words in management
  100. 100. Step 2: ACQUIRE Search for the best available scientific evidence
  101. 101.  ABI, BSP, PsycINFO  Scholarly journals, peer reviewed  1980 – 2013  English  performance, productivity, knowledge work* ACQUIRE
  102. 102. step 3: APPRAISE & AGGREGATE
  103. 103. Effect size?
  104. 104. Most studied & largest effect size 1. Social cohesion .5 / .7 2. Perceived supervisory support .5 3. Information sharing / TM .5 4. Vision / goal clarity .5 5. Trust .3 / .6
  105. 105. step 3b: CROSS VALIDATE – multiple sources
  106. 106. Step 4: APPLY
  107. 107. Three examples social cohesion supervisory support information sharing
  108. 108. Social cohesion
  109. 109. Social cohesion … a shared liking or team attraction that includes bonds of friendship, caring, closeness, and enjoyment of each other’s company.
  110. 110. Social cohesion
  111. 111. Measuring social cohesion
  112. 112. Perceived supervisory support
  113. 113. …how employees feel the supervisor helps them in times of need, praises them for a job well done or recognizes them for extra effort. Perceived supervisory support
  114. 114. Perceived supervisory support
  115. 115. Measuring perc. sup. support
  116. 116. Information sharing
  117. 117. Information sharing? …refers to how teams pool and access their knowledge and expertise – which positively affects decision making and team processes. This has led to the idea of a team ‘Transactive Memory System’ (TMS), which can be thought of as a collective memory in a collective mind - enabling a team to think and act together
  118. 118. Information sharing
  119. 119. Measuring information sharing
  120. 120. Step 5: ASSESS the outcome The departments with the lowest performance scored under average on most factors
  121. 121. Reactions Who knew?
  122. 122. So, when a new model crops up

×