IODA - The Promise & Perils of Narrative Research


Published on


Published in: Business, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

IODA - The Promise & Perils of Narrative Research

  1. 1. The Promise & Threats of Narrative Research<br />September 2011<br />
  2. 2. Measuring the Impact of OD Initiatives<br />This is an OD initiative because:<br /><ul><li> The organisation makes it’s money from providing consulting services
  3. 3. To ensure its staff have the skills necessary to do this, it trains them
  4. 4. The suite of training programs includes four core units
  5. 5. MBS was asked to validate that these units did indeed meet this goal and so, in theory, be able to help reshape the programs through feedback loops at each step from outcome, through design, implementation, workplace impacts, to the extent that they did not.</li></li></ul><li>Basic Discovery Process Map<br />Discovery /Dignosis<br />Design<br />Delivery<br />Evaluation<br />Performance<br />Behaviour in the workplace<br />Performance<br />Measuring behaviour in the workplace<br />Business Goals<br />Why we bothered in the first place<br />CompetenciesCapabilities<br />KnowledgeDo = SkillsBelieve these are things possessed by individuals that enableBehaviour i.e. the use of skills and knowledge based in values that individuals might get from / learn, in a program that acts as a catalyst so they can do certain things for the workplace<br />Performance ALL other<br />Things other than behaviour that contribute to performance in the workplace<br />
  6. 6. We undertook six activities to put us in a position to evaluate this Organisation Development program suite<br />We Did a document review of program materials<br />Met with & explored the content and intent of the 4 programs with client design & presenting staff <br />Did an international review of similar programs<br />Ran a workshop exploring the organisational objectives for each program and the full suite.<br />Set up Academic Directors and undertook initial observation of each program<br />Established a narrative capture data collection and analysis process based on 4 captures roughly associated with the work of Kirkpatrick and Brinkerhoff <br />We then began to collect data and run the Feedback Loops that would enable the Organisation to better achieve its goals<br />
  7. 7. Hybrid Kirkpatrick – Brickerhoff framework<br />
  8. 8. BRICKERHOFF’S Success Case Evaluation Method<br />Low end<br />High end.<br />This method examines participants who, after being surveyed turnout to be the least impacted by the training, these he interviews to understand why.<br />The other end of this method examines participants who, after being surveyed turnout to be the most successfully impacted by the training, these he interviews to understand why.<br />The MBS Mt Eliza Impact Evaluation System simultaneously does the Survey, manages the full statistical analysis of the results, plots these in multiple formats and has the data on which these are based immediately available for analysis without any interviewing.<br />
  9. 9. The 70-20-10 Development Philosophy<br />This philosophy defines development as occurring in 3 main ways: through on-the-job experiences; relationships, networking & feedback; and formal training opportunities. The “70-20-10” refers to how you should allocate professional development:- 70% of all development should occur on-the-job,- 20% from relationships, networking and feedback, and - 10% from formal training opportunities.We believe that most companies stress the 10%...this creates a problem related to achieving outcomes from training. So we wanted to try to get a better balance through our learning about how the training programs worked for the organisation.<br />
  10. 10. Question to discuss….What are your frustrations or challenges in evaluating complex initiatives?<br />
  11. 11. A Children’s Party ……<br />
  12. 12. Key Characteristics of complex systems..(sometimes called ‘Wicked Problems”)<br /><ul><li>context – do not happen exactly the same way twice
  13. 13. emergence – not predicatable
  14. 14. principles NOT prescriptive rules
  15. 15. many possible outcomes- nor ONE right answer
  16. 16. Too many factors to make predictable connections and predict futures
  17. 17. can only understand what happened and what connections were operating by looking back</li></li></ul><li>The Cynefin Framework<br />Disorder<br />Source: Cognitive Edge Pte Ltd<br />
  18. 18. Understanding and Intervening in Complex Issues - Methods<br /><ul><li>understanding or “ making sense of ‘ the current state of a defined system in order to take action and influence outcomes
  19. 19. narrative based – narratives hold and convey complex meaning
  20. 20. mass capture for patterns, insights and decision support
  21. 21. minimise role of ‘experts’ filtering data capture and analysis
  22. 22. evaluation focus on outcomes and impact</li></li></ul><li>Questions for Discussion:<br />How much of your practice is concerned with complex issues ?<br />What makes them complex?<br />
  23. 23. Elements of a SenseMakerTM Narrative Capture<br />Narrative Capture Entry Point<br />Prompting Questions<br />MCQ’s about experience<br /><ul><li> Specific to each instrument
  24. 24. Language tailored to target group
  25. 25. Designed to elicit range of experiences</li></ul>Experiences from participants<br />Filters about experience<br /><ul><li> Specific to each experience
  26. 26. Language tailored to target group
  27. 27. Provide basis for tracking relativenature of experiences over time
  28. 28. Established as double negative slider bars</li></ul>Demographics<br />General research questions<br />
  29. 29. Prompting Questions: to elicit a response<br />You have been asked to outline what has happened in applying the learning from your program in your workplace. What experience would you relate?<br />You have been asked by senior management to outline the benefits of the program to<br /><ul><li> Achieving business challenges
  30. 30. Improving workplace effectiveness
  31. 31. Improving the skills and capabilities of individuals</li></ul> <br />What experiences, yours or that of another participant would you tell in answer to this?<br />Key characteristics:<br /><ul><li> Place people in a context they can understand
  32. 32. Provoke people's memories
  33. 33. Allow the question to be answered in the third person
  34. 34. Do not privilege a certain type of experience over another
  35. 35. They relate to the issue at hand </li></li></ul><li>Narrative fragments: material captured in the field in its original form<br />Role play<br />Program X offered participants the opportunity to role play difficult client scenarios with Partners - which accelerated learning and has had the impact of readying me for client situations which have since transpired. I was pushed outside my comfort zone, but at all times felt supported.<br />Adapting to different working preferences<br />Program X reminded me that not everyone has the same working style and that I need to be able to adapt to their preferred working style. Using this model, I can determine the best way to communicate with colleagues and clients. The group activity during Program X provides a good opportunity to apply these skills.<br />Insightful<br />I have used a lot of the teachings from Program X. I have recently used performed a root cause analysis (i.e. fishbone analysis) to find the root cause of an issue a client was facing. The tool helped to to find solutions which effectively addressed the root cause of the problem.<br />Persistence<br />Immediately after leaving the training course I flew to a mine site and was tasked with developing a process manual for an end to end computer system used by all Queensland X mine sites. I flew there alone and was staying in the small country town near the mine (consisting of a pub and a petrol station) and met with considerable resistance from many employees at the mine who I was relying on for the information I would need to include. I sured many of the tools I learned from Program X during this project. I broke my tasks down into manageable chunks, I was very specific when asking for information and I used many tools of interpersonal understanding and negotiation<br /><ul><li> Narratives are the naturally occurring conversations that people have. They are not fully thought through crafted stories
  36. 36. Any narrative captured is displayed in it’s original form including any spelling or grammatical mistakes</li></li></ul><li>Signification - Triads:<br />Assessing the relationship between three competing Interests<br />Triads are one of the visual methods used to encourage participants to consider the dynamics that exist between three competing but linked aspects of an issue being explored.To respond, the participant needs to consider each aspect, moving the ball within the triad to the point they think best reflects the degree to which that aspect is dominant<br />
  37. 37. Signification - Polarities:<br />Opposing opposites to uncover layers of meaning<br />Ideal state<br />Absence of…<br />Excess of…<br />The questions are put in terms relative to two extremes of the area under consideration – an “absence of” on the left hand side and “excess of” on the right hand side. By sliding the ball to where they think their experience lies in relation to the two extremes, greater insight into the responses can be achieved when compared to a simple “good / bad” linear scale.<br />
  38. 38. Signification – Multiple Choice:<br />The person who provides the experience decides what it means<br />Multiple choice questions are also used to get the context of the experience from the participant. These questions can then be used as the lens through which the aggregate of all responses can be viewed. For example, multiple choice question could be used to view all responses from participants whose intent in telling their experience was to “inform” and about the personal outcomes associated with “flexibility and adaptability” <br />
  39. 39. SenseMakerTM<br />A look under the cover<br />
  40. 40. Hybrid Kirkpatrick – Brickerhoff framework<br />
  41. 41. What did we want to know more about?<br />
  42. 42. What insights did we find?<br />Collaborative learning<br />Overwhelmed withinsufficient time toshare experiences<br />Mostly useless & theyshould have shut up<br />Level of engagement (others)<br />Intense &obsessive<br />Detached & indifferent<br />Performance as an adult learner<br />Just along forthe ride<br />Accepting everythingwithout question<br />Facilitator / Presenter<br />Great fun & entertaining,but I did not learn<br />Too intimidating &dismissive for me to learn<br />Group functionality - collaboration<br />Having no concernfor anyone but themselves<br />So polite no onedared to challenge<br />Ideal State<br />Safe learning environment<br />Over protected<br />Too scared totry anything new<br />Quality of Facilitator / Presenter<br />Indifferent<br />Inhibiting<br />Level of engagement (self)<br />Avoiding beingnoticed<br />Centrestage<br />Boundary management – the social contract <br />A free for alldominated by a few<br />Limited & restricted by too many rules<br />Self responsibility<br />Someone else’s fault<br />Learnedhelplessness<br />
  43. 43. What insights did we find?<br />Cohort 1<br />Self responsibility<br />Cohort 2<br />Cohort3<br />Cohort 4<br />All cohorts<br />Learnedhelplessness<br />Someone else’sfault<br />Roleplaying<br />I think that the roleplays had the most impact - all well and good to theoretically teach new consulting skills but experience in putting them into practice I found far more effective in making the messages stick. While it seemed a little forced initially, once into full swing I found it valuable to get to deal with different possible behaviours of clients in interview and group meeting/presentation scenarios and then receive feedback on how it all went. It was also obviously easier to try things a little different and outside my confort zone in a controlled environment. <br />Role plays<br />The aspect that had the biggest impact on my learning was the mock interviews with partners. At first I felt nervous and a little intimidated but as we went through the 4 different interviews I realised that it was a safe environment in which to make mistakes and developed more confidence to give different interview techniques a go. I learnt a lot about how to manage difficult clients and will be less concerned about this if the issue arises in practice. By the end, I was very happy with my presentation and felt that I had overcome my fears to an extent.<br />
  44. 44. What insights did we find?3 months back in workplace: Behaviours<br />
  45. 45. What insights did we find?3 months back in workplace: Behaviours<br />
  46. 46. Question for Group Discussion:<br />What challenges in using this approach do you think we faced?<br />
  47. 47. Lessons learnt – about this style of Narrative Research<br /><ul><li>Under estimated how disruptive this would be to existing engagement processes and professional practice
  48. 48. Measuring program impact over time is not appropriate for all organisations
  49. 49. Upfront commitment to actively participate in collection, analysis and outcomes
  50. 50. A core team that changes needs new members being effectively inducted
  51. 51. Transparency and access to staff essential
  52. 52. Process is not a “one shot” deal
  53. 53. Need for assessment of program across all four stages to measure impact using feedback loops to improve programs
  54. 54. Being prepared for unintended consequences and access to the “right” decision makers</li></li></ul><li>In addition….. normal conditions required for OD initiatives apply<br /><ul><li>Commitment to the long term and measuring impact over time
  55. 55. Senior leadership sponsorship and support
  56. 56. A core team that will stay with the project
  57. 57. Transparency of programs – access to all staff
  58. 58. Commitment of time and resources for analysis and intervention
  59. 59. Willingness to embrace continuous improvement </li></li></ul><li>Contacts<br /><ul><li>Phill Boas
  60. 60. Viv Read
  61. 61. Chris Fletcher
  62. 62. Reference sites
  63. 63.
  64. 64.</li>