When performance is fuzzy (ispi 2013) v2


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

When performance is fuzzy (ispi 2013) v2

  1. 1. WHEN PERFORMANCE IS FUZZY:THE CRITICAL INCIDENT TECHNIQUE (CIT)Steven W. Villachica, PhDSteveVillachica@boisestate.eduOrganizational Performanceand Workplace Learning Download slides and handout at https://sites.google.com/a/boisestate. edu/ieeci/e2r2p/project-deliverables
  2. 2. Agenda 2 Introduction CIT Examples Your Turn Wrap Up
  3. 3. Where CPT Fits in Performance Improvement 3
  4. 4. What do you do when… 4 Exemplary performance is fuzzy? No one knows what a “good one” looks like? Managers and clients don‟t know how work gets done? There are no functional descriptions of workplace activities? Job descriptions  workplace tasks? What the organization says it values  what the organization really values?
  5. 5. 5Jonassen, Tessmer, Hoffman, Coffey, Ford, & Carnot (2001)& Hannum (1999a, 1999b) Harless (1986) Flanagan (1954, 1962)
  6. 6. CIT as Evidence-Based Practice 6 Data-Driven Decision Making Almost 60 Years of Use1. Determine questions you  In peer-reviewed want to answer journals2. Collect specific, relevant data  In a variety of disciplines from different, triangulated and workplace settings sources in the field  To create all sorts of3. Analyze the data performance4. Draw conclusions to answer improvement solutions the questions --in ways that improve valued performance More stuff that works!
  7. 7. CIT in Many Fields 7Researcher Used CIT toFlanagan (1954) Create procedures to select and train WWII aircrews.Thomas & Identify triggers that virtual teamsBostrom (2010) use to adapt their uses of technology during a projectKorte (2010) Investigate how newly hired engineers socialize themselves within a firm
  8. 8. CIT in Many Fields About 20 CIT appearances in PI and PIQ 8Researcher Used CIT toCraytor (1968) Create programmed instruction in therapeutic radiology for nursing studentsSmith (2009) Identify areas of expertise associated with ASTD‟s competency modelHale (2011) Create ISPI‟s proficiency-based certification for school improvement specialists
  9. 9. CIT in Many Fields About 20 CIT appearances in PI and PIQ 9Researcher Used CIT toLundberg, Conduct a needs assessmentElderman, investigating a problem with billableFerrell, & hours in a national retailer‟s parts andHarper (2010) service departmentBacdayan Create a test that quality improve-(2002) ment teams can use to determine the suitability of a given projectDean (1998) How to conduct CITMarrelli (2005) How to conduct CIT
  10. 10. Handout A Basic CIT Process pp. 1-2 10 4. Analyze data1. Determine the aim • Frame of reference of the CIT • Categories • Priorities 2. Plan the CIT • Verification• Observers• Observations 5. Report findings• Specific behaviors • Categories • Prototypical incidents 3. Collect incidents • Limitations• Observations • PI conclusions• Interviews CIT isn’t a rigid recipe. It’s a set• Focus groups of flexible set of principles.• Surveys (Flanagan, 1954; Woolsey, 1986)
  11. 11. Example 1: Making Service Standards Real 11 The opportunity The incidents • Client (anonymous) • Service Standards • Stephanie Clark  Professional • Amanda Collins  Respectful • Julie Kwan  Compassionate • Allison Sesnon  Helpful • What do the standards REALLY mean? • How do we operationalize the standards? • How do we close gaps in service performance? The results
  12. 12. What the ID Team Did 12 Use CIT to collect stories about exemplary and non-exemplary performance Generated competencies Ranked the criticality of the competencies Focused on two key competencies • Responding to clientele needs • Communicating with clientele and team Fixed the environment and provided training • Standards, feedback, process
  13. 13. A Service Standard Example: Helpful 13Exemplary Performance Non-Exemplary BehaviorAn elderly guest, using a cane, came A family is in a hurry to get on theinto the kitchen for a yogurt. A road. They were just informedvolunteer working in the kitchen unexpectedly that they need togreeted her and engaged in friendly check their child out of theconversation. The volunteer hospital this afternoon. They arerecognized that the guest was frantically trying to get everythinghaving difficulty going out the door, done to leave. A volunteer noticesso the volunteer offered to hold the the family is leaving and remindsdoor. The guest remembered she them to be sure to wash the roomneeded a spoon for her yogurt, and laundry before they leave.the volunteer fetched a plastic spoonfor the guest and assisted the guestout the door.
  14. 14. How CIT Helped 14 Made otherwise abstract standards visible by associating compelling stories with each Targeted service competencies needing improvement Provided a mechanism to fix environmental causes of the performance gap Provided sources of demos and http://www.perform ancexpress.org/2012 role-play activities for the training /08/tales-from-the- Made the effort “by and for the field-making-service- standards-real-for- volunteers and staff” families-in-need/
  15. 15. Example 2: Decreasing Time to Competent Engineering Performance 15 This material is based upon work supported by the National Science Foundation under Grant No. 1037808. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
  16. 16. Engineering Education Research to Practice E2R2P 16 Improve engineering education in ways that improve workplace performance. Education engineering for engineering education. Engineering Workplace Newly Graduated and Hired Students Skills “Freshout” EngineersEngineering Education Engineering Workplace Research-to-Practice Valley of Death
  17. 17. Our Shared Opportunity Decrease Ramp Up Time to Competent Performance 17 Company Costs Promotion! Desired $ Training Competency $ Errors Actual Competency $ MentoringPerformance $ Salary New Task/Project $ Opportunity $ Other projects Leave University/Enter Workforce $ Others? Time { } Increase Starting Skills - OR - REDUCE Change Learning Curve - OR - CO$T Make Boundaries Porous
  18. 18. Spanning Gaps between Actual and Desired Engineering Performance 18 Shared Decrease Ramp-up Time to Competent JobEducation Engineering Opportunity Performance in the Engineering Workplace ProblemIdentification Research Questions • What are newly graduated and hired “fresh out” engineers Root Escape doing/not doing in the workplace that they should? Cause Cause • What are the consequences of performance/non-performance Analysis Analysis in the workplace? • What workplace competencies should fresh outs possess? • In what workplace contexts do fresh outs apply the competencies? Escape • What are the root causes of workplace nonperformance? Corrective Corrective Action Action Focus Groups & Surveys Problem Inspection • Engineering managers, engineering leads, HR personnel, and Failures technical scientists who work with fresh out engineers • Fresh out engineers  Engineering Practice Survey
  19. 19. Method 19 Participants7 Focus Groups Company Manager Fresh-Out Qualitative design using critical Parametrix 5 0 Micron 4 3 incident technique (Flanagan, Motive Power 3 4 1954) CH2MHill 4 3 16 engineering managers, lead Total 16 10 engineers, supervising engineers, technical scientists, and HR personnel that work with freshouts to bring them up to speed in the workplace 10 freshouts
  20. 20. Method Procedure 20 Identify Company Arrange FocusRecruitment Sponsors Groups Facilitate Collect Data about Workplace Collect Data about Causes ofFocus Groups Performance Nonperformance Grow Share Results Work towards and CollaborativeCollaboration Sensemaking Corrective Action
  21. 21. Method Handout Instrumentation p. 4 21Critical Incident Card
  22. 22. Method Instrumentation 13Root Cause Analysis INFORMATION TOOLS MOTIVATION ENVIRONMENT • Data • Resources • Incentives • Expectations • Software • Rewards • Feedback • Tools • Consequences • Standard Operating • Support Procedures • Knowledge • Physical Capacity • Motives PERSON • Skills • Mental Capacity • Affect • Flexibility • Work Habits • Resilience • Drive
  23. 23. Comparing the Examples 23 ID team supporting  Collect instances nonprofit service  Group instances into standards behaviors or competencies Research team  Prioritize the groups conducting a  Make sense of the data performance analysis for “It’s the small stories freshout engineers gathered together that made that big ah-ah.”  --Allison Sesnon
  24. 24. Your Turn! Create and Classify Incidents for ISPI 24 ISPI wants to collect real-world stories about performance improvement1. Instances of successful HPT performance2. Instances of unsuccessful HPT performance3. Assign categories4. Leave completed incidents on the table You will need a pen or pencil!
  25. 25. Ground Rules 25 Doing this voluntarily. Can stop any time. Feel free to omit your name and email if you want. ISPI may contact you to learn more about an incident you provide. ISPI considers your data confidential. ISPI will report data in aggregate form, without mentioning individual contributors.
  26. 26. Name (optional) Email address (optional)Describe an incident that occurred to a practitioner(you or someone else) trying to improveperformance. Discrete Tasks 26 • Deliver a clientWhat level of performance was the practitionertrying to improve? presentation(Check all that apply) • Identify the root Worker  Organization  Enterprise  Society cause of a problemDoes this incident reflect (check one): • Analyze a data set Where the practitioner successfully performed a task related to improving performance? Where the practitioner was unsuccessful in • Keep stories short performing a task related to improving performance? • Focus on quick generationWhat were the general circumstances leading up tothis incident? Use Action VerbsWhat the practitioner was trying to accomplish? • Delayed productionHow did this incident affect the goals of thepractitioner’s project? • Increased costs • Satisfied customersHow often do incidents like this occur?  Only once  Sometimes • Met standards  Frequently
  27. 27. Successful Incident (With apologies to Harold Stolovitch)Describe an incident that occurred to a practitioner Describing a training request,(you or someone else) trying to improve performance. the client seemed focused on 27 means (schedules, compliance, length of training). Client didn’tWhat level of performance was the practitioner trying mention anything about theto improve? ends –the valued business(Check all that apply) goals that the training should Worker  Organization  Enterprise  Society produce.Does this incident reflect (check one): Where the practitioner successfully performed a Used probing questions to: task related to improving performance? • Frame statements of actual Where the practitioner was unsuccessful in and desired performance. performing a task related to improving • Align the gap with business performance? goals.What were the general circumstances leading up to Training request from humanthis incident? resources department.What the practitioner was trying to accomplish? Focus on valued performanceHow did this incident affect the goals of the Refocused client on delivering apractitioner’s project? valued success storyHow often do incidents like this occur?  Only once  Sometimes  Frequently
  28. 28. Unsuccessful Incidents 28 Generate incidents where a practitioner was unsuccessful in performing a task related to improving performance.
  29. 29. Code the Incidents You’ve Created Part 1 29CPT Standard Code of Ethics Cause Analysis(1-10) (A-F) (a-l)For every instance:1. Specify at least one relevant CPT standard (1-10)2. Specify at least one ethical code (A-F)
  30. 30. Code the Incidents You’ve Created Part 2 30CPT Standard Code of Ethics Cause Analysis(1-10) (A-F) (a-l)For unsuccessfulperformances,1. Indicate ONE potential root cause (a-l) --Based on Gilbert (1978)
  31. 31. Initial E2R2P Findings Problem Identification 31 What Freshouts Do on the Job— Successfully and Otherwise Communication and Teamwork Design 12% Analysis Technical fundamentals Other, 23% Software skills 12% Problem solving Motivation 2% Positive attitude 2% Leadership2% 9% Work Ethic Circuit debug2% 3% Trouble shooting and critical thinking 6% Real world engineering 3% 3% 3% Process Knowledge 6% 3% Programming 5% 5% Business System Knowledge Other
  32. 32. Initial E2R2P Findings Root Cause Analysis 32 19% 17% 18% Env. Info 17% 4% Env. Tool 5% 0% 3% 0% Env. Mot15% Ind. Know Ind. Cap 45% 57% Ind. Mot Managers 6% Freshouts 8% 11% 35% 11% 29% Dean (1997)
  33. 33. Next E2R2P Steps 33 Outreach to professional organizations, new company sponsors, and other universities Present survey, problem identification, and root cause analysis findings to company sponsors and participants for collaborative sensemaking Create a community of shared practice and concern https://sites.goog le.com/a/boisest Build to a corrective action forum ate.edu/ieeci/e2r with all stakeholders (a.k.a. “design solutions”) 2p
  34. 34. CIT Wrap Up 34 CIT has a track record spanning almost 60 years CIT is an evidence-based practice for performance improvement CIT is applicable in a wide variety of settings where performance is fuzzy CIT can be a valuable tool for performance improvement practitioners When performance is fuzzy, consider CIT!
  35. 35. References 35Bacdayan, P. (2002). Preventing stalled quality improvement teams: A written test of project selection ability. Performance Improvement Quarterly, 15(1), 47-66. doi: 10.1111/j.1937-8327.2002.tb00240.xButterfield, L.D., Borgen, W.A., Amundson, N.E., & Maglio, A.-S.T. (2005). Fifty years of the critical incident technique: 1954-2004 and beyond. Qualitative Research, 5(4), 475-497. doi: 10.1177/1468794105056924Clark, S., Collins, A., Kwan, J., & Sesnon, A. (2012). Tales from the field: Making service standards real for families in need. Performance Xpress, (August 1). http://www.performancexpress.org/2012/08/tales-from-the-field-making-service-standards-real-for-families-in-need/Craytor, J.K. (1968). Critical incident technique, programmed instruction and nursing education. NSPI Journal, 7(6), 12-18. doi: 10.1002/pfi.4180070606Dean, P.J. (1998). A qualitative method of assessment and analysis for changing the organizational culture. Performance Improvement, 37(2), 14-23. doi: 10.1002/pfi.4140370207Flanagan, J.C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327-358. doi: 10.1037/h0061470Flanagan, J.C. (1962). Measuring human performance. Pittsburgh, PA: The American Institute for Research.Hale, J.A. (2011). Competencies for professionals in school improvement. Performance Improvement, 50(4), 10-17. doi: 10.1002/pfi.20208Harless, J.H. (1986). Guiding performance with job aids. In M. Smith (Ed.), Introduction to performance technology (Vol. 1, pp. 106-124). Washington, DC: The National Society for Performance and Instruction.Hoffman, R.R., Coffey, J.W., Ford, K.M., & Carnot, M.J. (2001). Storm-lk: A human-centered knowledge model for weather forecasting. Paper presented at the Human Factors and Ergonomics Society 45th Annual Meeting, Minneapolis/St. Paul, MN.Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999a). Job task analysis. In Task analysis methods for instructional design (pp. 55-62). Mahwah, NJ: Lawrence Erlbaum Associates.Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999b). Procedural analysis. In Task analysis methods for instructional design (pp. 45-54). Mahwah, NJ: Lawrence Erlbaum Associates.Korte, R. (2010). „First, get to know them‟: A relational view of organizational socialization. Human Resource Development International, 13(1), 27 - 43. doi: 10.1080/13678861003588984Lundberg, C., Elderman, J.L., Ferrell, P., & Harper, L. (2010). Data gathering and analysis for needs assessment: A case study. Performance Improvement, 49(8), 27-34. doi: 10.1002/pfi.20170Marrelli, A.F. (2005). The performance technologists toolbox: Critical incidents. Performance Improvement, 44(10), 40-44. doi: 10.1002/pfi.4140441009Stone, D.L., Blomberg, S., & Villachica, S. (2009, April). Capturing and leveraging expert decision making and problem solving. Paper presented at the International Society for Performance Improvement, Orlando, FL. http://www.dls.com/1175_CTA.pdfThomas, D.M., & Bostrom, R.P. (2010). Vital signs for virtual teams: An empirically developed trigger model for technology adaptation interventions. MIS Quarterly, 34(1), 115-142.Van Tiem, D.M., Moseley, J.L., & Dessinger, J.C. (2012). Performance improvement/HPT model--an overview. In Fundamentals of performance improvement: A guide to improving people, process, and performance (3rd ed., pp. 41-59). San Francisco, CA: Pfeiffer. http://www.ispi.org/images/HPT-Model-2012.jpgWoolsey, L.K. (1986). The critical incident technique: An innovative qualitative method of research. Canadian Journal of Counselling, 20(4), 242-254.
  36. 36. Thank You 36Questions? Comments? SteveVillachica@boisestate.edu