Preparing prehealth advisees to be able to articulate their preprofessional characteristics is critical to holistic evaluation. PRELIMINARY VERSION ONLY.

1. Using the George Mason University Preapplication to Evaluate Competencies Emil Chuck, Ph.D. George Mason University Differences in perception among advisees, solicited references, and advisory committee evaluators Results from pilot surveys and evaluations from the George Mason University Medical Sciences Advisory Committee (2008-2011)

2. Agenda Competencies Vision and Change in Undergraduate Biology Education Competencies assessment and evaluation MR5 Innovation Lab pre-professional characteristics Assessment of MR5 characteristics Reflections on experience essays Preliminary evaluation of MR5 PPR

3. “The system produces the result for which it was designed.”Bob Witzberg MD, Boston University SOM, AAMC Holistic Review Project www.creid.ed.ac.uk/events/disabledlearners.../LTA_healey_roberts.ppt

4. Holistic review Is it about developing new exams to allow others to excel? http://punkeys.files.wordpress.com/2008/02/monkey-and-elephant.jpg?w=510

5. The admissions process (holistic or not) http://www.bioresonance-technologies.com/bioresonance-technologies-6men.jpg

6. The Pre-Application Is a formal request for institutional evaluation. Provides information for committee interviews. Provides for compliance with advising goals. Allows for programmatic and institutional evaluation. Is completely online (no roaming paper copies). Has a deadline for completion. Other “prehealthpreapplications” Standard demographic information Transcript Extracurricular activities Personal comments/essay draft

7. Preapplication: NEAAHP “advisor access” Weblink: http://www.surveymonkey.com/s/neaahp2011 Access open until April 30.

8. Personal and professional characteristics The communication challenge of the prehealth advisee mindset

9. Competency: Steps of development Newcomer: not aware of rules. Novice: learning the basics. Intermediate: building on basics. Proficient: comprehends basics, begins to apply to simple situations. Confident: adapts basics to controlled, more complex situations. Needs some assistance with more complex situations. Master: independently or collaboratively tests or develops new rules or approaches to uncontrolled complex situations. Expert: recognized as an innovator or leader, creating new rules or paradigms. Note: While the nomenclature is not the same as Epstein and Hundert (JAMA 2002), the concepts are still similar.

10. Vision and Change in Undergraduate Biology Education (2011: AAAS, NSF) Five biology core concepts Evolution Structure and function Information flow, exchange, and storage Pathways and transformations of energy and matter Interconnectedness of systems Six science core competencies Apply the process of science Use quantitative reasoning Use modeling and simulation Appreciate interdisciplinary nature of science Communicate and collaborate with other disciplines Understand the relationship between science and society

11. Core competencies and disciplinary practices Vision and Change. 2011, p.7.

12. Competency: Academic foundation Newcomer: has not taken class. Novice: not passing the class. Intermediate: passes class with some difficulty. Proficient: passes class without problem (B or A grade). Confident: appreciates how knowledge from class can address new problems or questions, understands and discusses secondary and primary literature Master: begins to initiate, participate, present, and assimilate new knowledge; adept with creating and interpreting data. Expert: honored for discovery (teaching?) of new knowledge, developing overarching theories or paradigms. Note: While the nomenclature is not the same as Epstein and Hundert (JAMA 2002), the concepts are still similar.

13. Self-Assessments Handouts have the raw data.

14. Implementing three assessments to inform admissions officers about applicants From AAMC “Stepping Outside the Box…” presentation, November 7, 2010

15. Holistic evaluation of pre-applicants Rubric Self-Assessments Annual Survey and Pre-app Pre-App Essays and MSAC Interviews Solicited Letters including Rubric assessment “Standardized” MSAC Evaluation Letter Admissions Committees Modified from AAMC “Stepping Outside the Box…” presentation, November 7, 2010

16. Comparing competenciesGMU, AAMC, ADEA Integrity and ethics Critical thinking Academic foundation Lifelong learning Reliability and dependability Professionalism Social intelligence Practical intelligence Service orientation Communication & interpersonal skills Emotional intelligence Social, interpersonal, & teamwork skills Health promotion Aesthetic intelligence Kinesthetic intelligence Desire to learn Practice management & informatics Oral communication skills Written communication Resilience and adaptability Patient care assessment, diagnosis, treatment planning Professional knowledge Patient care establishment & maintenance of oral health Relationship with mentors

17. SDN readers: What's the focus of your education? Student Doctor Network poll (2/27/2011)

18. SDN readers: Which of the following is your greatest weakness? Student Doctor Network poll (3/10/2011)

19. Pre-applicants are most confident in personal and interpersonal intelligence 2010-2011 EY2012 Pre-Application

20. Reflections on Experiences Mason Pre-Applications

21. Mason Reflection on Experience No extracurricular activities inventory Applicants write about each of the 9competencies by describing an experience, activity, or accomplishment that demonstrates it. Total of 15 essays. Responses have no character limit. Behavioral questions are asked.

22. Competencies… it’s coming… AMCAS 2012 will prompt applicants to summarize in essay form any three items that are indicated as “most meaningful experiences” (1325 character-limit per experience).

23. Competency: Integrity and Ethics (Responsible Conduct) Competencies Situation or Example Newcomer: just starting out. Novice: follows instructions without questioning. Intermediate: memorizes steps, needs more practice. Proficient: can perform the tasks satisfactorily (minor difficulties). Confident: fluent; can improve the task/outcome by making minor adjustments. Master: tests improvements to tasks in a systematic manner. Expert: honored for exceptional skill development. Newcomer: has not taken class. Novice: not passing the class. Intermediate: passes class with some difficulty. Proficient: passes class without problem (B or A grade). Confident: appreciates how knowledge from class can address new problems or questions. Master: participates, discusses, and presents new knowledge based on class material. Expert: honored for discovery (teaching?) of new knowledge. What other ways can one gain “expertise”?

24. Competency: Integrity and Ethics What was the worst thing you ever did or were an accomplice that you got away with (you were never discovered, caught, or punished for your actions)?

25. Competency: Integrity and Ethics What was the worst thing you ever did or were an accomplice that you got away with (you were never discovered, caught, or punished for your actions)? Text analysis (n=187) Lying (87, 45%) High School (79, 41%) Mom/Mother (66, 34%) Mistakes (61, 32%) Test (32, 16%) Book (31, 16%) Drink (26, 13%)

26. Some EY2012 Pre-application questions Outline your responses to the “On the spot essays”. Time: 2 minutes (think-pair/to/quad-share)

27. Some EY2012 Pre-application questions Outline your responses to the “On the spot essays”. Time: 2 minutes (think-pair/to/quad-share) OSCE! A classmate (Chris) wants you to look over his/her prehealth application essay draft. When you read the essay, it is difficult to understand Chris's thoughts or points made, though there are few grammar and spelling errors. When giving your feedback, you give Chris your essay as a model. Six months later, it is discovered by the admissions committee at a school that you both applied to that Chris's essay is 75% similar to yours. What would you do? Social intelligence, Practical intelligence Integrity and Ethics

28. Some EY2012 Pre-application questions Outline your responses to the “On the spot essays”. Time: 2 minutes (think-pair/to/quad-share) How would you give your patients a greater sense of dignity as a future health care provider? Social intelligence, Knowledge of the profession Service orientation

29. Chris plagiarism essays (n=187)

30. How would you give your patients a greater sense of dignity as a future health care provider? Wordle cloud: 10 essays, January 6, 2011

31. Case: Abstract writing Scientific abstracts Introduction statement. “Gap in knowledge” statement. Hypothesis statement. Methodology to test hypothesis. Some results. Preliminary conclusions and impact. Do you teach this to all your science majors? Don’t you teach this to all your science majors?

32. Me with Jorge Chaim PhDSACNAS 2010, Anaheim CA

33. Don’t be naïve… Many applicants now are being “coached.”

35. Their actions and resulting consequences

36. What they learned as a result

37. STAR-F method“Stepping Outside the Box…”, 2010 AAMC Annual Meeting; GemmaScallon, Interviewing Skills workshop, GMU PrehealthiTunesU podcasts

39. Source for my experience with writing interview/essay prompts.Me with John O’Hurley, USAD 2001 Anchorage AK

41. Career services offices offer behavioral interviewing workshops routinely.

42. Computer-based and online tools exist for practice.

43. More importantly, one CAN assess/train candidates for situational thinking skills.

44. Situation

45. Task

46. Action

47. Result

48. Reflection/FeelingGemmaScallon, Interviewing Skills workshop, GMU PrehealthiTunesU podcasts (free) Creative commons 2011 by-nc-sa 3.0 unported license.

49. On the spot essays: how to coach How would you give your patients a greater sense of dignity as a future health care provider? Situation (Problem) drawn from one’s experiences Health care setting Community service Personal experience Hypothetical Task (Method): Greater sense of dignity Must be able to own a definition of dignity Action (Data/Results) What happened in the situation? Result (Metrics of Progress) How did you measure an improvement in a sense of dignity? Reflection (Conclusion) How did this experience enrich your knowledge about being a health care professional? Creative commons 2011 by-nc-sa 3.0 unported license.

50. On the spot essays: how to coach How would you give your patients a greater sense of dignity as a future health care provider? Written answers are “untimed”, so get accustomed to thinking in STAR-F format. Repeat again (and again), but now use a different situation. Start condensing your answers within 3 minutes. Practice, practice, practice, practice your oral answers. Master class format: 6 people, 12 questions (or randomly draw from 20 topics or scenarios), one timer, 4 minute answers after 1 minute of “note taking”. Give time for collective critiques (write down). Variation: make it a “conversation” between two people, with the other four acting as judges. Creative commons 2011 by-nc-sa 3.0 unported license.

51. On the spot essays: how to coach What is your most meaningful experience that is germane to your pursuit of a health professional career? (Modified AMCAS experience mini-essays.) Situation (Problem): describe where, when, what? Task (Method): what was one’s role? Action (Data/Results): what did you do in that role? Result (Metrics of Progress): what was the impact? Reflection (Conclusion): why is this meaningful? How did this experience enrich your knowledge about being a health care professional? Creative commons 2011 by-nc-sa 3.0 unported license.

52. It may take a while to train… Elephants forced to walk across metal tightropes in Thailand tourist attraction, Daily Mail, 11/03/2010

53. The Multiple Mini-Interview: The view from behind the curtain “On your mark, get set, interview!” Inside Stanford Medicine, January 10, 2011. Dr. Garcia, Stanford. http://med.stanford.edu/ism/2011/january/interview-0110.html

54. If MMI performance cannot be coached… What are we assuming with MMI performance? Innate characteristics of applicant that cannot be changed? Learned behaviors that can take time/training to modify? Situational anxiety due to lack of preparation or familiarity? Overpreparation: running out of steam during later stations. Current data are looking at students with a 20-40% chance of acceptance. What about the ones triaged from the process (never get to interview phase)?

55. Final considerations

56. Assessment Gradient DIFFICULT HIGH Oral interview or presentations EASE OF ASSESSMENT POTENTIAL FOR ASSESSMENT OF LEARNING Essays, reports, research papers Short answer Models, concept maps,quantitative responses EASY Multiple choice, T/F LOW Vision and Change, 2011, p.24

57. Preliminary conclusions Majority of preapplicants self-rate highly in all six AAMC PPR characteristics. Solicited references tend to consider preapplicants as “expert” in AAMC PPR characteristics more than preapplicants. Committee interviewers view preapplicants favorably with respect to PPR characteristics, though not as highly as solicited references. Committee readers (no interview) are much less likely to rate preapplicants as master or expert in these competencies. Most applicants could be more easily eliminated based on inability to articulate competency development in the written essays.

58. Barriers to Implementation AAMC My own opinions Provides comparable information that can be triangulated with information from other sources. This information would be available to admissions committees early enough to select interviewees. Signals the value of personal competencies and situational reasoning to applicants and the pre-health community. Using these measures might change the mix and increase the racial/ethnic and economic diversity of the interview pool A trained evaluator must sift through applicant information and be immune to halo effects. Situational reasoning = Situation, Task, Action, Result, ReFlection (STAR-F) or Challenge, Action, Result, Reflection (CAR-F) method can be/is being taught (coached). My data suggest that holistic evaluation process has helped applicants from diverse background only if they are “highly recommended.”

59. Intrapersonal management skillsSocial, interpersonal, and teamwork skills An autographed copy of “House Calls” will be available for the Silent Auction. Please bid! Me with Dr. Patch Adams, 2011 AMSA National Conference, March 12, Arlington VA

60. Holistic AdmissionsResilience and Dependability “I meant what I said and I said what I meant, And an elephant's faithful, one hundred per cent!”