Can progress in “Holistic admissions be measured”?<br />Emil T. Chuck, Ph.D.<br />Health professions Advisor<br />Term Ass...
Can progress in “Holistic admissions be measured”?<br />Competency-Based Evaluation of Predental Applicants<br /><ul><li>R...
Discuss whether the evaluation criteria fit professional competencies that are valued by the dental profession.
Compare dental and medical admissions outcomes trends as measured by the holistic rubric.</li></li></ul><li>How “the publi...
How prehealth advisors(?) view admissions<br />Comment #184: “I must laugh at the credentials that this white student has....
AAMC GPA/MCAT (2005-2007 accepted %)<br />
“The system produces the result for which it was designed.”Bob Witzberg MD, Boston University SOM, AAMC Holistic Review Pr...
Holistic Review:Choosing from a pool of applicants<br />Established substantive criteria for admissions that<br />Reflect ...
Competencies and Learning<br />April 5, 2003, UNC ballroom dance club practice with RobertasMaleckis, Credit: Emil Chuck (...
Goals of competency-based evaluation<br />Criteria should align with articulated technical standards of admissions (based ...
GMU Prehealth Evaluation Rubric<br />Compare to technical standards of admissions<br />Academic foundation (science and no...
Competencies and Recommendations<br />Newcomer: just starting out.<br />Novice: follows instructions without questioning.<...
Example self-assessment<br />Awards declared: I have made the dean’s list multiple times.<br />
EY2011 preapplication pipeline<br />
Rubric self-assessment<br />EY2011 Applicants (n=147)<br />Most confident:personal, emotional resilience and intelligence<...
Rubric self-assessment<br />EY2011 Applicants (n=39) Predental<br />Most confident:personal, emotional resilience and inte...
Solicited evaluators assessment<br />PRELIMINARY DATA ONLY (n=52), for EY2011 applicants, as of 24 Feb 2010.<br />Solicite...
Committee evaluators assessment<br />PRELIMINARY DATA ONLY (n=72), for EY2011 applicants, as of 26Feb 2010.<br />Committee...
Evaluators’ summary assessment<br />Committee EvaluatorsPRELIMINARY DATA ONLY (n=72), for EY2011 applicants, as of 26 Feb ...
Solicited evaluators assessment<br />PRELIMINARY DATA ONLY (n=53), for EY2011 applicants, as of 26 Feb 2010.<br />Solicite...
360-assessment reveals calibration issues<br />Applicants claim most confidence in their intrapersonal skills (personal re...
“Percent chance of admission” EY 2009<br />
GPA applicants/accepted EY 2009<br />
Average GPA’s for EY 2009 applicants<br />All admissions decisions (GMU)<br />
Average GPA’s for EY 2009 applicants<br />All AADSAS 2009 admissions decisions (GMU)<br />
Average GPA’s for EY 2010 applicants<br />PRELIMINARY admissions decisions (GMU) as of February 19, 2010<br />
Average GPA’s for AADSAS EY 2010 applicants<br />PRELIMINARY AADSAS 2010 admissions decisions (GMU) as of February 19, 201...
Translating rubric to evaluators<br />Competency-Based Holistic Evaluation of Prehealth Applicants<br />Chuck ET. The Advi...
Can holistic admissions be measured?<br />Current challenges<br />Applicants consider their competencies in their emotiona...
Trend towards holistic competencies<br />AAMC Holistic Review Project<br />National Postdoctoral Association Core Competen...
It’s time for a checklist!<br />Rubrics are checklists with an assessment angle.<br />
Committee evaluators assessment<br />PRELIMINARY DATA ONLY (n=65), for EY2011 applicants, as of 24 Feb 2010.<br />Majority...
Average GPA’s for EY 2009 applicants<br />All applicants (GMU)<br />
Recommendations for EY 2009 applicants<br />All admissions decisions (GMU)<br />
Recommendations for EY 2010 applicants<br />All admissions decisions (GMU) as of February 19, 2010<br />
Upcoming SlideShare
Loading in …5
×

ADEA 2010 Competency Based Evaluation Of Predental Applicants

1,708 views

Published on

New Idea Session on Competency-Based Evaluation of Predental Students from the data-driven evaluation side (predental advising and letters of evaluation).

Published in: Education, Technology
1 Comment
0 Likes
Statistics
Notes
  • Thank you for presenting at the recent 87th ADEA Annual Session & Exhibition in Washington, DC. Below is a your evaluation summary averages results based on the four outlined areas listed below.

    Ranking Scale: 4.0 – 1.0 with 4.0 being the Strongly Agree (the highest rating)

    Number of Responses 19

    Demonstrated Knowledge of the audience and the subject matter. = 3.58
    The content presented was relevant and delivered in a logical manner. = 3.26
    I would participate in another session led by this presenter. = 3.32
    Grand Total = 3.39
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total views
1,708
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
0
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide
  • http://www.nytimes.com/2010/02/15/education/15medschools.html
  • http://www.nytimes.com/2010/02/15/education/15medschools.html
  • http://www.aamc.org/data/facts/applicantmatriculant/table24-mcatgpa-grid-3yrs-app-accpt.htm
  • Wirtzberg quote: http://www.aamc.org/opi/holisticreview/resources/witzburg.pdf Taken from www.creid.ed.ac.uk/events/disabledlearners.../LTA_healey_roberts.ppt .
  • Royalty free photos: http://www.fotosearch.com/bigcomp.asp?path=CSP/CSP176/k1761429.jpgAAMC diversity self-assessment: http://www.aamc.org/opi/holisticreview/resources/selfassessmentworksheet.pdf
  • April 5, 2003, UNC ballroom dance club practice with RobertasMaleckis (personal copy: Emil Chuck)
  • Multiple intelligence theory
  • AtulGawande speaking at Politics and Prose, photo by Emil Chuck.
  • ADEA 2010 Competency Based Evaluation Of Predental Applicants

    1. 1. Can progress in “Holistic admissions be measured”?<br />Emil T. Chuck, Ph.D.<br />Health professions Advisor<br />Term Assistant Professor of Biology<br />George Mason University<br />2010 ADEA Annual Session<br />Saturday, February 27, 2010<br />3:00 to 3:25 pm, Potomac 5 (New idea session)<br />Some data shown are preliminary and still being collected for future evaluation.<br />Competency-Based Evaluation of Predental Applicants<br />
    2. 2. Can progress in “Holistic admissions be measured”?<br />Competency-Based Evaluation of Predental Applicants<br /><ul><li>Recognize how principles of holistic review are applied to predental advising for applicants and committee-based evaluations.
    3. 3. Discuss whether the evaluation criteria fit professional competencies that are valued by the dental profession.
    4. 4. Compare dental and medical admissions outcomes trends as measured by the holistic rubric.</li></li></ul><li>How “the public” views admissions<br />“Peter Allen applied to 30 medical schools after graduating from the University of Pittsburgh last year. Twenty-eight said no. …<br />“His pre-med adviser told him that with his 3.3 grade-point average, he should apply only to osteopathic schools, but he persisted, and was admitted to The Commonwealth and New York Medical College in Valhalla, N.Y.”<br />“Expecting a surge in U.S. medical schools” by AnemonaHartocollis (Feb 14, 2010 NYT)<br />Peter Allen and Jessica Sidari, first-year students at The Commonwealth Medical College, a first-year school in Scranton, Pa<br />Credit: Niko J. Kallianiotis for The New York Times<br />
    5. 5. How prehealth advisors(?) view admissions<br />Comment #184: “I must laugh at the credentials that this white student has. As an advisor for pre-med students at an Ivy-league school, my Asian students with 3.7-3.8 GPAs in the sciences have a rough time even getting interviews. A 3.3 GPA is a laughable GPA and even more laughable coming from a school like Pitt.”<br />“Expecting a surge in U.S. medical schools” by AnemonaHartocollis (Feb 14, 2010 NYT)<br />Peter Allen and Jessica Sidari, first-year students at The Commonwealth Medical College, a first-year school in Scranton, Pa<br />Credit: Niko J. Kallianiotis for The New York Times<br />
    6. 6. AAMC GPA/MCAT (2005-2007 accepted %)<br />
    7. 7. “The system produces the result for which it was designed.”Bob Witzberg MD, Boston University SOM, AAMC Holistic Review Project<br />
    8. 8. Holistic Review:Choosing from a pool of applicants<br />Established substantive criteria for admissions that<br />Reflect a justifiable balance among academic accomplishments and personal factors in applicants given the school’s mission-related goals.<br />The school must have evidence for its admissions considerations with regard to:<br />Academic background (major, GPA, letters, essays, etc.)<br />Personal characteristics (background)<br />Personal attributes (altruism, leadership, service)<br />Personal experiences (relevant experiences, life experiences)<br />Other criteria (please specify)<br />
    9. 9. Competencies and Learning<br />April 5, 2003, UNC ballroom dance club practice with RobertasMaleckis, Credit: Emil Chuck (personal)<br />
    10. 10. Goals of competency-based evaluation<br />Criteria should align with articulated technical standards of admissions (based on multiple personality theory). <br />“Legal compliance” arguments tend to have more resistance to acceptance.<br />Criteria should be transparent and independent of professional degree pursued by applicant to should reflect qualities desired for an interprofessional health care workforce.<br />Can compare pools of applicants in the same year for different paths.<br />Applicants should become familiar with competency-based evaluations as it will be a facet of their professional training and practice.<br />Criteria should assist students with self-evaluation and references with confirmatory evaluation (360-evaluation)<br />Quantitative measures should inform advising, program evaluation, and admissions outcomes.<br />
    11. 11. GMU Prehealth Evaluation Rubric<br />Compare to technical standards of admissions<br />Academic foundation (science and non-science breadth/rigor)<br />Scholarship and lifelong learning (problem-solving, research, entrepreneurship)<br />Social intelligence (interpersonal, teamwork, collegial)<br />Personal awareness (maturity, ethics, resiliency)<br />Managerial experience (time, financial, project, leadership)<br />Aesthetic or observational skills (art, music, literature)<br />Kinesthetic dexterity (lab methods, sculpture, electronics)<br />Communications skills (written/oral English, other)<br />Knowledge of the profession and health care system<br />Evaluators usually cannot comment on specific fitwith institutional mission and culture! <br />
    12. 12. Competencies and Recommendations<br />Newcomer: just starting out.<br />Novice: follows instructions without questioning.<br />Intermediate: memorizes steps, needs more practice.<br />Proficient: can perform the tasks satisfactorily (minor difficulties).<br />Competent: fluent; can improve the task/outcome by making minor adjustments.<br />Expert: tests improvements to tasks in a systematic manner.<br />Master: honored for exceptional skill development.<br />No/Concern: general newcomer/novice to competency concepts.<br />General: generally satisfactory/proficient in competencies<br />Strong: shows additional development (competent) with competencies<br />Enthusiastic: generally expert in competencies<br />Highly: consistently honored (especially by peer review processes) to be exceptional<br />
    13. 13. Example self-assessment<br />Awards declared: I have made the dean’s list multiple times.<br />
    14. 14. EY2011 preapplication pipeline<br />
    15. 15. Rubric self-assessment<br />EY2011 Applicants (n=147)<br />Most confident:personal, emotional resilience and intelligence<br />Least confident: academic foundation; mentoring/advising relationships.<br />
    16. 16. Rubric self-assessment<br />EY2011 Applicants (n=39) Predental<br />Most confident:personal, emotional resilience and intelligence; observational skills; manual dexterity<br />Least confident: academic foundation; mentoring/advising relationships.<br />
    17. 17. Solicited evaluators assessment<br />PRELIMINARY DATA ONLY (n=52), for EY2011 applicants, as of 24 Feb 2010.<br />Solicited evaluators may overrate applicants’ skills development.<br />Solicited evaluators rarely can comment on manual dexterity skills.<br />
    18. 18. Committee evaluators assessment<br />PRELIMINARY DATA ONLY (n=72), for EY2011 applicants, as of 26Feb 2010.<br />Committee interviewers are more likely to detect lower-competency applicants (novice to proficient).<br />Committee interviewers have more difficulty assessing manual dexterity & applicants’ collegial relationships with other advisors or faculty.<br />
    19. 19. Evaluators’ summary assessment<br />Committee EvaluatorsPRELIMINARY DATA ONLY (n=72), for EY2011 applicants, as of 26 Feb 2010.<br />Solicited Evaluators PRELIMINARY DATA ONLY (n=52), for EY2011 applicants, as of 24 Feb 2010.<br />
    20. 20. Solicited evaluators assessment<br />PRELIMINARY DATA ONLY (n=53), for EY2011 applicants, as of 26 Feb 2010.<br />Solicited evaluators have a relatively casual relationship or a knowledgeable relationship with applicants.<br />5/39 evaluators consider themselves “mentors” to applicant.<br />
    21. 21. 360-assessment reveals calibration issues<br />Applicants claim most confidence in their intrapersonal skills (personal resilience, hardship, evaluating criticism, faith).<br />Applicants are least confident in academic skills and having strong mentoring relationships.<br />Few solicited evaluators are “mentors” to applicants.<br />Solicited evaluators will overrate/exaggerate the development of applicants’ skills (except for manual dexterity) compared to committee evaluators.<br />
    22. 22. “Percent chance of admission” EY 2009<br />
    23. 23. GPA applicants/accepted EY 2009<br />
    24. 24. Average GPA’s for EY 2009 applicants<br />All admissions decisions (GMU)<br />
    25. 25. Average GPA’s for EY 2009 applicants<br />All AADSAS 2009 admissions decisions (GMU)<br />
    26. 26. Average GPA’s for EY 2010 applicants<br />PRELIMINARY admissions decisions (GMU) as of February 19, 2010<br />
    27. 27. Average GPA’s for AADSAS EY 2010 applicants<br />PRELIMINARY AADSAS 2010 admissions decisions (GMU) as of February 19, 2010<br />
    28. 28. Translating rubric to evaluators<br />Competency-Based Holistic Evaluation of Prehealth Applicants<br />Chuck ET. The Advisor (NAAHP), June 2009.<br />Available online as PDF <br />http://www.slideshare.net/etchuck/advisor-june-2009-competency-based-holistic-evaluations<br />Writing better evaluation letters (solicited and committee)<br />Competencies as a checklist of topics to cover.<br />Training admissions readers and interviewers.<br />Compare competencies against technical standards and admissions criteria.<br />Communicate and provide feedback (compare with any competency assessments after D1/D2/D4 ).<br />
    29. 29. Can holistic admissions be measured?<br />Current challenges<br />Applicants consider their competencies in their emotional maturity, integrity, and character as “expert”, as do solicited evaluators.<br />Evaluators rate most applicants’ skills as “expert” though they do not usually have close mentoring relationships.<br />Must better align evaluation rubric criteria with competency experiences/standards from professional education.<br />Current opportunities with rubric<br />“Research”-based standards of ideal health care professionals can be assessed, communicated, and “measured” with admissions outcomes.<br />Improve prehealth advising and evaluation letters for applicants.<br />Research any interprofessional (inter-institutional?) comparison of applicant holistic qualities.<br />Introduce applicants to competency-based evaluation processes common in professional education and development.<br />
    30. 30.
    31. 31. Trend towards holistic competencies<br />AAMC Holistic Review Project<br />National Postdoctoral Association Core Competencies<br />AAMC/HHMI Scientific Foundations for Future Physicians<br />Assessment projects throughout medical education<br />
    32. 32. It’s time for a checklist!<br />Rubrics are checklists with an assessment angle.<br />
    33. 33. Committee evaluators assessment<br />PRELIMINARY DATA ONLY (n=65), for EY2011 applicants, as of 24 Feb 2010.<br />Majority (84.6%) of committee evaluators suggest that reapplicants should have another set of MSAC interviews.<br />
    34. 34. Average GPA’s for EY 2009 applicants<br />All applicants (GMU)<br />
    35. 35. Recommendations for EY 2009 applicants<br />All admissions decisions (GMU)<br />
    36. 36. Recommendations for EY 2010 applicants<br />All admissions decisions (GMU) as of February 19, 2010<br />
    37. 37. Recommendations for EY 2010 applicants<br />All admissions decisions (GMU) as of February 19, 2010<br />

    ×