Dr	
  Laura-­‐Jane	
  Smith,	
  Dr	
  Rosie	
  Belcher,	
  Dr	
  Deborah	
  Gill	
  	
  
Academic	
  Centre	
  for	
  Medi...
Upcoming SlideShare
Loading in …5
×

AMEE 2013 poster Does ePortfolio engagement predict performance in undergraduate medical education?

1,179 views

Published on

Poster presented at AMEE 2013 looking at whether engagement with an ePortfolio (the NHS Education for Scotland or NES ePortfolio that most Foundation Year doctors in the UK use) predicts performance in summative assessments.

Published in: Education
  • Be the first to comment

  • Be the first to like this

AMEE 2013 poster Does ePortfolio engagement predict performance in undergraduate medical education?

  1. 1. Dr  Laura-­‐Jane  Smith,  Dr  Rosie  Belcher,  Dr  Deborah  Gill     Academic  Centre  for  Medical  Educa?on,  UCL  Medical  School,  London,  United  Kingdom In  this  study  there  was  no  correla2on  between  engagement  with  an  ePor7olio  as  measured  by  a  novel  engagement   score,  and  end  of  year  summa2ve  assessment  performance.  A  focus  on  maximising  the  benefit  students  can  get  from   ePor7olio  use  in  rela2on  to  forma2ve  assessment  and  lifelong  learning  skills  is  advisable.     Take  home  message   Does  ePor7olio  engagement  predict  performance  in   undergraduate  medicine?   ePorDolios  are  a  core  component  of  postgraduate  medical  educa?on     and  training.  Increasingly,  undergraduates  mirror  postgraduate   learners  in  the  use  of  ePorDolios1.  Assessment  of  porDolios  is  usually   a  combina?on  of  grading  individual  components  and  a  measure  of   engagement  with  the  ePorDolio.  There  remains  some  debate  about   engagement  and  aMendance  as  predictors  of  success  in   undergraduate  and  postgraduate  medical  educa?on2.  Anecdotally  it   has  been  claimed  that  engagement  with  an  ePorDolio  is  a  predictor   for  other  outcomes  such  as  performance  in  end  of  year  wriMen  and   OSCE  examina?ons.     Background Summary  of  work   Students  were  required  to  complete  16  workplace-­‐based   assessments  (WPBAs),  aMend  and  record  4  mee?ngs  with  their   personal  tutor  and  upload  a  number  of  documents  to  their  personal   library  (both  wriMen  and  peer  assessments).  In  addi?on  students   were  invited  to  use  other  elements  of  the  ePorDolio  such  as  the   reflec?ve  logs  and  personal  development  plans,  but  these  were  not   required  or  assessed.  We  created  a  matrix  to  score  ePorDolio   engagement,  including  measures  of  students’  use  of  compulsory  and   non-­‐compulsory  elements  of  the  ePorDolio,  and  the  students’   organisa?on  of  contents  (Table  1).   Summary  of  results   As  ePorDolio  use  increases  in  undergraduate  medical  educa?on,  cau?on  must   be  exercised  to  ensure  that  they  are  not  used  as  surrogates  of  other  outcomes   without  a  robust  evidence  base.  We  plan  to  extend  the  study  to  the  three   collabora?ng  schools,  to  further  inves?gate  links  between  porDolio   engagement  and  assessment  outcomes.     Engagement  with  a  porDolio  is  required  for  postgraduate  learners  to  progress   in  training  and  retain  a  licence  to  prac?ce.  Rather  than  a  surrogate  for   knowledge  and  skills  acquisi?on,  porDolios  should  be  conceptualised  as   assessing  other  facets  of  professional  prac?ce  not  captured  by  exis?ng   assessment  methods.  Seen  in  this  light  they  offer  an  opportunity  to  support   and  develop  essen?al  lifelong  learning  skills,  freed  from  the  o[en  rigid   constraints  of  summa?ve  assessment  methodologies.             Conclusions We  assessed  90  randomly  selected  undergraduate  ePorDolios   (selected  using  a  random  number  generator)  and  compared   engagement  scores  to  end  of  year  summa?ve  assessment  scores.   We  used  SPSSv20  for  descrip?ve  sta?s?cs  and  to  calculate  the   Pearson  coefficient.     Mean  engagement  score  was  5.4  +/-­‐  1.75.  No  correla?on  was  found   between  engagement  score  (range  0  –  10)  and  overall  summa?ve   assessment  results  (r=0.11,  p=0.31),  or  with  individual  components   of  the  engagement  matrix.  For  example  when  WPBA  number  (range   9  –  40)  was  taken  as  a  con?nuous  variable  there  was  also  no   correla?on  (r=0.11,  p=0.32).     Authors 0 1 2 Workplace  based  assessments   <16   16    -­‐  20     21  –  30     Personal  Library  content*     Not  all  required   elements  present   All  required   elements  present   Personal  Library  content     No  extra  material   Extra  material     Personal  Library  organisa2on   Not  organised  into   folders  and/or  file   names  unclear   Some   organisa?on  into   folders   Completely   organised  into   folders   Reflec2ve  logs/  Personal   Development  Plan   Not  used   Used     Photo  upload  /  personal  profile   updated     No   Yes   Personal  tutor  reports     None   1  –  2   3  –  4     References 1.  Buckley,  Sharon  et  al.  “The  educa?onal  effects  of  porDolios  on  undergraduate  student  learning:  a  Best  Evidence  Medical  Educa?on  (BEME)  systema?c   review.  BEME  Guide  No.11.”  Medical  Teacher  (2009);  31(4):  282-­‐98   2.   Hyde,  Richard  M.,  and  D.J.  Flournoy.  “A  case  against  mandatory  lecture  aMendance.”  Journal  of  Medical  Educa?on  (1986);  61(3)  :  175-­‐176 Table  1.  Engagement  scoring  matrix   UCL  is  one  of  four  medical  schools   working  collabora?vely  with  NHS   Educa?on  for  Scotland  to  adapt  the   postgraduate  ePorDolio  currently  used   in  UK  Founda?on  Schools  for  use  in   undergraduate  medicine.  The   ePorDolio  collects  and  collates  large   amounts  of  electronic  data  on  students   about  their  engagement  with  the   programme,  and  the  first  year  of   adop?on  provided  an  opportunity  to   inves?gate  the  links  between  porDolio   engagement  and  assessment  success.   NHS   Educa?on   for   Scotland   UCL   Medical   School   Bristol   Medical   School   Glasgow   Medical   School   Brighton   and  Sussex   Medical   School   Maximum  score  was  14.  Students  comple?ng  the  compulsory  elements  with   no  addi?onal  elements  scored  4.       100   110   120   130   140   150   160   170   180   0   2   4   6   8   10   12   Summa2ve  end  of  year  examina2on   score   ePor7olio  engagement  score   Plot  of  ePor7olio  engagement  score  against  score  in   summa2ve  end  of  year  examina2on  score.    Plot  of  engagement  score  vs  examina2on  score  

×