The document discusses the WatchMe project, which aims to develop a mobile electronic portfolio system to improve feedback and assessment for competency-based higher education. The system will integrate workplace-based feedback and assessment using modules for feedback, visualization, and e-assessment of entrustable professional activities (EPAs). It will combine the EPA approach with learning analytics to provide tailored feedback based on assessments. The portfolio system, called ePASS, will offer just-in-time feedback to learners before assessments and aggregate assessment data in visualizations.
2. 1. Utrecht University, NL
2. University Medical Centre Utrecht, NL
3. Szent Istvan University, Hungary
4. University of Tartu, Estonia
5. Universitätsmedizin Charité Berlin, Germany
6. University of California San Francisco, USA
7. Maastricht University, NL
8. Mateum, NL
9. University of Reading, UK
10.Jayway, Denmark
11.NetRom, Rumania/NL
3. WATCHME’S AIM
Solve efficiency and quality problems of workplace-
based feedback and assessment by means of a
mobile electronic portfolio system that is enhanced
with a feedback module and a visualization module.
4. - Develop complex competences
- Integrated in context
- Demands long learning trajectories in workplace
- Deliberate practice: feedback and reflection
9. ENTRUSTABLE PROFESSIONAL ACTIVITY
Task based instead of construct based approach
Crucial question: would I entrust this candidate
unsupervised with this task? (with my sick mother,
animal or teach my daughter/son…)
An EPA is a task that an individual can be trusted to
perform unsupervised, in a given professional context,
once sufficient competence has been demonstrated.
International Competency-Based Medical Education Collaborators, March 18, 2014
Source: Ten Cate
10. COMBINE EPA APPROACH WITH
LEARNING ANALYTICS
Application of probabilistic student models that enable
tailored visualised feedback based on multi sorted
assessments
Bayesian models provide detailed model of trainees’
entrustability
21. EXAMPLE JUST-IN-TIME FEEDBACK
Before workplace-based
assessment is performed
EPASS displays just-in-time
feedback concerning learner.
E.g.: “remind your
communication skills
towards patient. Explain
treatment in simple words”.
25. COMBINING THE COMPETENCY-FRAMEWORK WITH
ENTRUSTABLE PROFESSIONAL ACTIVITIES
Medical expert
Communicator
Collaborator
Manager
Health advocate
Scholar
Professional
+
+
+
+
+
++
++
+
+
+
+
++
++
+
++
++
+
++
+
++
EPA1 EPA2 EPA3 EPA4 EPA5
AssessmentSource: Ten Cate
Editor's Notes
Portfolios contain selected evidence of performance and products accompanied by reflections.
Underuse of potential data and not very well tailored to worksituations (not timely).
Drawbacks:
Authentic, personal unique character of evidence, difficults scoring.
Full standardization impossible, asks for interpretation of assessor (no golden standard or external criterion).
Impacts reliability and validity of the assessment.
Limited impact on learning
Low motivation of students
Aim: improve feedback and assessment with an eportfolio; Continuing education at the workplace has seen increasing demand as a crucial means of acquiring the requisite professional knowledge and skills. Electronic Portfolios (e-portfolios) for workplace training provide a global view of each trainee’s progress but offer no dynamic feedback to exploit the rich learning assessment data that could be analysed to support responsive adaptation for more efficient and rewarding training.
De basis is een bestaand digitaal portfoliosysteem genaamd Epass, waarover zometeen meer.
Leg uit waarom 3 disciplines
Canmeds toelichten
Subdivisions Risk of checklist approach
142 milestones
PBLI on Evalue
When we use epas a combined task and construct approach occurs. Instead of analytic approach; combine
Acting is not allowed: more observation
Acting is allowed under direct supervision: supervision physically present; proactive supervision
Acting is allowed under indirect supervision: supervision not physically present, but quickly available if needed; reactive supervision
Acting is allowed “unsupervised”:under clinical oversight; distant/backstage supervision; post hoc report
Supervision may be provided to juniors (to junior residents, to medical students at level 2 or 3)
A formal acknowledgment happens at level 4
To improve possibilities for feedback and assessment, bring it near to dialogue
Crucial question for supervisors
clinical practice medical graduates have to cope with unfamiliar clinical situations and tasks. These are situations that. It is clinicians/supervisors’ daily practice to determine whether or not a medical trainne is ready to cope with unfamiliar situations and tasks in a clinical environment. They observe adspeak a trainee at work and assess how well the act in a clinical setting. ?next, they determin whehter they would trust them with tasks and situations that are not shown. When is a medical trainee ready for independent practice? This question bothers medical educators for decades and underlies the decision to entrust clinical tasks.
Features:
Independently executable within a time frame
Observable and measurable in process and outcome
Suitable for entrustment decisions.
Assessment-result always framed as supervision level
Typically, EPAs can be scheduled and allocated to individuals.
EPA concept was designed to make CBME work
Recent technological ddevelopmetns enable the colelciton and use of behaviorual data on massive scale. Todays’ cmputional poser, and high quality interactive gapi user interfaces make it possible to bringsophisticated analysis to end users quickl
Milan, 2010: Bayesian student models aloow for a sound and detaile evaluation of each student, according to the granularity level defined by teachers
In stead of having only a final grade to measure students’ performance, the system will be ble to provide a more detialed model of trainees entrustability which contains informaiton about which parts the crrciulum the student is strggling wiht and whih parts he/she has already mastered. This information is essential to provide feeback, remediation and personlized instruction
node
Basis of project which we expand with student models and visualization modules
Providing detailed view on progress & dynamic just-in-time feedback
VIA
VIZ tools
Providing detailed view on progress & dynamic just-in-time feedback
VIA
VIZ tools
Providing detailed view on progress & dynamic just-in-time feedback
VIA
VIZ tools