This document discusses developing an online assessment of digital competence for Estonian students based on an Evidence-Centered Design (ECD) framework. It involves two university teams combining deductive and inductive approaches to build an assessment aligned with the DigComp framework and Estonian curriculum. The ECD framework layers assessment from domain analysis to implementation. A Bayesian network and practical tasks are proposed to assess knowledge, skills, and abilities. Piloting is planned in schools to test scenario-based tasks and Bayesian scoring.

1. Evidence-Centered Approach to
Online Assessment of Students’
Digital Competence
Mart Laanpere, Kai Pata (Tallinn University)
Mario Mäeots , Leo Siiman (University of Tartu)

2. Problem space
• Need to assess digital competence: a newly
defined key competences in Estonian national
curriculum for primary and secondary schools
• Requirements:
– Compliant with DigComp framework
– Implemented online using e-exam system EIS
– Scalability
– Automated assessment
– Reliability, validity

3. Digital competence
• Three competing approaches to digital
competence/competencies:
– Theory-based knowledge, derived from the
academic discipline of computer science
(computational thinking)
– Pragmatic skills, needed at the workplace (ECDL)
– Key competence, contextualised, closer to
“learning to learn” skill
• DigComp follows the second approach,
Estonian curriculum combines 2nd and 3rd

4. Teaching and assessing ICT skills in
Estonia
• 1986: Computer science as a compulsory
theoretical subject, no exam, teacher-created
tests
• 1996: Informatics as an elective subject, learning
generic ICT skills for office jobs
• 2000: ICT and media as cross-curricular theme,
pilot exam on national level (test + practical task)
• 2011: informatics as an elective subject, digital
competence as a key competence

5. Research design and organisation
• Two teams of researchers from two different
universities (University of Tartu & Tallinn
University)
• Combination of two approaches:
– Inductive/descriptive, bottom-up (Tartu team):
collecting and analysing the best practice from
large-scale comparative and national studies
– Deductive/prescriptive, top-own (Tallinn team):
building a new framework from scratch, deducing
the new type of practice

6. ECD framework (Almond,
Mislevy et al. 2015)
inspired by e-exams of
Cisco Network Academy;
Has been applied in
various contexts:
Assessing language
competence,
Inquiry skills in STEM
Etc.

7. Layered assessment architecture of ECD
Layer Role Key entities Examples
Domain
analysis
Collect info about
domain, teaching
practice, frames
Concepts; terminology;
tools; representation
forms
Curricula, rubrics, test
Domain
modelling
Express assessment
argument in narrative
form
KSAs; potential work
products;
potential observations
Design patterns
Assessment
framework
CAF
Express assessment
argument as blueprints
for tasks or items
Student, evidence, and
task models; student-
model, rubrics etc
Task templates
Imple-
mentation
Implement assessment,
presenting tasks or
items and gathering and
analyzing responses
Task materials (incl.all
materials, tools,
affordances); work
products
Rendering protocols
for tasks; IMS/QTI
representation of
materials & scores;
Delivery Coordinate interactions
of students and tasks;
task and test-level
scoring; reporting.
Tasks as presented; work
products as created;
scores as evaluated.
summaries for
individual and group-
level reports;

8. Assessment framework (CAF)
What?
How?
Where?
Extent?

9. Domain analysis: example

10. Performance taxonomy
Fact Concept Procedure Rule
Knows
Understands
Applies
HOTS
Based on: Merrill ja Feisel-Schmitz

11. Bayesian network
Information &
Data Literacy
Digital content
creation
Communication
& collaboration
Digital safety
Problem solving

12. KSA (Knowledge-Skills-Abilities)
P_Info P_Comm P_Content P_Safety P_Probl
T_Info
T_Comm
T_Content
T_Safety
T_Probl

13. Example: practical task
• Create a presentation that compares two
countries: Estonia and Slovenia
• Criteria:
– At least 4 slides/screens
– Diagrams based on reliable and recent data
– Images (IPR need to be followed, licensing)

14. Piloting
• Not completed, planned in 6-8 schools this
October
• Bayesian data analysis, evidence accumulation
• Scenario-based design of new task templates
• Exploring the solutions for simulation tasks
(inspired by examples collected by Tartu
group)

15. Questions for discussion
• Any experiences with ECD?
• How to combine inductive and deductive
approaches?
• Semi-automated assessment of practical
tasks?