The document discusses the integration of evidence-based research into educational technology (edtech) to enhance children's learning experiences. It highlights the importance of collaboration among educators, designers, and researchers in developing effective edtech solutions, while also addressing issues like equity, accessibility, and the need for a scientific approach in edtech design. Key concepts include various types of evidence, the role of sensory learning, and the efforts to ensure that no child is left behind in the educational landscape.

1. EdTech
Evidence
Professor Natalia I. Kucirkova
University of Stavanger, Norway
WiKIT AS

2. • a global network with the shared vision of
integrating the science of reading with the practice
and policy of children’s digital books
• 100+ members from 23 countries
• a community of children’s digital publishers
with an interest in research-based e-books’ design,
implementation and evaluation • Kucirkova, N. (2016). iRPD—A framework for
guiding design‐based research for iPad apps. British
Journal of Educational Technology, 48(2), 598-610.
www.childrensdigitalbooks.com

3. Multisensory learning
• Sensory learning
• “Hidden” senses
• Future-proofing education
Kucirkova, N. (2021). The future of the self: Understanding personalization in childhood and beyond. Emerald
Publishing Limited.
Kucirkova, N. (2022). The explanatory power of sensory reading for early childhood research: The role of hidden
senses. Contemporary Issues in Early Childhood, 14639491221116915.
Ice-breaker exercise

4. WiKIT’s network of learning scientists
• University spin-out based in Norway, with a
global network of specially trained researchers
• Bridging science and academia to advance
learning for all learners
• Affordable access to latest science in various
subject areas at three Levels of Services
www.wikit.no
• We want EdTech to be evidence-driven and to integrate top-
level science with their products & services
• We match high-calibre scientists with ethical EdTech to
accelerate their evidence journey and innovation potential
Professor Natalia Kucirkova and Kari Wiig

5. Evidence definitions
• Obtained by rigorous, independent and objective research
Teachers’ views and usability reports are part of evidence-gathering but
are not “research”
By research we mean rigorous, independent and objective evaluation
of the EdTech’s use on children’s learning
Research is conducted to find out when and how an EdTech works
• For which children does it work best?
• For which skills?
• For which types of teaching?
Use can be by the child alone (independent learning) or in the
classroom (collaborative learning)
Teachers’ implementation of EdTech in their classroom (their pedagogy)
can be part of the evaluation study

6. The “5Es” of EdTech Evidence Evaluation
Use progressive
philanthropy to
advance initiatives
that support the
learning of all
students.
Support authentic
community insights
and multi-
stakeholder
collaborations
through diverse
grant types.
Protect innovation
through transparent
funding of EdTech’s
documented
commitment to an
efficacy portfolio.
Create funding
flows and
conditions
accountable to
EdTech users and
open data.
Deploy
unrestricted and
mission-oriented
funds to tackle
educational
disruptions
resulting from
climate change.
©WiKiT, 2023
EFFICACY EFFECTIVENESS EQUITY ETHICS ENVIRONMENT
Fair icons created by Eucalyp – Flaticon
https://www.alliancemagazine.org/blog/how-can-philanthropy-catalyse-a-system-wide-change-in-edtech/

7. Usability/Pedagogy
Conceptual
Qual/Quant
Validation
Level 1 “Evidence-ready”
Testbeds,
accelerators
certifications, reviews
Level 2 Mentorship
Subscription
Level 3 “Validation” with
partners
Customer reviews
Feasibility/usability testing
Pedagogy alignments
Conceptual evidence
Iterative Logic Model
Theory of change & action
Observations, Interviews
Experiments, Pre/post &
correlational studies
RCTs, replications &
longitudinal studies
Predictive validity
WiKIT’s partners
WiKIT
WiKIT &
partners
Evidence types
Evidence-based EdTech
Tiers of Evidence in
National Frameworks
EdTech’s evidence journey

8. High-quality EdTech Design
Kucirkova, N. (2017). iRPD—A framework for guiding design‐based research for iPad apps. British Journal of
Educational Technology, 48(2), 598-610.
iRPD: high-quality EdTech is
developed through a
partnership between
practitioners (teachers),
designers (developers) and
researchers (scientistsI
Influencing factors: different
government pressures,
financing options,
Shared epistemology: While
different stakeholders have
different motivations, they all
want to positively advances
children’s learning

9. Learning
sciences
and
EdTech
design
Return on investment AND return on
learning
Injecting learning science principles into
the design
Research-based principles and estimated
guesses
Pedagogy from the classroom

10. Approaches
to
evaluation
It is not just EdTech: push for evidence-based education for
more than a decade
Some educational researchers follow the efficacy movement
focused on Randomised Controlled Trials as the golden
standard
Some educational researchers value equally RCTs and other
types of evidence (descriptive studies)
International consensus is that each method answers
different questions and EdTech need a portfolio of evidence
to demonstrate impact
International drive is to ensure that no child is left behind:
and data- based technologies and science can support this
mission

11. Inequity and educational gaps
E.g. Access to hardware (PCs, laptops, iPads…)
Access to stable wifi
E.g. Teachers, caregivers who know
and can use EdTech with children
E.g. Content in local languages and in
alignment with national curriculum
Design that is data- and evidence-based
“The Digital Divides”

12. Missing privacy, quality design and localisation
• “How Dare They Peep into My Private Life?”, Human Rights Watch, 2022
• How educational are “educational” apps for young children? App store
content analysis using the Four Pillars of Learning framework, Meyer et al.,
2022
• Prevalence and Characteristics of Manipulative Design in Mobile
Applications Used by Children, Radesky et al., 2022
• What are we downloading for our children? Best-selling children’s apps in
four European countries, Sari et al., 2017
Hollow, D., & Jefferies, K. (2022). How EdTech Can Be Used to Help Address the Global Learning Crisis: A
Challenge to the Sector for an Evidence-Driven Future [Preprint]. EdTech Hub.
https://doi.org/10.53832/edtechhub.0115

13. Web-based evidence at providers’ site
https://www.amiralearning.com/research.html
https://sparx-learning.com/evidence/

14. Top Ten Evidence-Based EdTech
https://www.edtechdigest.com/2023/09/29/the-ten-top-evidence-based-edtechs/

15. A company’s evidence portfolio
Evidence portfolio
Reviews Internal research Scientific evidence (indirect) Scientific evidence (direct)

16. How much research do
you need to be
“evidence-based”?
Have read some literature about what is important
Causal testing and surveys with teachers
We had an experiment or external evaluation of our
programs
We ran an RCT trial
Foundational
Formative
Summative
Formative
Summative
At which level
of evidence is
your product?

17. Summative
Formative
Foundational
Surveys
Interviews
Literature review
Desktop research
Case study
Observational study
Usability study
Correlational study
Experiment
Randomised controlled trial
Figure adapted from: Zielezinski, M. (2019) Demystifying evidence in EdTech, presentation at EdSurge Fusion 2019, Available from SlideShare:
https://www.slideshare.net/molly_bullock/demystifying-evidence-in-edtech
Types of research (simplified)

18. Effectiveness versus efficacy
EFFICACY EFFECTIVENESS
Example questions Does the EdTech work as it was
designed to work?
Does the use of the EdTech benefit
children in the classroom?
Testing site “Ideal classroom” selected based on
strict criteria
Typical classroom
Participants Children selected based on
inclusion/exclusion criteria
All children in a given classroom
Intervention EdTech is used according to an
established protocol
EdTech is used flexibly, parallel with
other tools already in the classroom

19. Types of qualitative research
Qualitative
tools
Record-
keeping
Interviews
Observations
Case study
Qualitative
methods
Ethnography
Phenomeno
logy
Action
research
Narrative

20. Types of quantitative research
• Surveys
• Experiments
• RCTs

21. App
Communication
platform
Professional
dev. training
Learning
management
system
Online
assessment
EdTech
Meso: Classrooms
Micro: Child
Macro: : Countries
Machine
learning
technique
Micro
Meso
Macro
Your aims of impact

22. Which learning
outcomes are
you aiming for?
With your EdTech…
• Do children create or consume content?
• Do children improve their thinking or
behaviour?
• Do children improve their cognitive learning
(such as memory or language) or affective
learning (such as empathy and social skills)?
Aim for “both” in all categories!
Figure from Kucirkova, N. (2017) Digital Personalization in Early Childhood:
Impact on Childhood, London: Bloomsbury Academic.
Independent
learning
Collaborative
learning

23. EdTech learning design integrated with science
Design that integrates learning
and developmental science
supports children’s:
• inquiry and investigation
• application of knowledge to new
• production of ideas and solutions
• collaborative problem-solving
… Beyond rote learning …
Breaking away from the “learning
crisis”
The “whole child” approach
Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2020). Implications for educational practice of the
science of learning and development. Applied developmental science, 24(2), 97-140.

24. The scientific
mindset

25. Supporting
EdTech
founders to
develop a
scientific
mindset
• Scientific mindset is “the higher order
skills of analysis and ability to make a
scientific assessment of data and do this
creatively” (Pearson et al., 2009)
• Nurturing the capacity to independently
and innovatively interpret research to
address research problems.
• Developing skills that can help founders
interpret scientific results in an unbiased
and critical way.
• EdTech founders are invited to shadow our
researchers to be part of this evaluation
process and learn skills “on the job”.
Pearson, M., Cowan, A., & Liston, A. (2009). PhD education in science: Producing the scientific mindset in biomedical
sciences. In Changing practices of doctoral education (pp. 112-124). Routledge.

26. Conversation cards by WiKIT
www.wikit.no

27. Multimedia theory and learning principles
Mayer, R. E. (2005). Cognitive theory of multimedia learning. The Cambridge handbook of multimedia learning, 41, 31-
48.
(a) a multimedia effect—students learn more deeply from words
and pictures than from words alone
(b) a coherence effect—students learn more deeply when
extraneous material is excluded rather than included
(c) a spatial contiguity effect—students learn more deeply when
printed words are placed near rather than far from corresponding
pictures
(d) a personalization effect—students learn more deeply when
words are presented in conversational rather than formal style

28. Which metrics will you
implement?
Which research studies will
you prioritise?

29. Further reading
For ideas, critical theories, examples of practice, design new innovations

30. How to use EdTech in classrooms? Ideas for teachers

31. For references, follow-ups…
• Natalia.Kucirkova@uis.no
• https://www.linkedin.com
/in/
natalia-kucirkova/
• https://twitter.com/nkuc
irkova
• https://researchgate.net
/