1. COMPUTER SCIENCE
PASTEL: Evidence-based Learning
Engineering Method to create Intelligent
Online Textbook at Scale
Noboru Matsuda & Machi Shimmei
Center for Educational Informatics
Department of Computer Science
North Carolina State University

2. COMPUTER SCIENCE
Current MOOC Challenge
• Lack of individualization
– Ineffective learning (no learning!)
– Disengagement / drop-out
• Lack of systematic content creation & validation
– Where should we start from?
– How can we iteratively make it better?
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 2

3. COMPUTER SCIENCE
Current ITS Challenge
• Scalability / Generality
– Too expensive to build
– Mostly good for procedural skill acquisition
• What about conceptual learning?
• Robustness of Learning
– Luck of learning to solve with justifications
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 3

4. COMPUTER SCIENCE
Summary of Challenges
• To overcome the issues of MOOC and ITS, there
is a critical need to innovate a technology that
– provides adaptive instruction while promoting
synergetic learning
• An evidence-based curriculum development is
essential
– to build a large scale online course
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 4

5. COMPUTER SCIENCE
Our Solution
• Evidence-based learning engineering methods
– PASTEL (Pragmatic methods to develop Adaptive and Scalable
Technologies for next generation E-Learning)
• Adaptive Online Courseware
– CyberBook
= MOOC + Cognitive Tutors + Adaptive Control
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 5

6. COMPUTER SCIENCE
CyberBook
• Online courseware that provide macro-
adaptive scaffolding
– Problem balancing
– Mastery practice (cognitive tutoring)
– Proactive detection of unproductive failure
– Scaffolding on failure
• Incorrect quiz attempt and/or wheel-spinning
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 6

7. COMPUTER SCIENCE
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 7
CyberBook: Example

8. COMPUTER SCIENCE
CyberBook: Adaptive Scaffolding
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 8

9. COMPUTER SCIENCE
Skill Name Association
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 9

10. COMPUTER SCIENCE
CyberBook: Cognitive
Tutor Integration
Matsuda & Shimmei
CyberBook @ Intelligent Textbook WS 2019
10

11. COMPUTER SCIENCE
Technological Challenges
• Automatic validation of courseware content
• Rapid creation of high-quality skill-models
• Affordable creation of cognitive tutors
• Automatic creation of formative assessments
• Reliable prediction of unproductive failure
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 11

12. COMPUTER SCIENCE
Technology Innovations
• PASTEL: Evidence-based, iterative learning
engineering methods
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 12
SMART WATSON QUADL
RADARS RAFINE
Skill model
discovery
Cog tutor
authoring
Assessment
generation
Unproductive
failure prediction
Content
validation

13. COMPUTER SCIENCE
Evaluation Study
• Does macro-adaptive scaffolding promote learning?
• Middle School Science—Newton’s Law
– 11 units / 17 videos / 83 quizzes / 0 cog tutor
• High School Geometry—Coordinate Geometry
– 23 units / 0 videos / 179 quizzes / 14 cog tutors
• In-service teacher as a curriculum developer
13Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019

14. COMPUTER SCIENCE
Evaluation Study (Cont.)
• Participants
– 175 middle school science students
– 168 high school geometry students
• Pre-test, 3 intervention days, post-test
• Stratified RCT on pre-test
14Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019

15. COMPUTER SCIENCE
Results: Test Scores
15Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019

16. COMPUTER SCIENCE
Results – Num. Assessments
• Adaptive >> Non-adaptive for Science, but
not for Math
• No correlation between # assessments and
post-test when pre-test was controlled
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 16

17. COMPUTER SCIENCE
Results – Dynamic Link
• Surprisingly low use of dynamic link
– 0.4±1.8 in average
• Students figured out that corresponding
resources were almost always on the same
page
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 17

18. COMPUTER SCIENCE
Results – Video Use
• Adaptive == Non-adaptive
• Doer/Non-doer effect was not observed
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 18
(Koedinger et al., 2015)

19. COMPUTER SCIENCE
Results – Hint Use
• Hint on Failure Ratio
• Adaptive >> Non-adaptive for Science, but not for
Math. In particular low-prior students
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 19

20. COMPUTER SCIENCE
Conclusions
• CyberBook with the macro-adaptive scaffolding facilitated
learning
– Controlling the amount of formative assessment and
Dynamic link
– Only for the middle school Science, but not for high
school Math
• Hint use on failure among low-prior students was associated
with learning
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 20

21. COMPUTER SCIENCE
Future Works
• Redo the study with the balanced online
courseware
– Add cognitive tutors to Science
– Add videos to Math
• Redesign Dynamic Link
Matsuda & Shimmei CyberBook @ Intelligent Textbook WS 2019 21