SlideShare a Scribd company logo
c
Transforming Textbooks into
Learning by Doing Environments:
An Evaluation of Textbook-Based Automatic Question Generation
c
The Doer Effect and the AI Solution
The Doer Effect is the learning science principle that doing
formative practice while reading has a six times larger effect size
on learning outcomes than reading alone, a relationship that has
been shown to be causal [10–12].
Formative practice is good for students, but costly to create.
Automatic question generation can solve this problem.
c
The State of Research
In Kurdi et al.’s 2020 systematic review of research, there are
gaps identified in the current research:
• Only 1 study evaluated AG questions in a classroom setting
• Only 1 study generated feedback
• Only 1 study identified Bloom’s level
• Only 14 studies evaluated question difficulty
• There is no clear “gold standard” identified
c
AG Questions
c
The Data: 786,242 total
observations
c
Evaluating
Questions
We compare our AG questions to HA
questions in the same course using a
mixed-effects logistic regression model.
In the same course, how do our AG
questions compare to HA questions
on:
• Engagement
• Difficulty
• Persistence
c
Engagement
c
Engagement
• For all questions, the location of the
question in the course mattered.
• The recognition question types had
higher engagement than the recall
question types.
• The AG recognition type is similar to HA
recognition types, and the AG recall type
is similar to the HA recall type.
• There is no indication that students found
the AG question types problematic in
general and chose to answer them less
frequently.
To summarize the engagement analysis:
c
Difficulty
c
Difficulty
To summarize the difficulty analysis:
• AQ matching were often the the easiest, but
similar to other HA recognition types.
• AG FITB were similar to several other HA
questions, which were recognition types.
• HA FITB were often marginally to significantly
more difficult than AG FITB.
• Questions varied in difficulty across courses,
showing the impact of content.
c
Persistence
To summarize the persistence analysis:
• AG FITB had statistically different
persistence from all questions except HA
MCMS.
• HA FITB generally had lower persistence
than AG FITB.
• AG matching had similar persistence to other
HA questions.
c
In Summary
Students were not deterred by AG questions.
Engagement was impacted by placement in the course, and recognition vs recall
types.
AG questions were in the difficulty range of the HA questions.
Easy questions did not deter persistence, but very difficult questions did.
c
Thank You!
If you have any further questions, please contact us:
• Benny.Johnson@vitalsource.com
• Rachel.Vancampenhout@vitalsource.com
• Bill.Jerome@vitalsource.com

More Related Content

What's hot

Simposio “Ciencias e Inglés en la evaluación internacional”: Using assessment...
Simposio “Ciencias e Inglés en la evaluación internacional”: Using assessment...Simposio “Ciencias e Inglés en la evaluación internacional”: Using assessment...
Simposio “Ciencias e Inglés en la evaluación internacional”: Using assessment...
Instituto Nacional de Evaluación Educativa
 
Leveraging Canvas Quiz Analysis to Improve Your Online Assessments
Leveraging Canvas Quiz Analysis to Improve Your Online AssessmentsLeveraging Canvas Quiz Analysis to Improve Your Online Assessments
Leveraging Canvas Quiz Analysis to Improve Your Online Assessments
Michael Wilder
 
Peggy presentation II
Peggy presentation IIPeggy presentation II
Peggy presentation II
Tim Cheng
 
Dashboard design By Anu Vijayan
Dashboard design By Anu VijayanDashboard design By Anu Vijayan
Dashboard design By Anu Vijayan
Sigmoid
 
Leya5.11
Leya5.11Leya5.11
Leya5.11
guest31f81c
 
Sat hyderabad
Sat hyderabadSat hyderabad
Sat hyderabad
Manhttan Review
 
What is a test
What is a testWhat is a test
What is a test
zulfiqaralibehan
 
Powerpointfor Uploadpdf
Powerpointfor UploadpdfPowerpointfor Uploadpdf
Powerpointfor Uploadpdf
Jacqueline Rhoades
 
Standardized Tests, by Kathy and Mary
Standardized Tests, by Kathy and MaryStandardized Tests, by Kathy and Mary
Standardized Tests, by Kathy and Mary
marz_bar_angel_9999
 
Specialist research presentation
Specialist research presentationSpecialist research presentation
Specialist research presentation
Etowah High School
 
Reliability copy
Reliability   copyReliability   copy
Reliability copy
Saleha Jabeen
 
Instructional Strategies Divided Into Specific Behaviors
Instructional Strategies Divided Into Specific BehaviorsInstructional Strategies Divided Into Specific Behaviors
Instructional Strategies Divided Into Specific Behaviors
Amanda Nickel
 
Patuxent High School Collaborative Action Research Plan
Patuxent High School Collaborative Action Research PlanPatuxent High School Collaborative Action Research Plan
Patuxent High School Collaborative Action Research Plan
Matt Stone
 
What Does All This Testing Mean
What Does All This Testing MeanWhat Does All This Testing Mean
What Does All This Testing Mean
MtBethelElem
 
Declan Cathcart: Assessment of inquiry during a woodlice investigation
Declan Cathcart: Assessment of inquiry during a woodlice investigationDeclan Cathcart: Assessment of inquiry during a woodlice investigation
Declan Cathcart: Assessment of inquiry during a woodlice investigation
Joasia van Kooten
 
Itbs Presentation
Itbs PresentationItbs Presentation
Itbs Presentation
Jennifer J. Hebert
 
MEC JULY 2012 SPEAKING SET 2
MEC JULY 2012 SPEAKING SET 2MEC JULY 2012 SPEAKING SET 2
MEC JULY 2012 SPEAKING SET 2
Kisas Muet
 
Language testing and evaluation
Language testing and evaluationLanguage testing and evaluation
Language testing and evaluation
esra66
 

What's hot (18)

Simposio “Ciencias e Inglés en la evaluación internacional”: Using assessment...
Simposio “Ciencias e Inglés en la evaluación internacional”: Using assessment...Simposio “Ciencias e Inglés en la evaluación internacional”: Using assessment...
Simposio “Ciencias e Inglés en la evaluación internacional”: Using assessment...
 
Leveraging Canvas Quiz Analysis to Improve Your Online Assessments
Leveraging Canvas Quiz Analysis to Improve Your Online AssessmentsLeveraging Canvas Quiz Analysis to Improve Your Online Assessments
Leveraging Canvas Quiz Analysis to Improve Your Online Assessments
 
Peggy presentation II
Peggy presentation IIPeggy presentation II
Peggy presentation II
 
Dashboard design By Anu Vijayan
Dashboard design By Anu VijayanDashboard design By Anu Vijayan
Dashboard design By Anu Vijayan
 
Leya5.11
Leya5.11Leya5.11
Leya5.11
 
Sat hyderabad
Sat hyderabadSat hyderabad
Sat hyderabad
 
What is a test
What is a testWhat is a test
What is a test
 
Powerpointfor Uploadpdf
Powerpointfor UploadpdfPowerpointfor Uploadpdf
Powerpointfor Uploadpdf
 
Standardized Tests, by Kathy and Mary
Standardized Tests, by Kathy and MaryStandardized Tests, by Kathy and Mary
Standardized Tests, by Kathy and Mary
 
Specialist research presentation
Specialist research presentationSpecialist research presentation
Specialist research presentation
 
Reliability copy
Reliability   copyReliability   copy
Reliability copy
 
Instructional Strategies Divided Into Specific Behaviors
Instructional Strategies Divided Into Specific BehaviorsInstructional Strategies Divided Into Specific Behaviors
Instructional Strategies Divided Into Specific Behaviors
 
Patuxent High School Collaborative Action Research Plan
Patuxent High School Collaborative Action Research PlanPatuxent High School Collaborative Action Research Plan
Patuxent High School Collaborative Action Research Plan
 
What Does All This Testing Mean
What Does All This Testing MeanWhat Does All This Testing Mean
What Does All This Testing Mean
 
Declan Cathcart: Assessment of inquiry during a woodlice investigation
Declan Cathcart: Assessment of inquiry during a woodlice investigationDeclan Cathcart: Assessment of inquiry during a woodlice investigation
Declan Cathcart: Assessment of inquiry during a woodlice investigation
 
Itbs Presentation
Itbs PresentationItbs Presentation
Itbs Presentation
 
MEC JULY 2012 SPEAKING SET 2
MEC JULY 2012 SPEAKING SET 2MEC JULY 2012 SPEAKING SET 2
MEC JULY 2012 SPEAKING SET 2
 
Language testing and evaluation
Language testing and evaluationLanguage testing and evaluation
Language testing and evaluation
 

Similar to Transforming Textbooks into Learning by Doing Environments: An Evaluation of Textbook-Based Automatic Question Generation

Advancing Intelligent Textbooks with Automatically Generated Practice: A Larg...
Advancing Intelligent Textbooks with Automatically Generated Practice: A Larg...Advancing Intelligent Textbooks with Automatically Generated Practice: A Larg...
Advancing Intelligent Textbooks with Automatically Generated Practice: A Larg...
Sergey Sosnovsky
 
254457273-LET-Review-Assessment-of-Learning-Test-Items.pptx
254457273-LET-Review-Assessment-of-Learning-Test-Items.pptx254457273-LET-Review-Assessment-of-Learning-Test-Items.pptx
254457273-LET-Review-Assessment-of-Learning-Test-Items.pptx
ItsssClarizza
 
Guidelines-on-Test-Questions_to-present.pptx
Guidelines-on-Test-Questions_to-present.pptxGuidelines-on-Test-Questions_to-present.pptx
Guidelines-on-Test-Questions_to-present.pptx
febefaegane2
 
TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
 TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013) TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
TESTA winch
 
EDUC 6 ADEPT.pptx20240514.nnnnnnnnnnnnnnnn
EDUC 6 ADEPT.pptx20240514.nnnnnnnnnnnnnnnnEDUC 6 ADEPT.pptx20240514.nnnnnnnnnnnnnnnn
EDUC 6 ADEPT.pptx20240514.nnnnnnnnnnnnnnnn
Marjorie Malveda
 
TESTA, University of Leeds: 'Talking @ Teaching' (September 2013)
TESTA, University of Leeds: 'Talking @ Teaching' (September 2013)TESTA, University of Leeds: 'Talking @ Teaching' (September 2013)
TESTA, University of Leeds: 'Talking @ Teaching' (September 2013)
TESTA winch
 
B.ed. CET
B.ed. CETB.ed. CET
B.ed. CET
DrPritiSonar
 
TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA, Universtiy of Warwick SCAP Conference (July 2013)TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA winch
 
Changing the assessment narrative
Changing the assessment narrativeChanging the assessment narrative
Changing the assessment narrative
Tansy Jessop
 
What is the question? Impact of question length and illustration support on t...
What is the question? Impact of question length and illustration support on t...What is the question? Impact of question length and illustration support on t...
What is the question? Impact of question length and illustration support on t...
Technological Ecosystems for Enhancing Multiculturality
 
From 0 to 100, or how Collaboration & a State Bill Result in Redesign!
From 0 to 100, or how Collaboration & a State Bill Result in Redesign!From 0 to 100, or how Collaboration & a State Bill Result in Redesign!
From 0 to 100, or how Collaboration & a State Bill Result in Redesign!
EddieMath
 
TESTA - UNSW, Sydney Australia (September 2011)
TESTA - UNSW, Sydney Australia (September 2011)  TESTA - UNSW, Sydney Australia (September 2011)
TESTA - UNSW, Sydney Australia (September 2011)
TESTA winch
 
Grading : Concep,Question andd suggestions
Grading : Concep,Question andd suggestionsGrading : Concep,Question andd suggestions
Grading : Concep,Question andd suggestions
JRNRV Udaipur
 
1 why do testa
1 why do testa1 why do testa
1 why do testa
Tansy Jessop
 
Mc qs b.ed and principles of teaching for head ship test etc_ver-a-nov10-2015
Mc qs b.ed and principles of teaching for head ship test etc_ver-a-nov10-2015 Mc qs b.ed and principles of teaching for head ship test etc_ver-a-nov10-2015
Mc qs b.ed and principles of teaching for head ship test etc_ver-a-nov10-2015
Sabeeta Lohana
 
TESTA, University of Greenwich Keynote (July 2013)
TESTA, University of Greenwich Keynote (July 2013)TESTA, University of Greenwich Keynote (July 2013)
TESTA, University of Greenwich Keynote (July 2013)
TESTA winch
 
A study on effectiveness of online education
A study on effectiveness of online educationA study on effectiveness of online education
A study on effectiveness of online education
Adi Aditya
 
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
cccscoetc
 
LSE Assessment
LSE AssessmentLSE Assessment
LSE Assessment
Tansy Jessop
 
Excerpts from the GRE Playbook 2014
Excerpts from the GRE Playbook 2014Excerpts from the GRE Playbook 2014
Excerpts from the GRE Playbook 2014
BenchPrep Social
 

Similar to Transforming Textbooks into Learning by Doing Environments: An Evaluation of Textbook-Based Automatic Question Generation (20)

Advancing Intelligent Textbooks with Automatically Generated Practice: A Larg...
Advancing Intelligent Textbooks with Automatically Generated Practice: A Larg...Advancing Intelligent Textbooks with Automatically Generated Practice: A Larg...
Advancing Intelligent Textbooks with Automatically Generated Practice: A Larg...
 
254457273-LET-Review-Assessment-of-Learning-Test-Items.pptx
254457273-LET-Review-Assessment-of-Learning-Test-Items.pptx254457273-LET-Review-Assessment-of-Learning-Test-Items.pptx
254457273-LET-Review-Assessment-of-Learning-Test-Items.pptx
 
Guidelines-on-Test-Questions_to-present.pptx
Guidelines-on-Test-Questions_to-present.pptxGuidelines-on-Test-Questions_to-present.pptx
Guidelines-on-Test-Questions_to-present.pptx
 
TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
 TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013) TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
TESTA, SIAST Universities of Regina & Saskathewan Webinar (November 2013)
 
EDUC 6 ADEPT.pptx20240514.nnnnnnnnnnnnnnnn
EDUC 6 ADEPT.pptx20240514.nnnnnnnnnnnnnnnnEDUC 6 ADEPT.pptx20240514.nnnnnnnnnnnnnnnn
EDUC 6 ADEPT.pptx20240514.nnnnnnnnnnnnnnnn
 
TESTA, University of Leeds: 'Talking @ Teaching' (September 2013)
TESTA, University of Leeds: 'Talking @ Teaching' (September 2013)TESTA, University of Leeds: 'Talking @ Teaching' (September 2013)
TESTA, University of Leeds: 'Talking @ Teaching' (September 2013)
 
B.ed. CET
B.ed. CETB.ed. CET
B.ed. CET
 
TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA, Universtiy of Warwick SCAP Conference (July 2013)TESTA, Universtiy of Warwick SCAP Conference (July 2013)
TESTA, Universtiy of Warwick SCAP Conference (July 2013)
 
Changing the assessment narrative
Changing the assessment narrativeChanging the assessment narrative
Changing the assessment narrative
 
What is the question? Impact of question length and illustration support on t...
What is the question? Impact of question length and illustration support on t...What is the question? Impact of question length and illustration support on t...
What is the question? Impact of question length and illustration support on t...
 
From 0 to 100, or how Collaboration & a State Bill Result in Redesign!
From 0 to 100, or how Collaboration & a State Bill Result in Redesign!From 0 to 100, or how Collaboration & a State Bill Result in Redesign!
From 0 to 100, or how Collaboration & a State Bill Result in Redesign!
 
TESTA - UNSW, Sydney Australia (September 2011)
TESTA - UNSW, Sydney Australia (September 2011)  TESTA - UNSW, Sydney Australia (September 2011)
TESTA - UNSW, Sydney Australia (September 2011)
 
Grading : Concep,Question andd suggestions
Grading : Concep,Question andd suggestionsGrading : Concep,Question andd suggestions
Grading : Concep,Question andd suggestions
 
1 why do testa
1 why do testa1 why do testa
1 why do testa
 
Mc qs b.ed and principles of teaching for head ship test etc_ver-a-nov10-2015
Mc qs b.ed and principles of teaching for head ship test etc_ver-a-nov10-2015 Mc qs b.ed and principles of teaching for head ship test etc_ver-a-nov10-2015
Mc qs b.ed and principles of teaching for head ship test etc_ver-a-nov10-2015
 
TESTA, University of Greenwich Keynote (July 2013)
TESTA, University of Greenwich Keynote (July 2013)TESTA, University of Greenwich Keynote (July 2013)
TESTA, University of Greenwich Keynote (July 2013)
 
A study on effectiveness of online education
A study on effectiveness of online educationA study on effectiveness of online education
A study on effectiveness of online education
 
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
Results are (coming) in: First Look at Colorado’s Scorecard for Developmenta...
 
LSE Assessment
LSE AssessmentLSE Assessment
LSE Assessment
 
Excerpts from the GRE Playbook 2014
Excerpts from the GRE Playbook 2014Excerpts from the GRE Playbook 2014
Excerpts from the GRE Playbook 2014
 

More from Sergey Sosnovsky

Harnessing Textbooks for High-Quality Labeled Data: An Approach to Automatic ...
Harnessing Textbooks for High-Quality Labeled Data: An Approach to Automatic ...Harnessing Textbooks for High-Quality Labeled Data: An Approach to Automatic ...
Harnessing Textbooks for High-Quality Labeled Data: An Approach to Automatic ...
Sergey Sosnovsky
 
Toward Eliminating Hallucinations: GPT-based Explanatory AI for Intelligent T...
Toward Eliminating Hallucinations: GPT-based Explanatory AI for Intelligent T...Toward Eliminating Hallucinations: GPT-based Explanatory AI for Intelligent T...
Toward Eliminating Hallucinations: GPT-based Explanatory AI for Intelligent T...
Sergey Sosnovsky
 
Layout- and Activity-based Textbook Modeling for Automatic PDF Textbook Extra...
Layout- and Activity-based Textbook Modeling for Automatic PDF Textbook Extra...Layout- and Activity-based Textbook Modeling for Automatic PDF Textbook Extra...
Layout- and Activity-based Textbook Modeling for Automatic PDF Textbook Extra...
Sergey Sosnovsky
 
Exploring the Content Ecosystem of the First Open-source Adaptive Tutor and i...
Exploring the Content Ecosystem of the First Open-source Adaptive Tutor and i...Exploring the Content Ecosystem of the First Open-source Adaptive Tutor and i...
Exploring the Content Ecosystem of the First Open-source Adaptive Tutor and i...
Sergey Sosnovsky
 
Creating Session Data from eTextbook Event Streams
Creating Session Data from eTextbook Event StreamsCreating Session Data from eTextbook Event Streams
Creating Session Data from eTextbook Event Streams
Sergey Sosnovsky
 
Augmenting Digital Textbooks with Reusable Smart Learning Content: Solutions ...
Augmenting Digital Textbooks with Reusable Smart Learning Content: Solutions ...Augmenting Digital Textbooks with Reusable Smart Learning Content: Solutions ...
Augmenting Digital Textbooks with Reusable Smart Learning Content: Solutions ...
Sergey Sosnovsky
 
Interactions of reading and assessment activities
Interactions of reading and assessment activitiesInteractions of reading and assessment activities
Interactions of reading and assessment activities
Sergey Sosnovsky
 
Parallel Construction: A Parallel Corpus Approach for Automatic Question Gene...
Parallel Construction: A Parallel Corpus Approach for Automatic Question Gene...Parallel Construction: A Parallel Corpus Approach for Automatic Question Gene...
Parallel Construction: A Parallel Corpus Approach for Automatic Question Gene...
Sergey Sosnovsky
 
YAI4Edu: an Explanatory AI to Generate Interactive e-Books for Education
YAI4Edu: an Explanatory AI to Generate Interactive e-Books for EducationYAI4Edu: an Explanatory AI to Generate Interactive e-Books for Education
YAI4Edu: an Explanatory AI to Generate Interactive e-Books for Education
Sergey Sosnovsky
 
Automatic Question Generation for Evidence-based Online Courseware Engineering
Automatic Question Generation for Evidence-based Online Courseware EngineeringAutomatic Question Generation for Evidence-based Online Courseware Engineering
Automatic Question Generation for Evidence-based Online Courseware Engineering
Sergey Sosnovsky
 
Reading Comprehension Quiz Generation using Generative Pre-trained Transformers
Reading Comprehension Quiz Generation using Generative Pre-trained TransformersReading Comprehension Quiz Generation using Generative Pre-trained Transformers
Reading Comprehension Quiz Generation using Generative Pre-trained Transformers
Sergey Sosnovsky
 
Mathematical Language Processing via Tree Embeddings
Mathematical Language Processing via Tree EmbeddingsMathematical Language Processing via Tree Embeddings
Mathematical Language Processing via Tree Embeddings
Sergey Sosnovsky
 
Contextual Definition Generation
Contextual Definition GenerationContextual Definition Generation
Contextual Definition Generation
Sergey Sosnovsky
 
Generation of Assessment Questions from Textbooks Enriched with Knowledge Models
Generation of Assessment Questions from Textbooks Enriched with Knowledge ModelsGeneration of Assessment Questions from Textbooks Enriched with Knowledge Models
Generation of Assessment Questions from Textbooks Enriched with Knowledge Models
Sergey Sosnovsky
 
Using Semantics of Textbook Highlights to Predict Student Comprehension and K...
Using Semantics of Textbook Highlights to Predict Student Comprehension and K...Using Semantics of Textbook Highlights to Predict Student Comprehension and K...
Using Semantics of Textbook Highlights to Predict Student Comprehension and K...
Sergey Sosnovsky
 
Dental TutorBot: Exploitation of Dental Textbooks for Automated Learning
Dental TutorBot: Exploitation of Dental Textbooks for Automated LearningDental TutorBot: Exploitation of Dental Textbooks for Automated Learning
Dental TutorBot: Exploitation of Dental Textbooks for Automated Learning
Sergey Sosnovsky
 
What's in a textbook
What's in a textbookWhat's in a textbook
What's in a textbook
Sergey Sosnovsky
 
Using Programmed Instruction to Help Students Engage with eTextbook Content
Using Programmed Instruction to Help Students Engage with eTextbook Content Using Programmed Instruction to Help Students Engage with eTextbook Content
Using Programmed Instruction to Help Students Engage with eTextbook Content
Sergey Sosnovsky
 
Adding Intelligence to a Textbook for Human Anatomy with a Causal Concept Map...
Adding Intelligence to a Textbook for Human Anatomy with a Causal Concept Map...Adding Intelligence to a Textbook for Human Anatomy with a Causal Concept Map...
Adding Intelligence to a Textbook for Human Anatomy with a Causal Concept Map...
Sergey Sosnovsky
 
Interlingua: Linking Textbooks Across Different Languages
Interlingua: Linking Textbooks Across Different Languages Interlingua: Linking Textbooks Across Different Languages
Interlingua: Linking Textbooks Across Different Languages
Sergey Sosnovsky
 

More from Sergey Sosnovsky (20)

Harnessing Textbooks for High-Quality Labeled Data: An Approach to Automatic ...
Harnessing Textbooks for High-Quality Labeled Data: An Approach to Automatic ...Harnessing Textbooks for High-Quality Labeled Data: An Approach to Automatic ...
Harnessing Textbooks for High-Quality Labeled Data: An Approach to Automatic ...
 
Toward Eliminating Hallucinations: GPT-based Explanatory AI for Intelligent T...
Toward Eliminating Hallucinations: GPT-based Explanatory AI for Intelligent T...Toward Eliminating Hallucinations: GPT-based Explanatory AI for Intelligent T...
Toward Eliminating Hallucinations: GPT-based Explanatory AI for Intelligent T...
 
Layout- and Activity-based Textbook Modeling for Automatic PDF Textbook Extra...
Layout- and Activity-based Textbook Modeling for Automatic PDF Textbook Extra...Layout- and Activity-based Textbook Modeling for Automatic PDF Textbook Extra...
Layout- and Activity-based Textbook Modeling for Automatic PDF Textbook Extra...
 
Exploring the Content Ecosystem of the First Open-source Adaptive Tutor and i...
Exploring the Content Ecosystem of the First Open-source Adaptive Tutor and i...Exploring the Content Ecosystem of the First Open-source Adaptive Tutor and i...
Exploring the Content Ecosystem of the First Open-source Adaptive Tutor and i...
 
Creating Session Data from eTextbook Event Streams
Creating Session Data from eTextbook Event StreamsCreating Session Data from eTextbook Event Streams
Creating Session Data from eTextbook Event Streams
 
Augmenting Digital Textbooks with Reusable Smart Learning Content: Solutions ...
Augmenting Digital Textbooks with Reusable Smart Learning Content: Solutions ...Augmenting Digital Textbooks with Reusable Smart Learning Content: Solutions ...
Augmenting Digital Textbooks with Reusable Smart Learning Content: Solutions ...
 
Interactions of reading and assessment activities
Interactions of reading and assessment activitiesInteractions of reading and assessment activities
Interactions of reading and assessment activities
 
Parallel Construction: A Parallel Corpus Approach for Automatic Question Gene...
Parallel Construction: A Parallel Corpus Approach for Automatic Question Gene...Parallel Construction: A Parallel Corpus Approach for Automatic Question Gene...
Parallel Construction: A Parallel Corpus Approach for Automatic Question Gene...
 
YAI4Edu: an Explanatory AI to Generate Interactive e-Books for Education
YAI4Edu: an Explanatory AI to Generate Interactive e-Books for EducationYAI4Edu: an Explanatory AI to Generate Interactive e-Books for Education
YAI4Edu: an Explanatory AI to Generate Interactive e-Books for Education
 
Automatic Question Generation for Evidence-based Online Courseware Engineering
Automatic Question Generation for Evidence-based Online Courseware EngineeringAutomatic Question Generation for Evidence-based Online Courseware Engineering
Automatic Question Generation for Evidence-based Online Courseware Engineering
 
Reading Comprehension Quiz Generation using Generative Pre-trained Transformers
Reading Comprehension Quiz Generation using Generative Pre-trained TransformersReading Comprehension Quiz Generation using Generative Pre-trained Transformers
Reading Comprehension Quiz Generation using Generative Pre-trained Transformers
 
Mathematical Language Processing via Tree Embeddings
Mathematical Language Processing via Tree EmbeddingsMathematical Language Processing via Tree Embeddings
Mathematical Language Processing via Tree Embeddings
 
Contextual Definition Generation
Contextual Definition GenerationContextual Definition Generation
Contextual Definition Generation
 
Generation of Assessment Questions from Textbooks Enriched with Knowledge Models
Generation of Assessment Questions from Textbooks Enriched with Knowledge ModelsGeneration of Assessment Questions from Textbooks Enriched with Knowledge Models
Generation of Assessment Questions from Textbooks Enriched with Knowledge Models
 
Using Semantics of Textbook Highlights to Predict Student Comprehension and K...
Using Semantics of Textbook Highlights to Predict Student Comprehension and K...Using Semantics of Textbook Highlights to Predict Student Comprehension and K...
Using Semantics of Textbook Highlights to Predict Student Comprehension and K...
 
Dental TutorBot: Exploitation of Dental Textbooks for Automated Learning
Dental TutorBot: Exploitation of Dental Textbooks for Automated LearningDental TutorBot: Exploitation of Dental Textbooks for Automated Learning
Dental TutorBot: Exploitation of Dental Textbooks for Automated Learning
 
What's in a textbook
What's in a textbookWhat's in a textbook
What's in a textbook
 
Using Programmed Instruction to Help Students Engage with eTextbook Content
Using Programmed Instruction to Help Students Engage with eTextbook Content Using Programmed Instruction to Help Students Engage with eTextbook Content
Using Programmed Instruction to Help Students Engage with eTextbook Content
 
Adding Intelligence to a Textbook for Human Anatomy with a Causal Concept Map...
Adding Intelligence to a Textbook for Human Anatomy with a Causal Concept Map...Adding Intelligence to a Textbook for Human Anatomy with a Causal Concept Map...
Adding Intelligence to a Textbook for Human Anatomy with a Causal Concept Map...
 
Interlingua: Linking Textbooks Across Different Languages
Interlingua: Linking Textbooks Across Different Languages Interlingua: Linking Textbooks Across Different Languages
Interlingua: Linking Textbooks Across Different Languages
 

Recently uploaded

Your Skill Boost Masterclass: Strategies for Effective Upskilling
Your Skill Boost Masterclass: Strategies for Effective UpskillingYour Skill Boost Masterclass: Strategies for Effective Upskilling
Your Skill Boost Masterclass: Strategies for Effective Upskilling
Excellence Foundation for South Sudan
 
Leveraging Generative AI to Drive Nonprofit Innovation
Leveraging Generative AI to Drive Nonprofit InnovationLeveraging Generative AI to Drive Nonprofit Innovation
Leveraging Generative AI to Drive Nonprofit Innovation
TechSoup
 
IGCSE Biology Chapter 14- Reproduction in Plants.pdf
IGCSE Biology Chapter 14- Reproduction in Plants.pdfIGCSE Biology Chapter 14- Reproduction in Plants.pdf
IGCSE Biology Chapter 14- Reproduction in Plants.pdf
Amin Marwan
 
Beyond Degrees - Empowering the Workforce in the Context of Skills-First.pptx
Beyond Degrees - Empowering the Workforce in the Context of Skills-First.pptxBeyond Degrees - Empowering the Workforce in the Context of Skills-First.pptx
Beyond Degrees - Empowering the Workforce in the Context of Skills-First.pptx
EduSkills OECD
 
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptxC1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
mulvey2
 
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UPLAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
RAHUL
 
How to Make a Field Mandatory in Odoo 17
How to Make a Field Mandatory in Odoo 17How to Make a Field Mandatory in Odoo 17
How to Make a Field Mandatory in Odoo 17
Celine George
 
The History of Stoke Newington Street Names
The History of Stoke Newington Street NamesThe History of Stoke Newington Street Names
The History of Stoke Newington Street Names
History of Stoke Newington
 
Chapter wise All Notes of First year Basic Civil Engineering.pptx
Chapter wise All Notes of First year Basic Civil Engineering.pptxChapter wise All Notes of First year Basic Civil Engineering.pptx
Chapter wise All Notes of First year Basic Civil Engineering.pptx
Denish Jangid
 
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching AptitudeUGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
S. Raj Kumar
 
Hindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdfHindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdf
Dr. Mulla Adam Ali
 
Liberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdfLiberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdf
WaniBasim
 
Wound healing PPT
Wound healing PPTWound healing PPT
Wound healing PPT
Jyoti Chand
 
How to Create a More Engaging and Human Online Learning Experience
How to Create a More Engaging and Human Online Learning Experience How to Create a More Engaging and Human Online Learning Experience
How to Create a More Engaging and Human Online Learning Experience
Wahiba Chair Training & Consulting
 
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptxPrésentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
siemaillard
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
Nguyen Thanh Tu Collection
 
Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...
PsychoTech Services
 
Constructing Your Course Container for Effective Communication
Constructing Your Course Container for Effective CommunicationConstructing Your Course Container for Effective Communication
Constructing Your Course Container for Effective Communication
Chevonnese Chevers Whyte, MBA, B.Sc.
 
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdfANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
Priyankaranawat4
 
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem studentsRHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
Himanshu Rai
 

Recently uploaded (20)

Your Skill Boost Masterclass: Strategies for Effective Upskilling
Your Skill Boost Masterclass: Strategies for Effective UpskillingYour Skill Boost Masterclass: Strategies for Effective Upskilling
Your Skill Boost Masterclass: Strategies for Effective Upskilling
 
Leveraging Generative AI to Drive Nonprofit Innovation
Leveraging Generative AI to Drive Nonprofit InnovationLeveraging Generative AI to Drive Nonprofit Innovation
Leveraging Generative AI to Drive Nonprofit Innovation
 
IGCSE Biology Chapter 14- Reproduction in Plants.pdf
IGCSE Biology Chapter 14- Reproduction in Plants.pdfIGCSE Biology Chapter 14- Reproduction in Plants.pdf
IGCSE Biology Chapter 14- Reproduction in Plants.pdf
 
Beyond Degrees - Empowering the Workforce in the Context of Skills-First.pptx
Beyond Degrees - Empowering the Workforce in the Context of Skills-First.pptxBeyond Degrees - Empowering the Workforce in the Context of Skills-First.pptx
Beyond Degrees - Empowering the Workforce in the Context of Skills-First.pptx
 
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptxC1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
C1 Rubenstein AP HuG xxxxxxxxxxxxxx.pptx
 
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UPLAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
 
How to Make a Field Mandatory in Odoo 17
How to Make a Field Mandatory in Odoo 17How to Make a Field Mandatory in Odoo 17
How to Make a Field Mandatory in Odoo 17
 
The History of Stoke Newington Street Names
The History of Stoke Newington Street NamesThe History of Stoke Newington Street Names
The History of Stoke Newington Street Names
 
Chapter wise All Notes of First year Basic Civil Engineering.pptx
Chapter wise All Notes of First year Basic Civil Engineering.pptxChapter wise All Notes of First year Basic Civil Engineering.pptx
Chapter wise All Notes of First year Basic Civil Engineering.pptx
 
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching AptitudeUGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
 
Hindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdfHindi varnamala | hindi alphabet PPT.pdf
Hindi varnamala | hindi alphabet PPT.pdf
 
Liberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdfLiberal Approach to the Study of Indian Politics.pdf
Liberal Approach to the Study of Indian Politics.pdf
 
Wound healing PPT
Wound healing PPTWound healing PPT
Wound healing PPT
 
How to Create a More Engaging and Human Online Learning Experience
How to Create a More Engaging and Human Online Learning Experience How to Create a More Engaging and Human Online Learning Experience
How to Create a More Engaging and Human Online Learning Experience
 
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptxPrésentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
Présentationvvvvvvvvvvvvvvvvvvvvvvvvvvvv2.pptx
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
 
Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...
 
Constructing Your Course Container for Effective Communication
Constructing Your Course Container for Effective CommunicationConstructing Your Course Container for Effective Communication
Constructing Your Course Container for Effective Communication
 
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdfANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
ANATOMY AND BIOMECHANICS OF HIP JOINT.pdf
 
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem studentsRHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
RHEOLOGY Physical pharmaceutics-II notes for B.pharm 4th sem students
 

Transforming Textbooks into Learning by Doing Environments: An Evaluation of Textbook-Based Automatic Question Generation

  • 1. c Transforming Textbooks into Learning by Doing Environments: An Evaluation of Textbook-Based Automatic Question Generation
  • 2. c The Doer Effect and the AI Solution The Doer Effect is the learning science principle that doing formative practice while reading has a six times larger effect size on learning outcomes than reading alone, a relationship that has been shown to be causal [10–12]. Formative practice is good for students, but costly to create. Automatic question generation can solve this problem.
  • 3. c The State of Research In Kurdi et al.’s 2020 systematic review of research, there are gaps identified in the current research: • Only 1 study evaluated AG questions in a classroom setting • Only 1 study generated feedback • Only 1 study identified Bloom’s level • Only 14 studies evaluated question difficulty • There is no clear “gold standard” identified
  • 5. c The Data: 786,242 total observations
  • 6. c Evaluating Questions We compare our AG questions to HA questions in the same course using a mixed-effects logistic regression model. In the same course, how do our AG questions compare to HA questions on: • Engagement • Difficulty • Persistence
  • 8. c Engagement • For all questions, the location of the question in the course mattered. • The recognition question types had higher engagement than the recall question types. • The AG recognition type is similar to HA recognition types, and the AG recall type is similar to the HA recall type. • There is no indication that students found the AG question types problematic in general and chose to answer them less frequently. To summarize the engagement analysis:
  • 10. c Difficulty To summarize the difficulty analysis: • AQ matching were often the the easiest, but similar to other HA recognition types. • AG FITB were similar to several other HA questions, which were recognition types. • HA FITB were often marginally to significantly more difficult than AG FITB. • Questions varied in difficulty across courses, showing the impact of content.
  • 11. c Persistence To summarize the persistence analysis: • AG FITB had statistically different persistence from all questions except HA MCMS. • HA FITB generally had lower persistence than AG FITB. • AG matching had similar persistence to other HA questions.
  • 12. c In Summary Students were not deterred by AG questions. Engagement was impacted by placement in the course, and recognition vs recall types. AG questions were in the difficulty range of the HA questions. Easy questions did not deter persistence, but very difficult questions did.
  • 13. c Thank You! If you have any further questions, please contact us: • Benny.Johnson@vitalsource.com • Rachel.Vancampenhout@vitalsource.com • Bill.Jerome@vitalsource.com

Editor's Notes

  1. Define these metrics