Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Putting the world to work for            ITS: Open community authoring of  targeted worked example          problems   Ale...
Current situation in tutoring2    systems    •   Development is very laborious        •   (e.g. estimates of 200-300 hrs f...
Wikipedia not the right model3
Towards a collaborative4    community       • Volunteers                              • Others rate and         submit new...
Broad research questions5       If you make it, will they come?       Can the wheat be separated from the chaff?       ...
Overview of the study6       Web site where people contribute worked        example problems       In registering, indic...
Task7    •   Artifact: Worked example problem        –   Leads to better and more efficient learning when            added...
Zack and Slater want to build a bike jump. They have           two parts of the ramp constructed but they need toProblem  ...
Authoring tool9
Open authoring hypotheses10        H1: Identifying the good from the bad         contributions is easy. We expect that al...
Student profiles11        Goal of realism        Varied on social and cognitive attributes        16 profiles         ...
Profile hypotheses12     Profiles in experimental condition versus generic control        condition        H3: Student pr...
Participants and contributions13     •   Participation URL posted on web sites         (educational and otherwise) offerin...
Machine filtered14         Some have just a         worthless drawing.         Or nothing at all.
Quality ratings15     Human experts rated the machine vetted submissions      Numerical      Rating      value          ca...
Quality rating examples16        Excellent statement with poor solution (1124)        Worthy statement with excellent so...
17   Open authoring
Quality of pool18
Quality by contributor expertise19     Statement quality                   Solution quality      Teacher   Sign.   Mean   ...
20   Student profiles
Tailoring to social attributes21                                                    With profiles                         ...
For profile with a home     outside town22
For profile who lives in tall     apartment building23
Tailoring to cognitive attributes24     Verbal skill in profile                                              General math ...
Shakespeare for profile     in “top of English class”25
Effects of profiles26     On effort                  On quality        Problem statements        No main effect of      ...
27   Conclusions
Recap of Hypotheses28     Hypothesis                        Short        Long Answer                                      ...
Current and future work29        • Volunteers                              • Others rate and          submit new          ...
Current and future work30        • Volunteers                              • Others rate and          submit new          ...
Current and future work31        • Volunteers                              • Others rate and          submit new          ...
Acknowledgements32        Thanks to ASSISTment project, Ken         Koedinger and Sara Kiesler for data and         feedb...
Gratis participants33        Still 93 submissions from 92 participants        Of these 38 submissions from 21 participan...
Upcoming SlideShare
Loading in …5
×

Putting the world to work for ITS

392 views

Published on

Presentation at Intelligent Tutoring Systems conference in 2008 on open community authoring of targeted worked example problems.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Putting the world to work for ITS

  1. 1. Putting the world to work for ITS: Open community authoring of targeted worked example problems Aleahmad, Aleven and Kraut6/27/2008 ITS 2008
  2. 2. Current situation in tutoring2 systems • Development is very laborious • (e.g. estimates of 200-300 hrs for 1 hr instruction) • Small groups with much effort per person • Distribute the development • Open source • Open content • How to make a “Wikipedia” for ITS?
  3. 3. Wikipedia not the right model3
  4. 4. Towards a collaborative4 community • Volunteers • Others rate and submit new critique material Generate Evaluate Improve Use • Link resources • Others make the into tutoring contribution systems or better create new ones
  5. 5. Broad research questions5  If you make it, will they come?  Can the wheat be separated from the chaff?  How to structure and support authoring?  For quality  For diversity to engage students – Contextualization, personalization, and provision of choices can improve student motivation and engagement in learning (Cordova and Lepper, 1996 ) – Personalization improves performance gains and even at start (Anand and Ross, 1987; Ku and Sullivan 2002; López and Sullivan 1992)
  6. 6. Overview of the study6  Web site where people contribute worked example problems  In registering, indicated their professional status  Tested a mechanism to increase quality and diversity  Asked some authors to target to a specific person  Increase their effort?  Increase diversity/adaptivity of corpus?
  7. 7. Task7 • Artifact: Worked example problem – Leads to better and more efficient learning when added to interactive tutoring (McLaren et al., 2006; Schwonke et al., 2007) – Instruct and foster self-explanation (Renkl and Atkinson, 2002) – Customizability – both to the student and the interaction • Domain: Pythagorean Theorem – Most difficult skill on the Massachusetts Comprehensive Assessment System curriculum standards (ASSISTment data)
  8. 8. Zack and Slater want to build a bike jump. They have two parts of the ramp constructed but they need toProblem know the length of the final piece of the jump. They have two parts of the ramp built, one is 3 ft long andStateme the other is 4 ft long and they are constructed as shown in the diagram. What is the length of the missing section that Zack and Slater still need tont construct? + Work ExplanationSolution The unknown is the hypotenus which is represented by c in the 3^2 + 4^2steps equation. Therefore I input both a and b into the equation first. Following the equation I square both of these numbers. = 9 + 16 = These two numbers are added 25 together first because of theWhole parenthesis. To complete the equation I takeworked Square the square root of 25 which is five. This problem also demonstrates root of 25example is 5 and this is the the common Pythagoras triangle. solution.8
  9. 9. Authoring tool9
  10. 10. Open authoring hypotheses10  H1: Identifying the good from the bad contributions is easy. We expect that all contributions are good, easily fixed, or easily filtered.  H2: Math teachers submit the best contributions.
  11. 11. Student profiles11  Goal of realism  Varied on social and cognitive attributes  16 profiles  4 Hobbies x 4 Homes  4 realistic skill profiles distributed  2 genders distributed
  12. 12. Profile hypotheses12 Profiles in experimental condition versus generic control condition  H3: Student profiles lead to tailored contributions.  H4: Student profiles increase the effort of authors.  H5: Student profiles lead to higher quality
  13. 13. Participants and contributions13 • Participation URL posted on web sites (educational and otherwise) offering $4-12 • 1427 people registered, of which 570 used the tool to submit 1130 contributions • After machine filtering, 281 participants were left having submitted 551 contributions Participation Math teachers Other teachers Amateurs Registered 131 170 1126 Contributed also 70 72 428 Passed vetting 26 35 220 also
  14. 14. Machine filtered14 Some have just a worthless drawing. Or nothing at all.
  15. 15. Quality ratings15 Human experts rated the machine vetted submissions Numerical Rating value category Definition No use in teaching and it would be easier to 0 Useless write a new one than improve this one. Has some faults, but they are obvious and 1 Easy fix can be fixed easily, in under 5 minutes. Worthy of being given to a student who matches on the difficulty and subject matter. 2 Worthy Assume that the system knows whats in the problem and what is appropriate for each student, based on their skills and interests. Excellent example to provide to some student. Again, assume that the system 3 Excellent knows whats in the problem and what is appropriate for each student, based on their
  16. 16. Quality rating examples16  Excellent statement with poor solution (1124)  Worthy statement with excellent solution (337)
  17. 17. 17 Open authoring
  18. 18. Quality of pool18
  19. 19. Quality by contributor expertise19 Statement quality Solution quality Teacher Sign. Mean Std Teacher Sign. Mean Std status diffs quality Err status diffs quality Err Math A 1.80 0.12 Math A B 0.70 0.10 teacher teacher Other B 1.54 0.09 Other B 0.53 0.08 teacher teacher Not B 1.48 0.09 Not B 0.76 0.03 teacher teacher
  20. 20. 20 Student profiles
  21. 21. Tailoring to social attributes21 With profiles With With profiles not F-test F-test Attribute GENERIC mentioning mentioning (G-M) (N-M) (G) attribute (M) attribute (N) Female pronoun 5% 4% 16% 9.68* 12.82** Male pronoun 19% 14% 19% 0.004 1.19 Sports word 9% 9% 24% 18.01** 11.89** TV word 4% 4% 10% 8.36* 2.63† Music word 2% 2% 9% 6.92* 8.93** Home word 14% n/a 20% 3.60* n/a Probabilities of authoring matching an attribute †p<.10 *p<.05 **p<.001
  22. 22. For profile with a home outside town22
  23. 23. For profile who lives in tall apartment building23
  24. 24. Tailoring to cognitive attributes24 Verbal skill in profile General math skill in profile Verbal Sign Mean Std Math Sign Probability Std Err skill . reading Err skill . of using 3- shown diffs level of shown diffs 4-5 contribution triangle High A 3.78 0.24 High A 16% 0.05 Medium A B 3.56 0.32 Medium A B 26% 0.05 Low B 2.93 0.33 Low B 27% 0.04 GENERI B 3.20 0.16 GENER A B 21% 0.03 C IC Correspondence of verbal and math skill levels with the authoring interface
  25. 25. Shakespeare for profile in “top of English class”25
  26. 26. Effects of profiles26 On effort On quality  Problem statements  No main effect of in profile condition profiles on quality were 25% longer  No interaction with  No significant teacher status either difference in time spent (median 5 each minutes on statement and solution)
  27. 27. 27 Conclusions
  28. 28. Recap of Hypotheses28 Hypothesis Short Long Answer Answer 1 Quality control is easy Yes Filtering trivial; rating by experts take less than a minute 2 Math teachers contribute Partly Amateurs and non-math teachers the best worked examples wrote okay problem statements and amateurs wrote better solutions 3 Profiles lead to tailoring Yes Every aspect of profiles was tailored to 4 Profiles increase effort Inconclusiv A quarter longer problem e statement, but no difference in time 5 Profiles lead to higher No No difference in machine filtering quality contributions or human rated quality
  29. 29. Current and future work29 • Volunteers • Others rate and submit new critique material Generate Evaluate Improve Use • Link resources • Others make the into tutoring contribution systems or better create new ones
  30. 30. Current and future work30 • Volunteers • Others rate and submit new critique material Generate Evaluate Improve Use • Link resources • Others make the into tutoring contribution systems or better create new ones
  31. 31. Current and future work31 • Volunteers • Others rate and submit new critique material Generate Evaluate Improve Use • Link resources • Others make the into tutoring contribution systems or better create new ones
  32. 32. Acknowledgements32  Thanks to ASSISTment project, Ken Koedinger and Sara Kiesler for data and feedback  Work supported by IES and NSF  It’s going to take a lot of connected work to build a scalable shared ITS for the world  Let’s talk more about how  http://OpenEducationResearch.org
  33. 33. Gratis participants33  Still 93 submissions from 92 participants  Of these 38 submissions from 21 participants pass machine vetting  41% pass rate of machine vetting compared to 49% rate in experiment  Not significantly different by Fishers Exact Test (p=0.16)

×