Case Study: Evaluation of a Tool for Searching inside a Collection of Multimodal e-Lectures  Marco Ronchetti Angela Fogaro...
Logical  Architecture Synchronous ( webcast ) Asynchronous  ( On line , or  download - podcast ) Asynchronous( CD - DVD )
Acquisition setup
Cognitive interface <ul><li>The main focus is on  the projected slide ,  </li></ul><ul><li>a  clear voice  is very importa...
LODE: the user interface
New Flash version
Lode - portability Supporto della didattica tradizionale Mac – Win - Linux Player MP4 (no slides yet) Presently attempting...
Student feedback <ul><li>Enthusiastic .  </li></ul><ul><li>We were “forced” to extend the experiment to a second course! <...
Advantages from student’s perspective <ul><li>Better time management: </li></ul><ul><ul><li>ability to  recover lectures l...
Other advantages <ul><li>Miscellaneous </li></ul><ul><ul><li>perception by the student of a better service provided by the...
Can we do more? <ul><li>Scenario: </li></ul><ul><li>Thousands of recorded lectures. </li></ul><ul><li>Can we dig in them t...
Needle - search results
Needle - architecture ASR Multimodal search!
Questions <ul><li>Is it useful? </li></ul><ul><li>Is it effective? </li></ul>
Scenario <ul><li>100% accurate transcription (manually corrected) </li></ul><ul><li>Goal: extract some info from a 2 hour ...
Experiment <ul><li>68 students, after 20 hours of a 50 hour Java course. </li></ul><ul><li>Group A: LODE </li></ul><ul><li...
Experiment <ul><li>Pre-Test: 5 questions, 5 minutes  </li></ul><ul><li>LODE group: 70 minutes for the lecture, plus </li><...
Pre-test results
Test results
Analysis <ul><li>In the subset of questions, which require students to have a  complete understanding  of a part of the le...
Further investigations <ul><li>Multivariate regression: </li></ul><ul><li>no significant results </li></ul><ul><li>Cluster...
Conclusion <ul><li>NEEDLE tool is more geared towards information finding than towards actual learning.  </li></ul><ul><li...
Upcoming SlideShare
Loading in …5
×

Case study: evaluation of a tool for searching inside a collection of multimodal e-lectures

1,022 views

Published on

Published in: Economy & Finance, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,022
On SlideShare
0
From Embeds
0
Number of Embeds
37
Actions
Shares
0
Downloads
25
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Case study: evaluation of a tool for searching inside a collection of multimodal e-lectures

  1. 1. Case Study: Evaluation of a Tool for Searching inside a Collection of Multimodal e-Lectures Marco Ronchetti Angela Fogarolli Dipartim. Informatica e Telecomunicazioni Universit à di Trento Italy
  2. 2. Logical Architecture Synchronous ( webcast ) Asynchronous ( On line , or download - podcast ) Asynchronous( CD - DVD )
  3. 3. Acquisition setup
  4. 4. Cognitive interface <ul><li>The main focus is on the projected slide , </li></ul><ul><li>a clear voice is very important, </li></ul><ul><li>the video carries additional information like gestures , and can show the environment just enough </li></ul><ul><li>Navigation is possible </li></ul><ul><li>It must be also possible to use the blackboard! </li></ul>
  5. 5. LODE: the user interface
  6. 6. New Flash version
  7. 7. Lode - portability Supporto della didattica tradizionale Mac – Win - Linux Player MP4 (no slides yet) Presently attempting a port to mobile phones
  8. 8. Student feedback <ul><li>Enthusiastic . </li></ul><ul><li>We were “forced” to extend the experiment to a second course! </li></ul>75% anticipated using the system often or very often 5% 63% total 2 nd exam (23 stud.) 1 st exam (45 stud.) students who… 4% 6% … followed the course completely off-line 87% 51% … used the system to review at least one lecture
  9. 9. Advantages from student’s perspective <ul><li>Better time management: </li></ul><ul><ul><li>ability to recover lectures lost due to forced absence (illness, work or other time-frame incompatibility); </li></ul></ul><ul><ul><li>ability to better organize their time , deciding not to be present at some lecture (elective absence); </li></ul></ul><ul><li>Better understanding </li></ul><ul><ul><li>review some critical point (cases of poor understanding of a section due to concentration drop, excessive speed in an explanation or intrinsic difficulty); </li></ul></ul><ul><ul><li>ability to check the correctness of notes taken during a lecture; </li></ul></ul>
  10. 10. Other advantages <ul><li>Miscellaneous </li></ul><ul><ul><li>perception by the student of a better service provided by the university ; </li></ul></ul><ul><ul><li>support foreign (italian) students who might have difficulties with the Italian (English) language ; </li></ul></ul><ul><ul><li>enrichment of the e-learning portfolio; </li></ul></ul><ul><ul><li>possibility for the teacher to view himself; </li></ul></ul><ul><ul><li>possibility to reuse lectures (across time, or in different contexts!) </li></ul></ul>
  11. 11. Can we do more? <ul><li>Scenario: </li></ul><ul><li>Thousands of recorded lectures. </li></ul><ul><li>Can we dig in them to extract material for (less formal) learning? </li></ul><ul><li>Cfr. approach by Mike Wald </li></ul><ul><ul><li>(subtitling for accessibility) </li></ul></ul>
  12. 12. Needle - search results
  13. 13. Needle - architecture ASR Multimodal search!
  14. 14. Questions <ul><li>Is it useful? </li></ul><ul><li>Is it effective? </li></ul>
  15. 15. Scenario <ul><li>100% accurate transcription (manually corrected) </li></ul><ul><li>Goal: extract some info from a 2 hour lecture in half of the time… </li></ul><ul><li>Assess whether the tool could reduce the learning time and help students in their learning task: we do not aim at claiming or proving that a course delivery mode is better than the other. </li></ul><ul><li>Comparing delivery modes (e.g. distance learning with traditional classroom-based instruction) (Sener 2005) for the purpose of establishing the superiority of one delivery mode over another is specious, irrelevant and counterproductive because there is not an uniformity of practice. </li></ul>
  16. 16. Experiment <ul><li>68 students, after 20 hours of a 50 hour Java course. </li></ul><ul><li>Group A: LODE </li></ul><ul><li>Group B: NEEDLE </li></ul>
  17. 17. Experiment <ul><li>Pre-Test: 5 questions, 5 minutes </li></ul><ul><li>LODE group: 70 minutes for the lecture, plus </li></ul><ul><li>24 minutes for the post-test (8 questions): total 94 minutes to accomplish the task </li></ul><ul><li>NEEDLE group: 48 minutes total for responding to the same 8 questions (50% of the time) </li></ul>
  18. 18. Pre-test results
  19. 19. Test results
  20. 20. Analysis <ul><li>In the subset of questions, which require students to have a complete understanding of a part of the lecture both groups performed in the same way. </li></ul><ul><li>In answering those questions that also required some reasoning about concepts , group LODE – who had listened to the entire lecture – performed better. </li></ul><ul><li>In the questions where the students where asked to provide the right definition for a concept group NEEDLE performed better. </li></ul>
  21. 21. Further investigations <ul><li>Multivariate regression: </li></ul><ul><li>no significant results </li></ul><ul><li>Clustering techniques </li></ul><ul><li>Indication that some feature seem to be present - investigation is still on its way </li></ul>
  22. 22. Conclusion <ul><li>NEEDLE tool is more geared towards information finding than towards actual learning. </li></ul><ul><li>A similar pattern could be reasonably expected in a similar experiment performed by reading some assigned material versus searching quick answer with Google. </li></ul><ul><li>However, our finding seems to show that Needle works very well as a tool for extracting information from a video source. </li></ul>

×