Your SlideShare is downloading. ×
0
A Usability Analysis System for            e-Learning Authoring Tools                                                     ...
PresentationContents   Introduction   Story-line authoring   Ratings and outcomes   Time-lapse visualization   Future work...
Introduction: authoring & tool evaluationContents                              System adoption requires content   Introdu...
Story-line authoring: adventure game authoringContents   Introduction   Story-line authoring   Ratings and outcomes   Time...
Story-line authoring: WeevContents   Introduction   Story-line authoring   Ratings and outcomes   Time-lapse visualization...
Story-line authoring: initial questionContents   Introduction   Story-line authoring   Ratings and outcomes   Time-lapse v...
Ratings and outcomes: a-b testing with ratingsContents                              A-B testing: very popular in web usab...
Ratings and outcomes: evaluating ratingsContents                              N users are requested to rate M users each ...
Ratings and outcomes: statistical treatmentContents                                                      1x b>a   Introduc...
Ratings and outcomes: experimentContents                              Experiment with   Introduction   Story-line authori...
Ratings and outcomes: simulationsContents   Introduction   Story-line authoring   Ratings and outcomes   Time-lapse visual...
Ratings and outcomes: more simulations                                                         Increase ratings-per-userCo...
Ratings and outcomes: increasing ratingsContents   Introduction   Story-line authoring   Ratings and outcomes   Time-lapse...
Ratings and outcomes: increasing usersContents   Introduction   Story-line authoring   Ratings and outcomes   Time-lapse v...
Time-lapse visualization: ideaContents                              UI instrumentation sends data to server   Introductio...
Time-lapse visualization: interface                              user / recordContents                      selection     ...
Time-lapse visualization: insights                                User is stuckContents   Introduction   Story-line author...
Time-lapse visualization: tasks and directionsContents                               Tasks for UI evaluation   Introducti...
Future work: next experimentContents                               Server   Introduction   Story-line authoring          ...
Concluding remarksContents                              A/B usability testing methodology for   Introduction   Story-line...
Questions or comments?Contents   Introduction   Story-line authoring   Ratings and outcomes   Time-lapse visualization   F...
Upcoming SlideShare
Loading in...5
×

2011 10 07 (uam) emadrid mfreire ucm sistema analisis usabilidad herramientas autoria contenidos e learning

698

Published on

2011 10 07
(uam)
emadrid
mfreire
ucm
sistema analisis usabilidad herramientas autoria contenidos e learning

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
698
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "2011 10 07 (uam) emadrid mfreire ucm sistema analisis usabilidad herramientas autoria contenidos e learning"

  1. 1. A Usability Analysis System for e-Learning Authoring Tools Manuel Freire MoráneMadrid seminar on Adaptive Systems - 7 October 2011 manuel.freire@fdi.ucm.es
  2. 2. PresentationContents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks 1/20
  3. 3. Introduction: authoring & tool evaluationContents System adoption requires content Introduction Story-line authoring  authoring Ratings and outcomes  reuse Time-lapse visualization Future work Better authoring requires creativity support Concluding remarks  imagine, create, play, share, reflect creative thinking spiral - Resnick, 2008  low threshold, high ceiling, wide walls  make it as simple as possible - and maybe simpler Shneiderman et al, 2006  evaluate your tools 2/20
  4. 4. Story-line authoring: adventure game authoringContents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks 3/20
  5. 5. Story-line authoring: WeevContents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks 4/20
  6. 6. Story-line authoring: initial questionContents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks 5/20
  7. 7. Ratings and outcomes: a-b testing with ratingsContents A-B testing: very popular in web usability Introduction Story-line authoring  Split participants randomly into two groups Ratings and outcomes  Each group uses an interface variant (A and B) Time-lapse visualization  Outcomes of each group are compared Future work Concluding remarks Outcome of a creative task?  Objective measures: correctness, completeness  Subjective measures: questionnaires (author satisfaction), ratings (comparative quality) 6/20
  8. 8. Ratings and outcomes: evaluating ratingsContents N users are requested to rate M users each Introduction Story-line authoring  All users rated same number of times Ratings and outcomes  No user rated twice by same rater Time-lapse visualization  No user rates self Future work Concluding remarks Unknown individual rating distribution. How to determine if results are significant?  H0: count(A rated better than B) is a fair coin toss  one-sided binomial distribution 7/20
  9. 9. Ratings and outcomes: statistical treatmentContents 1x b>a Introduction Story-line authoring Ratings and outcomes Time-lapse visualization 5/10 4/10 7/10 8/10 6/10 a b a a b Future work Concluding remarks 4 x a>b 8/20
  10. 10. Ratings and outcomes: experimentContents Experiment with Introduction Story-line authoring  20 users (first-time Weev users) Ratings and outcomes  5-minute tutorial on tool use (applicable to A&B) Time-lapse visualization  "Little Red Riding-Hood" script and resources Future work Concluding remarks  1 hours time; after editing, rating screen Outcome  70 total A vs B comparisons; in 40, B > A  p-value: 0.141: does not reject null hypothesis 9/20
  11. 11. Ratings and outcomes: simulationsContents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks 10/20
  12. 12. Ratings and outcomes: more simulations Increase ratings-per-userContents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks Increase number of users 11/20
  13. 13. Ratings and outcomes: increasing ratingsContents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks 12/20
  14. 14. Ratings and outcomes: increasing usersContents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks 13/20
  15. 15. Time-lapse visualization: ideaContents UI instrumentation sends data to server Introduction Story-line authoring  server timestamps records for each user Ratings and outcomes  UI screen captures Time-lapse visualization  Action logs Future work Concluding remarks  last screen capture sample used for rating  ratings-file is just one more record type How do you begin to analyze this data?  Build videos from screen-captures  Visualization tool to scan time-lapse data 14/20
  16. 16. Time-lapse visualization: interface user / recordContents selection time-lapse view of Introduction records for first selected user Story-line authoring time-lapse view for Ratings and outcomes next selected user Time-lapse visualization Future work Concluding remarks detail / difference view 15/20
  17. 17. Time-lapse visualization: insights User is stuckContents Introduction Story-line authoring Ratings and outcomes Creative flow Time-lapse visualization Future work Concluding remarks Rectangular placement 16/20
  18. 18. Time-lapse visualization: tasks and directionsContents  Tasks for UI evaluation Introduction Story-line authoring What did this user do? Ratings and outcomes - glance at timeline Time-lapse visualization  What happened between here and there? Future work - use difference function Concluding remarks  Still missing  Variable-length time lapses ("time zoom")  Graphical display of textual log data  Filter & Sort 17/20
  19. 19. Future work: next experimentContents  Server Introduction Story-line authoring  Finer-grained data collection Ratings and outcomes  Better rating distributions, more ratings Time-lapse visualization  Support for experimenter note-taking Future work Concluding remarks  Analysis tool improvements  Test new automatic layout assistant  Do people use it?  Do people rate "automatically-improved" storylines as better than-before? 18/20
  20. 20. Concluding remarksContents A/B usability testing methodology for Introduction Story-line authoring creative/aesthetic tasks Ratings and outcomes Reusable server Time-lapse visualization Future work  Collects images, logs, rating results Concluding remarks  Configurable rating, allows real-time monitoring Reusable client  Specific handling for each record-type (ie.: logs) 19/20
  21. 21. Questions or comments?Contents Introduction Story-line authoring Ratings and outcomes Time-lapse visualization Future work Concluding remarks 20/20
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×