Greek Evaluation


Published on

Published in: Economy & Finance, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Greek Evaluation

  1. 1. User-Centered Evaluation of an On-Line Modern Greek Language Course <ul><ul><li>Panayiotis Zaphiris </li></ul></ul><ul><ul><li>The Centre for HCI Design </li></ul></ul><ul><ul><li>City University </li></ul></ul><ul><ul><li>Giorgos Zacharia </li></ul></ul><ul><ul><li>Center for Biological and Computational Learning </li></ul></ul><ul><ul><li>Massachusetts Institute of Technology </li></ul></ul>
  2. 2. Kypros-Net Inc. ( http:// / ) <ul><li>Formation </li></ul><ul><ul><li>By a group of volunteers in 1996. </li></ul></ul><ul><ul><li>Composed mainly of Computer Science, Information Technology and Human-Computer Interaction students. </li></ul></ul><ul><li>Purpose: </li></ul><ul><ul><li>“ to utilize the Internet and particularly the World Wide Web (WWW) system for the promotion of the world of Cyprus“ </li></ul></ul>
  3. 3. Learn Greek Online ( ) <ul><li>Course Description </li></ul><ul><ul><li>Independent study course to learn Greek Online </li></ul></ul><ul><ul><ul><li>105 audio lessons, transcriptions, vocabulary, spell checker, dictionary </li></ul></ul></ul><ul><ul><li>Objectives </li></ul></ul><ul><ul><ul><li>Assist people in learning Greek language </li></ul></ul></ul><ul><ul><ul><li>Develop site content using Participatory Design </li></ul></ul></ul>
  4. 4. Learn Greek Online Home Page
  5. 5. Participatory Design Methodology <ul><li>(a) Building bridges with the intended users </li></ul><ul><li>(b) Map user needs and suggestions to the system </li></ul><ul><li>(c) Develop a prototype </li></ul><ul><li>(d) Integrate feedback and continue the iteration. </li></ul>
  6. 6. Design Methodology
  7. 7. STEP 1: Building bridges with the intended users <ul><li>Open lines of communication between intended users and the development team. </li></ul><ul><ul><li>Initiate multidisciplinary development team </li></ul></ul><ul><ul><li>Identify key groups of end users </li></ul></ul><ul><ul><li>Create new methods of communication with users. </li></ul></ul>
  8. 8. Intended Users <ul><li>Bridges with users build through years of work at providing information about Cyprus through the web pages of Kypros-Net, Inc. who primarily attracts the same user population as the Greek language online course. </li></ul><ul><ul><li>People of the Greek Diaspora </li></ul></ul><ul><ul><li>Travelers to Cyprus and Greece and other Greek speaking areas </li></ul></ul><ul><ul><li>People who are generally interested in the Greek culture and language or languages in general. </li></ul></ul>
  9. 9. STEP 2: Map user needs and suggestions to the system <ul><li>Conceptual design model: </li></ul><ul><ul><li>“ to design an online effective Greek language course that can build and sustain an online learning community of students”. </li></ul></ul><ul><li>Tried to match user needs with our conceptual design model: </li></ul><ul><ul><li>Easy to follow </li></ul></ul><ul><ul><li>Both Elementary and Advanced </li></ul></ul><ul><ul><li>Could attend at their own pace </li></ul></ul>
  10. 10. STEP 3: Develop a prototype <ul><li>The project consists of 105 audio files, which were originally recorded as Radio lessons in Modern Greek for English speakers back in 1960's. </li></ul><ul><li>The lessons were retrieved from the archives of the Cyprus Broadcasting Corporation, digitized in Real Audio 5.0 format and published on-line. </li></ul>
  11. 11. Prototype (Continued) <ul><li>Although, an optional textbook accompanied the original Radio lessons, the lessons were designed as a complete standalone course. </li></ul><ul><li>We used several tools to assist the students of the lessons, including an online English-Greek-English dictionary, a Greek spell checker and a web-based discussion board for the students of the lessons. </li></ul>
  12. 12. STEP 4: Integrate feedback and continue the iteration. <ul><li>Feedback from our users and suggestions are continuously incorporated into our design through a series of additions and corrections. </li></ul><ul><li>An important element in the participatory design methodology is the direct involvement of the user in all stages of the design process (including the evaluation). </li></ul>
  13. 13. Student Involvement <ul><li>Users started exchanging through email written notes taken by the advanced users. </li></ul><ul><ul><li>We provided the users with the capability to post their notes on the project's site. </li></ul></ul><ul><li>Two months later, the students of the Audio lessons have managed to transcribe 81 out of the 105 lessons, correct them through a peer review process among themselves and post them on the project's website. </li></ul><ul><li>Six months after that, the students had transcribed and peer reviewed all 105 lessons. </li></ul>
  14. 14. Evaluation Stages <ul><li>1. Analysis of server logs </li></ul><ul><li>2. Analysis of discussion board postings </li></ul><ul><li>3. Questionnaires </li></ul><ul><li>4. Usability Testing </li></ul>
  15. 15. Key Evaluation Questions <ul><li>How useful is the course? </li></ul><ul><li>How functional is the course? </li></ul><ul><li>How well does it meet on-line course standards? </li></ul><ul><li>What can be done to improve the course? </li></ul>
  16. 16. Analysis of server logs
  17. 17. Questionnaires
  18. 18. Questionnaire 1 - Results
  19. 19. Questionnaire 1: User Satisfaction
  20. 20. Questionnaire 1: Findings <ul><li>User Expectations </li></ul><ul><ul><li>High satisfaction </li></ul></ul><ul><ul><li>Positive comments </li></ul></ul><ul><ul><ul><li>Easy to understand </li></ul></ul></ul><ul><ul><ul><li>Fun </li></ul></ul></ul><ul><ul><ul><li>It helped refresh my Greek </li></ul></ul></ul><ul><ul><ul><li>Finally after surfing for a long time I found that you could learn Greek from the internet </li></ul></ul></ul>
  21. 21. Questionnaire 2: (Demographics) <ul><li>Total of 16,848 registered students (6,098 completed the demographics questionnaire) </li></ul><ul><ul><li>49% Male, 51% Female </li></ul></ul><ul><ul><li>65% found the page through a link from an other page </li></ul></ul><ul><ul><li>Age 15-30 (44%), 31-50 (35%), 51- (10%) </li></ul></ul><ul><ul><li>Internet usage: 1-3 years (34%), 4-6 years (37%), 7+ (16%) </li></ul></ul><ul><ul><li>Primary language: 72% English </li></ul></ul>
  22. 22. Questionnaire 2: (Demographics) <ul><ul><li>Education: High School (16%), Some College (20%), College Graduate (22%), Master (10%), PhD (3%) </li></ul></ul><ul><ul><li>Location: Europe (38%), USA (43%) </li></ul></ul><ul><ul><li>Computer Platform: Windows 98 (54%), Windows 2000 (20%), MAC (2%) </li></ul></ul>
  23. 23. Analysis of Discussion Board (371 postings) <ul><li>Category/Subject # % Total % </li></ul><ul><li>Technical </li></ul><ul><li>Problems with needing/installing keyboard fonts 56 15.1% </li></ul><ul><li>Problems with/wanting to download lessons 49 13.2% </li></ul><ul><li>Problems with installing/using RealPlayer 40 10.8% 39.1% </li></ul><ul><li>Content </li></ul><ul><li>Postings of on-line notes 73 19.7% </li></ul><ul><li>Questions about availability of text, CDs or </li></ul><ul><li>including links to web sites 37 10.0% 29.7% </li></ul><ul><li>Comprehension </li></ul><ul><li>Questions about vocabulary 28 7.5% </li></ul><ul><li>Questions about grammar/spelling 17 4.6% </li></ul><ul><li>Questions about dialect 2 5.4% 17.5% </li></ul><ul><li>Miscellaneous 69 18.6% 18.6% </li></ul>
  24. 24. Formal Usability Test <ul><li>How functional is the course? </li></ul><ul><ul><li>Do the operations work? </li></ul></ul><ul><ul><li>Is the architecture successful? </li></ul></ul><ul><li>Findings </li></ul><ul><ul><li>Site is subject to typical Internet problems </li></ul></ul><ul><ul><li>Learn Greek! architecture can be improved </li></ul></ul>
  25. 25. Formal Usability Test - Findings <ul><li>Operations </li></ul><ul><ul><li>The site generally works </li></ul></ul><ul><ul><li>(Translations, lessons, etc. function) </li></ul></ul><ul><ul><li>Web operations less dependable </li></ul></ul><ul><ul><li>(Server downtime, Slow lesson downloads, Intermittent transmission) </li></ul></ul><ul><li>Architecture </li></ul><ul><ul><li>Labeling </li></ul></ul><ul><ul><li>(Incorrect, Confusing, Out-of-order) </li></ul></ul><ul><ul><li>Content </li></ul></ul><ul><ul><li>(No directions, No descriptions, No “Help” or FAQ, Multiple translations) </li></ul></ul>
  26. 26. Formal Usability Test - Site Architecture-Labeling
  27. 27. Formal Usability Test - Site Architecture-Formats
  28. 28. Discussion <ul><li>Students created an open online community whose collaboration has boosted the learning experience of the whole community. </li></ul><ul><li>The web-based discussion board has proven to be the most constructive tool for the students learning experience and the main source of feedback for the maintainers of the project. </li></ul>
  29. 29. Discussion <ul><li>The experiences shared on the discussion board include tricks and tips on how to record the audio files, installation of Greek fonts, learning methodologies and questions about the Greek language itself that arise from the lessons. </li></ul><ul><li>Also used for announcing the availability of their personal notes. </li></ul>
  30. 30. User Leaders <ul><li>The advanced users (some of them retired teachers of foreign languages) have taken a lead role in the vast majority of the threads on the discussion board </li></ul><ul><li>Answering most of the questions and encouraging the beginners to study the lessons further. </li></ul><ul><li>Communication interface between the maintainers of the project and the community's requests. Maintain the online notes pages </li></ul>
  31. 31. Discussion <ul><li>The accesses the audio lessons, the language tools, and the total access of the message board and the notes pages, all kept increasing exponentially. </li></ul><ul><li>Once we allowed our users to publish their own notes, there was a dramatic shift of traffic from the message board to the notes pages. Probably </li></ul><ul><ul><li>Because it was no longer necessary to visit the message board to find out where other users had posted their notes, since all were aggregated in a central location. </li></ul></ul>