Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ar3 X0 Final Research Paper G Compris Usability Research Report


Published on

  • Be the first to comment

  • Be the first to like this

Ar3 X0 Final Research Paper G Compris Usability Research Report

  1. 1. GCompris Usability Testing 1 GCompris Usability Testing Research Report Shakarah McCrae - 06004107 AR3X0 – Research Paper May 15, 2009 Supervisor: David Bain Course Moderator: Dr. Ramsay Table of Contents
  2. 2. GCompris Usability Testing 2 Executive Summary………………………………………………………………… ...2 Introduction…………………………………………………………………………… 3 Research Question…………………………………………………………......3 Null Hypothesis………………………………………………………………..3 Background…………………………………………………………………….3 Worth of Study………………………………………………………………....6 Objectives………………………………………………………………………6 Literature Review………………………………………………………………………8 Methodology…………………………………………………………………………...11 Procedure………………………………………………………………………12 Participant Summary…………………………………………………………..13 Gcompris Set-up……………………………………………………………….14 Data Presentation………………………………………………………………………16 Analysis………………………………………………………………………………..17 Conclusions and Recommendations ………………………………………………….23 Limitations…………………………………………………………………… References……………………………………………………………………………..25 Appendices………………………………………………………………………….....27 Executive Summary
  3. 3. GCompris Usability Testing 3 This study is in line with the requirements of the University of the West Indies where a research project has to be done. This topic was chosen in accordance with my field of study and to discover how influential or effective this medium is in the classroom. Gcompris was chosen because it is free and can be used as a tool for teaching basic math to students. The research is aimed at showing how user friendly the software is and how the students will react to using it. Usability testing is used to measure the appropriateness of software for its user. It is aimed at assessing the effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in particular environments or contexts of use. The chosen open source software for this purpose is GCompris, which is an educational software suite comprising of numerous activities for children, aged 2 to 10. Some of the activities are game orientated, but nonetheless educational. More specifically, I will be looking at the extent to which G-Compris, as a tool for teaching basic mathematics skills can improve or aid learning in mathematics for grade one students at August Town Primary School. Introduction Research Question:
  4. 4. GCompris Usability Testing 4 How effective is Gcompris- an open source software- for teaching basic mathematics skills to grade one students at August Town Primary school? Null Hypotheses: Gcompris as an educational tool is not beneficial in improving the performance of grade one (1) students in school. Background The disappointing performance of Jamaican students in the mathematics subject area continues to be an increasing concern to the wider Jamaican society. According to Francis (2006): “New statistics from the CXC show that of the 35,428 students who sat the 2006 mathematics exam, only 32 per cent managed to achieve grade one to three. This represents a decrease over last year's 36 per cent. This figure is also three per cent below the regional average” (para. 2). In fact, Wesley Barrett Pro-Registrar for the Caribbean Examination Council states quot;These results are disappointing and it is still a very worrying problem,quot; ” (Francis, 2006, para. 2). Poor mathematics performance is not only at the Caribbean Secondary Education Level but also at the Primary Level. With this in mind and out of the need to stem the problem from a tender age, I have decided to carry out a usability study of an open source software for teaching basic mathematics skills to grade one students in Jamaican primary schools. Usability testing “is a technique for
  5. 5. GCompris Usability Testing 5 ensuring that the intended users of a system can carry out the intended tasks efficiently, effectively and satisfactorily.” (Gaffney, 1999) The chosen open source software for this purpose is GCompris, which is an educational software suite comprising of numerous activities for children, aged 2 to 10. Some of the activities are game orientated, but nonetheless educational. ( )More specifically, I will be looking at the extent to which G-Compris, as a tool for teaching basic mathematics skills can improve or aid learning in mathematics for grade one students at August Town Primary school. Worth of Study The worth of this research is multi-facet in nature. If we can identify software that can aid and improve the performance of children from a tender age, we will not only be facilitating improvement in the grades but also a greater appreciation and a more fun way of learning the subject. It is of academic value because it provides information to educators and their stakeholders as to the value of educational software to students and the implications it can have. In addition, it provides valuable information to be taken into consideration by developers of educational software for students. Objectives of the study include: 1. To determine whether or not Gcompris is effective and appropriate for children six (6) and seven (7) years of age 2. To use the findings of this research to make recommendations to Educators as to one way of improving students performance in school
  6. 6. GCompris Usability Testing 6 3. To prove whether or not Gcompris as an educational tool is beneficial to children 4. To evaluate students learning potential based on the activities available on GCompris 5. To document findings that are as a result of the usability research done. Dominant terms that will be used throughout this research are Usability, usability testing, educational, multimedia, multimedia cognitive theory, activity theory and children. There is an Independent Variable and a Dependent Variable. The independent variable is GCompris and the dependent variable is the August Town Primary Grade one (1) students. The main theories that will be guiding this research are the Multimedia Cognitive theory, Cognitive Load theory, The Multimedia Principle and the activity theory. For this testing, eighteen grade one primary student with little or no computer experience will be chosen randomly. The eighteen students will be further divided into two groups of nine and exposed to the software for 20 minutes on two separate occasion. The students will be observed, recorded, and a post-test evaluation will be done. This research report provides background, findings and recommendations of the Gcompris Usability testing. It documents and discusses theories and methodologies of the testing between March 17 and April 21, 2009. The report provides participant demographics, test evaluation methods and detail additional information that is necessary for this report. Our recommendations are based on the data-driven findings and the user-centered theory.
  7. 7. GCompris Usability Testing 7 Literature Review According to Sashoi Green, teacher at Jamaica College (Clarke, 2009, para 6) quot;There is a phobia of mathematics, but when students relate it to things around them, they connect more.quot; In
  8. 8. GCompris Usability Testing 8 a report in the Jamaica Gleaner, She said students develop a dislike for mathematics at an early age. (Clarke, 2009, para. 8). This line of thought is in keeping with the train of thought that exists. In fact, with the current low performance in mathematics within Jamaican Schools, new policies have been designed and researches are being conducted in an effort to stem the problem. Another perception of the cause of this low performance is provided by the Government of Jamaica who has admitted in a statement recently by Opposition spokesman on education, Senator Basil Waite that “Jamaican students are performing poorly in the area of mathematics because many primary school teachers did not pass the subject at high school level.” (Douglas, 2009, para. 1) This problem has led to developments, research and policies that have surfaced. One example of such is the New Horizons for Primary Schools (NHP), which was implemented by the United States Agency for International Development (USAID) in 1999 and lasted until 2004. “The objective of the program was to improve the language arts and mathematics performance of 72 of Jamaica’s poorer performing schools, through a school-based model of intervention.” (Marlaine, Harris, Gammill, & Barrow, 2006, p.1 ). “The NHP employs a holistic, student centre approach to improvement of instruction and learning in mathematics and reading (n.d.) Another current example is the National Mathematics Strategy being undertaken by the Education Transformation Team under the Ministry of Education. National Mathematics coordinator, Tamika Benjamin, says the launch of this strategy was the “beginning of a quot;long and difficult journeyquot;. (Anonymous, 2008, para. 9) “Through this strategy, direction will be given to stem the continued decline in student performance and to increase national awareness about the importance of
  9. 9. GCompris Usability Testing 9 numeracy.”(Anonymous, 2008 ,Para. 2) the establishment of a National Mathematics and Numeracy Policy, which is aimed at improving the performance of students’ island wide. While there are many theories that speak to learning and the use of multimedia application to aid the learning, the theories I will be focusing on for the purpose of this research are The Cognitive Theory of Multimedia, Cognitive Load theory, The Multimedia Principle and the activity theory. According to Mayer & Moreno (2002) The Cognitive Theory of Multimedia states that multimedia narration and graphical images produce verbal and visual mental representations, which integrate with prior knowledge to construct new knowledge.(to cite). According to Mayer and Moreno (1998) and Mayer (2003), the Cognitive Theory of Multimedia Learning is based on several assumptions. First, working memory includes auditory and visual channels, which are equivocal to the phonological loop and the visu CognitiveTheory & Multimedia Instruction spatial sketch pad in Baddeley’s (1986) theory of working memory. Second, each subsystem of working memory has a limited capacity, consistent with Cognitive Load Theory (Sweller, 1988, 1994). Third, humans are knowledge-constructing processors who produce meaningful learning when they attend to relevant incoming information, organize the information in coherent representational structures, and then integrate it with other existing knowledge (Mayer, 1996, 1999). Fourth, connections can be made only if corresponding visual and verbal representations are in working memory at the same time, which is similar to Paivio’s (1986; Clark and Paivio, 1991) Dual Coding Theory.
  10. 10. GCompris Usability Testing 10 “Cognitive Learning Theory is concerned with the way in which a learner’s cognitive resources are focused and used during learning and problem solving, suggesting that for instruction to be effective, care must be taken to design instruction in a way as to not overload the mind’s capacity for processing information.”(to cite) Multimedia Principle The multimedia principle states that better transfer occurs from animation/pictures and narration/ words than from words alone. When words and pictures are both presented, learners have the chance to construct verbal and visual cognitive representations and integrate them. Limitation to this study includes the fact that there have not been much recorded accounts of similar studies conducted. This limits the resources available to the researcher and the information for which to base the research on. The Activity Theory based usability testing methodology offers evaluators an insight into the natural context of use in an artificial laboratory setting which offers a high degree of control. (to cite) Methodology For the purpose of this research, the group that I have decided to study is grade one (1) students at the August Town Primary School. Eighteen students were randomly chosen, Nine from each of the two grade one students classes that were present. In an effort to effectively choose methods that would reflect the required data needed, two main usability evaluation methods based on Preece (1993), were considered and chosen. These are
  11. 11. GCompris Usability Testing 11 observational evaluation and survey evaluation. “Observational evaluation implies collecting data that provide information about what users do when interacting with educational software.” Preece (1993) stated that two broad categories of data may be obtained: how users tackled the tasks given, where the major difficulties lie and what can be done; and performance measures like frequency of correct task completion, task timing, frequency of participant errors. “Surveys are employed to know users' opinions or to understand their preferences about an existing or potential product through the use of interviews or questionnaires.” The techniques that were used include direct observation, video recording, software logging, verbal protocols and Questionnaire administered by interviews. The Gcompris usability evaluation was conducted as monitored test sessions. Eighteen (18) students were tested. While this number is not representative of the general grade one population in Jamaica for a true experimental design, a minimum of 10 to 12 participants per condition must be used (Spyridakis, 1992). On the other hand, new research has shown that four to five participants can reveal up to 80% of usability insufficiencies in products, which often times tend to be most of the major problems (Virzi, 1990). The test sessions were held in a computer lab on the computer science department of the University of the West Indies, Mona campus. Procedure The eighteen students were divided into two groups of nine and the tests carried out in four sessions, each group being exposed to two. Each testing lasted for twenty minutes per session. From henceforth on, the two groups will be referred to as Group A and Group B. For the first session, Group A had three persons monitoring the test and Group B had five persons. This was due to the availability of persons to participate as monitors. The students were
  12. 12. GCompris Usability Testing 12 transported from their school to the lab. On arrival, the participants were lead to the computer lab where there were each placed in front of a computer which had the software on screen. The students were then given one minute without instructions to see if they would know what to do. After the initial minute, the participants were first introduced to the computer keyboard aspect of the software where they were allowed to do one (1) activity of falling letters to allow them to familiarize themselves with the keyboard. After the basic introduction with the computer, the students were then exposed to basic addition and subtraction mathematics activities. All 9 students were exposed to the mouse where they were allowed to do an activity that required them using the mouse to click the correct answer. Two activities were chosen for this. 1. where they were allowed to connect the numbers to draw an object. This not only aided their mouse using skill but also tested their skills in counting. 2. Others were tested based on their ability to read and identify objects where they were shown different objects and they were required to click the letter that was missing. This first session was used as an introductory session. The students were monitored and comments and expressions were monitored. The second session however was of a different format, as the students were first exposed to mathematics addition activities for the first 10 minutes then subtraction activities for the other ten minutes. Their individual score was documented, after which a simple questionnaire was administered to the participants individually.
  13. 13. GCompris Usability Testing 13 Participant Summary Eighteen participants participated in the Gcompris usability study. Participants were randomly selected based on their availability to participate in the study from the two grade one classes at August Town Primary. The selected participants were mixed in their performance in mathematics. That means some participants were exceptional performers, some needed extra help and the others were rather slow. The participants were each given permission letters to give to their parents/guardians, which were signed and returned to the researcher. Participant Age Group Sex Previous Computer Used other Previous Usage(1-Expert, 5- educational Gcompris Novice) software Usage 1 6 1 F 5 None None 2 6 1 M 5 None None 3 6 1 M 5 None None 4 7 1 F 5 None None 5 7 1 F 5 None None 6 6 1 F 5 None None 7 7 1 M 4 None None 8 7 1 M 5 None None 9 6 1 F 4 None None 10 7 2 F 5 None None 11 7 2 M 5 None None 12 6 2 M 4 None None 13 7 2 M 3 None None 14 6 2 M 5 None None 15 7 2 M 5 None None 16 6 2 F 5 None None
  14. 14. GCompris Usability Testing 14 17 6 2 F 5 None None 18 6 2 F 5 None None Gcompris Set-up Gcompris was pre-installed for each participant on the linux operating system at the computer science computer lab. During the pretesting familiarizing section of the test, the participants were introduced to the welcome screen where no instruction was given at first. The participatory observation as an instrument of data collection is more appropriate in this case than other instruments that exist. In that , while dealing with the children it is much better to be able to interact with them while observation their reaction and behavior towards the software. Furthermore, this is best suited as a usability study consists of and generally involves measuring how well test subjects respond in four areas: efficiency, accuracy, recall, and emotional response, which will be useful in what we are trying to achieve. The aim of the evaluation was to validate the usability of the activities specifically the addition and subtraction mathematical activities which, are features of Gcompris. The testers were given activities to complete and their effectiveness, efficiency, reaction and satisfaction of the software were noted.
  15. 15. GCompris Usability Testing 15 Data Presentation Of the eighteen participant respondents, nine were female and the remaining nine male. Giving a fifty-fifty percentage. However, group one had five females and four males while group two had four females and five females. The respondents are all between the age of six and seven. In fact of the ten of the participants are six years of age while the remaining eight are seven years of age. Usability Problems Number of Participants affected Time given for 18 activities Needs Supervision 14
  16. 16. GCompris Usability Testing 16 Navigation 14 Participants were 18 unsure what to do Figure 1: Major Usability Problems and participants affected. Figure one shows the major usability problems and the number of participants affected by it. The Time given to complete the activities is at the top because it is the greatest concern among the participants. However with time and with practice, the respondents will be allowed to complete the activities within the given time. Analysis From the test conducted and from general observations and posttest evaluation, various findings from which generalizations can be made were found. With reference to the effectiveness of the GCompris it was found out that although the majority of the participants enjoyed playing the other activities they did not like the math activities that were available for more than one reasons, time availability and their ability perform well at the activities. Although the math questions being asked in the activities were appropriate, 80% of the participants felt that they needed to be guided when participating in the activities. This is due to their ability to calculate and function on their own as these participants were not as academically inclined as the others.
  17. 17. GCompris Usability Testing 17 While 50% of the participants did not have a problem with the mathematical activities, the time issues was of a major concern to these participants as they used their fingers to help them calculate. However based on the researcher’s observations, the participants will eventually perform better once they are exposed to the software and general computer usage skills on a weekly basis. Twenty percent of the participants managed the mathematical program very well because they were able to apply the skills taught in basic computation to those that were being tested on the software. For example, one student was given a computation of 10-6=___, immediately she held up her ten fingers and subtracted 6 and was able to reel the answer (4) in seconds. Another option was the reading skills which asked them to fill in the missing letters. This was done with ease as well because the students already had a grasp of the words and so identifying the missing letters was pretty easy. Owing to the fact that they were using the computers for the first time the difficulty came when they had to identify the keys on the keyboard in order to get the right answer. It is also important to note that the age group used for this exercise was 6-7 so one might say that the level of work is very easy, but the ability of the students was mixed and so there were those who were performing at a lower level and those who were excelling. Nevertheless, the students of both mental capacities were able to use the software with relative ease and explanation at times. Overall, the participants were enthusiastic about the software and the activities available on the software. They enjoyed playing the various games and it for example (insert comments here)
  18. 18. GCompris Usability Testing 18 The list below represents some findings based on our participant responses to the feedback survey: • Of our 18 participants, 14 indicated that they had no previous computer exposure. • Two of the four with previous experience said they had access to a computer at home. . • Of the 18 participants, none had previous exposure to Gcompris 10 out of the 18 participants suggested they would only use the software again if they had someone to help them as these students stated that the math activities were too hard. These 10 students happen to be the ones that scored the lowest on the activities 1. Participant 11 Boy is confused when it comes it comes to math, you have to completely break it down for him or work with him. He is easily distracted and works better on other activities 2. Participant #12 is unable to do basic addition, she is very slow and loses focus easily when she gets the answers incorrect. She s unable to recognize basic mathematics signs on her own. 3. Participant number 10 is very fast, she uses her fingers to assist her in solving problems. She works off her own initiative and is twice as fast as the others 4. Participant #1 likes math Likes to do it on the computer Likes the “thing” that she is using to do the math 5. Participant #7 likes subtraction -father encourages him to do math
  19. 19. GCompris Usability Testing 19 -likes the software -the math was too hard -wanted someone to help him because he did not know what to do -would not use it again because it was too hard. 6. Participant #5 – likes it because its nice Likes the game with the fish It was too hard because he can not do the math He would use it again if he had somebody to help him 6. Participant # 3 it was fun Fish game is her favourite The math was not hard Yes would use it again Has a computer at home. 7. Participant #2 thought some of the math were hard, wanted someone to help with the ones that were hard, could do the math When asked if they would like the teacher to use it in class 90% said yes Activities the children liked
  20. 20. GCompris Usability Testing 20 1. Letters that fall from the sky - 2 2. One participant response was her stomach hurts her when she does not know what to do. 3. Another one likes the addition Disapproval - The time was too short - They were not familiar with the layout of the computer keyboard It is also important to note that the age group used for this exercise was 6-7 so one might say that the level of work is very easy, but the ability of the students was mixed and so there were those who were performing at a lower level and those who were excelling. Nevertheless, the students of both mental capacities were able to use the software with relative ease and explanation at times. Conclusions and Recommendations From this usability testing research on the extent to which G-Compris, as a tool for teaching basic mathematics skills can improve or aid learning in mathematics for grade one students at August Town Primary school, numerous recommendations can be made. Some of these recommendations have been put forward by monitors and participants themselves and the others
  21. 21. GCompris Usability Testing 21 will be put forward by the researcher based on the process and findings of the research. Recommendations put forward by the monitors/evaluators/participants include; o The length of the time on the software needs to be slower which will seek to accommodate students that have learning disabilities and are not so quick. o Students on initial exposure to the software have to be closely monitored in order to fully utilise and captivate the benefits that can be had from the software. Recommendations put forward by the researcher include: • A comparative study to be done between a preparatory and a primary school. • More recorded research including that of usability testing should be done in the Caribbean so that Caribbean researchers including students may have readily available literature to them when conducting research of this nature and others. • The developers of children software needs to take into consideration children who are not tech-savvy hence the researcher is recommending that the developers of GCompris allot more time to the completion of the initial activities in order to allow users to get familiar and comfortable with the product. • A computer lab should be made available to the students with academic software such as Gcompris installed so as to exposure them to alternative methods of learning which will be beneficial to the teachers and students alike. • The Ministry of Education should re-visit the syllabus In conclusion, the researcher discovered that while GCompris can indeed improve or aid the performance of students in mathematics, the timing available for completion of the activities needs to be revisited for students that are not that academically inclined and needs more time to work on the activities. The fact that the most of the participants who were all unfamiliar with
  22. 22. GCompris Usability Testing 22 GCompris could learn to use it fairly with just two exposures suggests that GCompris is generally usable for the intended target population. However the developers of the software needs to look in lengthening the time given for completion of each activity as this present timing is not conducive or receptive of slow learners.
  23. 23. GCompris Usability Testing 23 References Clarke, O (2009, April 18) Curing the math phobia - Math competition shows problem solving to be a good tool. Retrieved January 27, 2009 from Douglas, L (2009, April 5) 'WE HAVE A MATH PROBLEM'. Retrieved January 27, 2009 from 20_OBS__WE_HAVE_A_MATH_PROBLEM__.asp Francis, P (2006, September 15) Poor CXC results - Jamaican students lag behind region in Maths, English. Retrieved January 26, 2009 from Gaffney, G (1999) Information & Design: Usability Testing Series. Retrieved January 26, 2009 from Marlaine, L., Harris, A, Gammill, P & Barrow, K (2006) IMPACT OF NEW HORIZONS FOR PRIMARY SCHOOLS ON LITERACY AND NUMERACY IN JAMAICA 1999-2004. Retrieved on February 5, 2008 From Preece, J. (1993). A Guide to Usability: human factors in computing. Addison Wesley, the Open University.
  24. 24. GCompris Usability Testing 24 Rubin, Jeffrey. (1994). Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests. New York: Wileyl Spyridakis, J. H. (1992). Conducting Research in Technical Communication: the application of true experimental designs. Technical Communications, Fourth Quarter, 607-624. Virzi, R. A. (1990). Streamling in the Design Process: Running Fewer Subjects. Proceedings of the Human Factors Society, 291-294. Educators count on new maths strategy (2008, June 4) Retrieved on January 27, 2009 from Mayer, R. E. (1996). Learning strategies for making sense out of expository text: The SOI model for guiding three cognitive processes in knowledge construction. Educational Psychology Review, 8, 357-371. Multimedia in Jamaican Schools retrieved on from pdf Appendices
  25. 25. GCompris Usability Testing 25 Caribbean Institute of MEDIA and COMMUNICATION The University of the West Indies 3 Sherlock Drive, Mona Campus Kingston 7 Jamaica, W.I Tel. (876) 927-1481, 977-2111, 977-0898, 977-1398, 977-9990, 702-3353, Fax (876) 977-1597 E-mail Caribbean Institute of MEDIA and COMMUNICATION Web Dear Parent/Guardian, Re: Permission for your Child/Ward to participate in a computer study This letter seeks permission for your child/ward to participate in a multimedia computer study to be conducted at the University of the West Indies Mona Campus. Although we would have preferred to conduct the study on the premises of the school, the computer lab is currently unavailable for use, hence the need for the study to be done on UWI’s campus. The project will run for 2 to 3 weeks commencing on Tuesday March 17, 2009. It will require that your child visits the UWI computer lab on two occasions within one of the following sessions (Tuesdays and Thursdays 9:00 am – 10:30 am) during duration of the study. Shakarah McCrae, a final year student at the Caribbean Institute of Media and Communication (CARIMAC), will conduct the study, which is intended to fulfil her research requirement to investigate multimedia software and children in the classroom. Your child will be required to use a computer with Mathematics learning software for a period of 20 to 30 minutes. Sincerely, ---------------------------------------- David Bain Multimedia Co-ordinator CARIMAC, UWI Mona
  26. 26. GCompris Usability Testing 26 ----------------------------------------- Shakarah McCrae Final Year Student CARIMAC UWI Mona ----------------------------------------- Morlton Wilson Principal, August Town Primary School -------------------------------------------------------------------------------------------------------------------- - I ____________________________________ have given permission for Parent/Guardian __________________________ my child/ward to participate in the multimedia computer study to be held between March 17, 2009 and March 26, 2009 at the UWI campus.