1University of Aizu, Graduation Thesis. March, 2012 s1170173AbstractThere is scant literature in language studiessuggesting the efficiency with which EFL learnerswith low English language proficiency are able toprocess English website content, and their opinionson the same. Structured website analysis has thepotential to be a stimulating exercise because itaddresses various types of questions that are veryunique to the medium and might not be immediatelyapplicable to other contexts. This article reported onan experiment where 41 junior-level Japanese EFLreaders were asked to study an English tourismwebsite closely, and then answer a set of questionswhich focused entirely on readers’ ability to excavateinformation by navigating through the websitecontent. Further, several survey questionnaires (QUIS,CSUQ, and MPRC) were used wherein readers wereasked to self report their levels of comfort with thewebsite, and the words they would choose to describetheir feeling about the website. Results suggestedrelatively higher levels of proficiency ordering thesteps required to navigate and search for specificinformation from the given website. Moreover,readers were mostly comfortable searching forinformation from all over the website. Self-reportssuggested relatively moderate levels of comfort withdifferent tasks and access features (overall reaction tothe website; webpage design, terminology andwebsite information; learning; website capabilities)related to the website. Finally, data shows that thenumber of positive words chosen to describe thetourism website is way more than the negative wordschosen.1. IntroductionReaders of a traditional printed text read in orderof ascendency. However, website readers mustchoose between hyperlinks which could possibly takethem to different related or unrelated sections of thebroader webpage, and often to other web pages aswell. Carrell (1987) suggests that organization of atext affects reading comprehension as well as recall.She said that signaling devices in the text helpreaders recall the information in the hierarchicalstructure of the text and improves text comprehension.The signaling devices help connect one part of thetext to another and improve content comprehension(Kintsch & van Dij, 1978). However, suchconclusions on reading ability and comprehensionmight be completely lost when reading hypertext-richwebsites are accessed. In this context the hierarchicalstructure of the text is completely lost because of theskipping around to different parts of the text that isinvolved (Charney, 1994).The efficiency in reading depends on readers’schema which helps detect the structure of a text. Thereaders’ schema functions as an abstract script andthe reader expects the script to match the text he/shereads (de Beaugrande, 1980). The readers’ schema isoften influenced by the website’s graphicmanifestations that lead to a shift from the centralityof text (Burbules, 1998). However, the authorsuggested that image does not replace the need fortext. Rather, the idea behind use of graphics is tosummarize the importance of the message that textcommunicated, but in a quick glance. Graphics tendto serve a pointing function. Boardman (2004)suggested that webpage creators often choose short,dense phrases, rather than full sentences tosummarize the core of the message.However, we are not entirely sure of how thisexperience translates to an EFL context wherereaders’ English proficiency is at a very low level.On a very specific level the question is the efficiencywith which EFL learners with low English languageproficiency are able to process English websitecontent, and what is their opinion about the website?Structured website analysis with open-endedfeedback as a tool for foreign language learning hasthe potential to be a stimulating exercise because itaddresses various types of questions that are veryunique to the medium and might not be immediatelyapplicable to other contexts. Very little is knownabout user’s ability to provide open-ended feedbackon website content (Elling et al., 2012). Moreover, inan EFL context, website analysis might deal with arange of issues related to learning difficultiesresulting from lack of language proficiency, variableEFL Student’s Ability for Website Information Comprehension andPerceptions on Website UsabilityMakoto Yoshida s1170173 Supervised by Prof. Debopriyo Roy
2University of Aizu, Graduation Thesis. March, 2012 s1170173use of cognitive and meta-cognitive strategies forinformation comprehension on the web (Lomicka,1998) or in an intensive English learning context(Hong-Nam and Leavell, 2006), individualdifferences in learning with hypermedia (Knight,1994), besides complexity in the web content itself.These above-mentioned areas of research directly orindirectly contribute to our general understandingabout how L2 learners perceive English websitecontent and design, and how specific readingstrategies (self-reported) might impact designdecisions.As part of the experiment reported in this article,Japanese EFL readers were asked to study a websiteclosely, answer design questions related to thewebsite and how it is structured, and then answer aset of questions which focused entirely on the contentof the website. The purpose behind this questionnairewas to judge whether readers actually have readthrough the website, and could identify where theinformation is placed in the website. Answering thequestions in this questionnaire successfullydemanded both reading ability and identifying linksthat takes the reader to the required information.Further, readers were asked to self report what theythought specifically about the website with differentquestionnaires, and the words they would choose todescribe their feeling of the website. This paper issolely based on exploring readers’ ability to identifyand read website content successfully, and self-reports of their feelings of the website. The purposebehind this activity was to identify what botheredthem about the website information comprehension,what they liked about the navigation, their levels ofmotivation and feelings about the website etc. Thiswill allow L2 researchers to get some sense of howEFL readers with low language proficiency approachthe task of reading an English website.2. MethodParticipants (N=17) for the first stage of theanalysis (Week # 1) are junior level students (agegroup: 18-20 years) in their third year undergraduateprogram specializing in computer science in aJapanese technical university.First Week: During the 1st week of the actualexperiment, a randomly selected section of thestudents analyzed the Belize tourism website based onthe 8 open-ended design questions about the website.However, the analysis of how efficiently studentscould answer the design questions is beyond thescope of this study.Second Week: During the 2ndweek, readers (allstudents in the class - N = 59) were given a set ofquestionnaires to answer on the same Belize tourismwebsite. The websites focused on three differentaspects related to information comprehension andonline reading strategies for an EFL context. Thequestionnaires focused on three different aspects.• Their ability to navigate through and look forinformation from the website.• Their ability to meaningfully self-report on theusability of the website in terms of navigation,content, and organization.• Their ability to meaningfully self-report on theirreading strategies when analyzing the Belize tourismwebsite.All the questionnaires with the above-mentionedfocus were completed in class.Instruments:The Belize tourism website was chosen with thefollowing reasonable conditions in mind.• The content in the website is not text heavy andclear navigation is possible.• Information could be searched directly from thehome page.• Attractive pictures are available to keep the readerengaged in the task of finding information.• Japanese version of the webpage is NOT availableso that readers are forced to look for informationfrom the English version alone.The instructions for the first week of the assignment(where readers had to respond to 8 open-endedquestions) were all in English, largely because readersalready had practice from the weeks before regardingwhat is expected of them. For the second part of theexperiment (held during the 2ndweek), all thequestionnaires were administered in Japanese, toensure proper understanding of the content andinstructions. Readers were handed out an instructionsheet in both Japanese and English explaining what isexactly expected of them for the task. The second partof the experiment comprised of several tasks.- 1stTask: To check for readers’ ability to navigatethrough and look for information from the Belizetourism website, a website information comprehensionquestionnaire was designed specific to the Belizetourism website. The questionnaire had to answer 8questions. For the 1stquestion, readers were given aspecific scenario where they were asked to find out a
3University of Aizu, Graduation Thesis. March, 2012 s1170173hotel in a specific location in a given price range.Readers were given a list of steps they had to follow inorder to find the information from the Belize tourismwebsite. For all other questions except one, they had tochoose the correct option regarding information that isavailable in the website. They can only choose thecorrect option when they have actually tried to look forinformation from the website. This should not becalled a questionnaire in the true sense, because thequestions were geared towards searching forinformation from the website and were completelyunrelated from each other, with no pattern or similarityor were not part of any specific sub-group of analysis.- 2ndTask: A set of three usability questionnaires washanded out as part of this task. The questionnaires arenamed as QUIS, CSUQ, and Microsoft ProductReaction Card respectively. The first twoquestionnaires used a Likert scale for each question.The review of the literature has more details aboutthese questionnaires. These questionnaires aredesigned not only to elicit readers’ impression of theBelize tourism website in terms of content, navigationand usability, but also judge the extent to which theirimpression on items matches or deviates from thejudgement made by the more experienced coders.Assessing Website Usability:According to the IBM technical report, most usabilityevaluations gather both subjective and objectivequantitative data in the context of realistic scenarios-of-use, as well as descriptions of the problemsrepresentative participants have trying to complete thescenarios. Subjective data are measures ofparticipants opinions or attitudes concerning theirperception of usability. Objective data are measures ofparticipants performance (such as scenario completiontime and successful scenario completion rate) (Lewis,1993). Usability is recognized as an important qualityfactor of any modern website (Avouris et al., 2003).Literature on web usability has reported the use ofvariety of questionnaires for reporting perceivedusability of interactive systems (Tullis & Stetson,2004). As part of this study, several questionnaireswere used, based on the study performed by Tullis andStetson (2004) on comparison of questionnaires forassessing website usability.QUIS (Questionnaire for User Interface Satisfaction)was developed at the University of Maryland, andcomposed of 27 questions. However, for websitecontext, we adopted 24 questions that are appropriate(http://www.isr.umd.edu/news/news_story.php?id=4099). This questionnaire was used for obtaining self-reported impression of the website on five categoriesof information. They are (1) Overall ComprehensiveEvaluation of the website (2) Website design (3)Terminology and website information (4) Learningfrom the website and (5) Website capabilities.CSUQ (Computer System Usability Questionnaire)also used for this study was developed at IBM and wasoriginally developed for computer systems. However,it was adopted for websites in this context(http://drjim.0catch.com/usabqtr.pdf). The purpose ofthis website is to get self-reports on overall ease ofusing the website, and related efficiency of use.Finally, the Microsoft Product Reaction Card wasused for this study, to obtain qualitative reviews on theimpression about the website. A modified list, basedon originally developed 121 words, was presented tothe user and was asked to choose the words that bestrepresented their interaction with a website. Readerswere free to choose any many or as few words as theydeemed appropriate (Benedek and Miner, 2002). Theabove questionnaires were used for the study asreported in this article.For the Website Information ComprehensionQuestionnaire, QUIS questionnaire, CSUQ andMicrosoft Product Reaction Card: The three coderswere independently asked to grade the questionnairesthat readers completed during the second weekanalysis and reporting.For the website information comprehensionquestionnaire, grading was straightforward in terms ofwhether the reader could correctly identify the correctoption after searching for information from the website.However, for each question, I asked the coders toconsult amongst themselves as to whether there issome information that is truly difficult to locate, or ifthere are alternate ways of finding the sameinformation. However, for each reader, and for eachquestion there was only one grade assigned (2=corrector 0= incorrect). The project coordinator offeredcoders with expert advice as to how they can searchfor answers to the questions asked in the questionnaire,and then grades the responses for accuracy. For thisquestionnaire, each response was graded only once bythree coders who worked together to decide on a grade.For the QUIS, CSUQ and Microsoft Product ReactionCard questionnaires, coders recorded the self-reportsby the readers coming together as a group. This groupactivity was important to make sure that no wrong datawas entered into the system during data collection.
4University of Aizu, Graduation Thesis. March, 2012 s11701733. Findings3.1. Website Information ComprehensionQuestionnaireThe website information comprehensionquestionnaire showed relatively moderate levels ofaccuracy, indicating the fact that text comprehensionfrom the web pages, understanding the navigationstyles of the Belize website, understanding the linkedpages, etc. were moderately successful. However,since the questions in this questionnaire weredesigned to test whether readers are actually capableof searching through the information in the website,the different questions in the questionnaire can’t beconsidered to be testing anything specific. Rather,different questions tested searching ability fordifferent types of information from the website, andin no order of complexity or similarity of content.This was evident from the reliability test done for thequestionnaire, with a Chronbach’s alpha valueof .086 (from Q2 ~ Q9). So, it probably won’t makesense to term this as a questionnaire. Instead thepurpose would be to see the accuracy with whichindividual questions were answered and relate it tothe self-reports in other questionnaires.For most questions, students scored in the range of1.19 ~ 1.97 showing some variability in the scores.This indicates that some questions were relativelyeasier to answer, while some were more difficultdepending on how much the information had to besearched from the linked pages in the website. Table1 shows the details of the website informationcomprehension data. On an average, for questions 2 ~8, the data shows accuracy of searching forinformation from the website at around 83%. For thefirst question which asked readers to sequence stepswhen searching for specific information from theEnglish website, the accuracy could be pegged at86.6%. Table 1 shows the mean accuracy for thedifferent questions asked in the website informationcomprehension questionnaire. The mean accuracyscores from Q2 ~ Q8, shows high values for Q2 andQ3, but a drop in the mean score for Q3, and Q4, andmore overall fluctuations.Data shows significant correlation vales betweenaccuracy scores for the different questions asked.Results indicate statistically significant correlationvalues involving Q2, Q7, and Q8. This suggestscomparable amounts of accuracy between thequestions, indicating searching for some kinds ofinformation and/or levels of difficulty whensearching for information from the website werecomparable for the questions concerned.We also wanted to test if there is any statisticallysignificant difference between the accuracy scores inthe website information comprehension questionnaire.Results of the Friedman test suggested that responsesare indeed significantly different (χ2(2) = 78.120, P =0.000). Since p-value = 0.00 ≤ 0.01 = α, we rejectedthe null hypothesis that there is no difference betweenthe mean ranks for the accuracy scores.Following the website information comprehensionquestionnaire, participants self-reported theirimpression of the software on different categoriesusing the CSUQ questionnaire.3.2. CSUQ and QUIS Questionnaire ResultsThe purpose of the CSUQ Questionnaire was toexplore self-reported ease of using the Belize tourismwebsite. Results show a significant correlation inmost cases between all questions asked in thequestionnaire, with a handful of exceptions. QUISquestionnaire had 5 different categories namely (1)overall reaction to the website (2) screen (3)terminology and website information (4) learning and(5) website capabilities. Table 5 shows the mean andstandard deviation values for all 5 categories in theQUIS questionnaire. For this questionnaire, in thiscase of reliability testing, a Chronbach’s alphaof .687 was obtained for category 1, .809 for category2, .637 for category 3, .785 for category 4 and .731for category 5. Data also reported the percentagelevels of agreement with the different categories inthe QUIS questionnaire. 100% agreement shows amean value of 5 on a category in a 1 ~ 5 Likert scalewhereas a mean value of 1 shows 0% agreement witha statement. Results here suggest 59.4% agreementon the” overall reaction to the website” category,which goes up to 64.8 for “web page design”category, but then slides back to around the 44 ~ 54%level for the other three categories. Data also showthe number of cases in various categories whereresponses to questions show significant correlationwith responses to other questions either in the samecategory or in a different category. Except forquestion 2, 23, and 24, all questions demonstratedsignificant levels of correlation with all otherquestions across the questionnaire. This resultsuggests significant similarity in what website readersthink about the different features of the website, easeof learning from the website, ease of reading andusing the website interface, navigating through thewebsite, website accessibility and so on.
5University of Aizu, Graduation Thesis. March, 2012 s1170173Next, we wanted to explore if there is significantsimilarity or difference between the self-reportedscores on the Likert scale in the CSUQ questionnaire.Null Hypothesis: Mean Ranks for all the self-reportedscores on the CSUQ questionnaire are equal;Alternative Hypothesis: Not all the mean ranks areequal. Table 7 shows us a Friedman analysis forresponses to all the questions combined.The Friedman non-parametric statistical analysiswas performed to find out if there is an overallstatistically significant difference between the meanranks of the self-reported scores in the differentquestions asked. This test tells us whether there areoverall differences between self-reported scores butdoes not pinpoint which questions in particular differfrom each other. To do this we need to run post-hoc tests, but post-hoc analysis was not considered aspart of this study. Results suggest that there was astatistically significant difference in self-reportedsores for a combination of all the questions asked inthe CSUQ questionnaire, depending on the overallweight of how readers felt about accessing theEnglish website, χ2(2) = 126.626, P = 0.000. Since p-value = 0.00 ≤ 0.01 = α, we reject the null hypothesis.Next, data explored if there is significant similarityor difference between the self-reported scores on theLikert scale in the QUIS questionnaire for thecombined responses on the 5 different categoriesstudied. Friedman analysis was done for each of the 5categories in the QUIS questionnaire and wasconsidered separately. This test tells us whether thereare overall differences between self-reported scoreswithin a category, but does not pinpoint whichquestions in particular differ from each other. To dothis we need to run post-hoc tests, but post-hocanalysis was not considered as part of this study.Results suggest that there was a statisticallysignificant difference in self-reported sores for acombination of all the questions asked in the “overallimpression of the website” (χ2(2) = 21.532, P =0.000), “web page design” (χ2(2) = 24.818, P =0.000), “terminology and website information” (χ2(2)= 60.670, P = 0.000) and “website capabilities” (χ2(2) = 75.683, P = 0.000) categories. However, datashows there was no significant difference betweenresponses when it comes the “learning” (χ2(2) =5.769, P = .217) category. Since p-value = 0.00 ≤0.01 = α, we reject the null hypothesis.3.3. Microsoft Product Reaction Card ResultsFor the MPRC questionnaire that was handed outto participants in the final stage, following interactionwith QUIS and CSUQ questionnaire, we obtained thefollowing results. In total, 116 words were selectedby the readers (out of 121). Readers chose 77 positivewords and 41 negative words. Also, the totalfrequency of positive words chosen was way morethan the total frequency of negative words chosen.For the 10 most frequently chosen words, 8 of thesewords are positive ones and 2 of these words has anegative sense.4. DiscussionQuestion 1 in the website informationcomprehension questionnaire intended to explore theefficiency with which readers are able to follow andconfirm the sequence of activities that is necessaryfor finding certain information. A score of 8.66 in ascale of 10 for all correct answers, suggest high levelsof efficiency in following the steps correctly from anEnglish webpage. Also, scores on other questions(related to searching for information from Belizewebsite) in this questionnaire suggest moderate meanscores. However, there is probably hardly anyliterature to suggest that ability for search in anEnglish context, from an English website might alsoindicate higher levels of English contentcomprehension ability.Further, not a significant number of correlationswere observed between accuracy scores on thewebsite information comprehension questionnaireand self-reported scores on the CSUQ and QUISquestionnaire. This goes on to suggest that self-reports on the questionnaires had a general outlookabout the website and might not have been specific tothe accuracy scores. But, the Friedman test ofaccuracy scores did suggest a significant variationbetween the results of accuracy scores. This goes onto show that in significant number of cases, somereaders could and some could not search through therequired information and locate the answer in thismultiple-choice website information comprehensionquestionnaire. This possibly indicates towards thefact that individuals with high reading and contentcomprehension ability might have a higher statisticalprobability to search for and locate answers fromEnglish website more successfully.The CSUQ questionnaire made positive statementsin different categories and asked readers to respond towhat extent they agree with the statement. Self-reported scores in the range of 3 ~ 4.3 suggestmoderate agreement with the statements. This also
6University of Aizu, Graduation Thesis. March, 2012 s1170173indicates towards the fact that readers were not highlycomfortable with the website as for its features,design, etc. Further, we see a very high number ofsignificant correlations between the self-reportedscores in the CSUQ questionnaire, suggesting similarrange of responses for most questions. However,Friedman test suggests a different result because thisnon-parametric analysis considered the mean rank foreach question, and all the results were consideredtogether, and not in a bivariate context.Finally, we could clearly see that the number ofpositive words chosen to describe the Belize tourismwebsite is way more than the negative words chosen,and also the positive words are chosen with moretotal frequency. This goes on to show a positiveoverall impression of the website. However, thisshould just be taken as an indicator and whether ittranslates to overall better comprehension of thewebsite information, and general learning is adifferent matter altogether.5. ConclusionFrom this study we see conclusive evidence thatthere is wide variability in the efficiency with whichreaders in this EFL context are able to analyze anEnglish website. Future studies could systematicallyfocus on testing accuracy related to different aspectsof searching and reading information from Englishwebsites. These aspects could be directly related todemonstrating efficiency in understanding websitedesign, website capabilities / features, learning aspectof the website, terminology and website informationetc. Further, these accuracy scores would then bematched to self-reported scores on the CSUQ andQUIS questionnaire. This exploratory analysis asdiscussed in this article could help us obtain initialdata on how EFL readers in a typical context as thiswould perform with English website information andthe type of impression they have about the website.This will help us judge their levels of proficiency andthe types of English websites they could be exposedto for various kinds of assignments, task-basedlanguage learning etc.6. ReferencesCarrell, P.L. (1987). Content and formalschemata in ESL reading. TESOL Quarterly, 21,461-481. 1987 TESOL InternationalAssociationVan Dijk, T. A., & Kintsch, W. (1983).Strategies of discourse comprehension. NewYork: Academic Press.Charney, D. (1994). The effect of hypertext onprocesses of reading and writing. New York:Modern Language Association.de Beaugrande, 1980 Text, Discourse, andProcess: Toward a Multidisciplinary Science ofTextsNicholas C. Burbules. "Rhetorics of the Web:Hyperreading and critical literacy" in Page toScreen: Taking Literacy into the Electronic Era,I. Snyder (editor), Routledge, London, 1998.Boardman, M. (2004). The Language ofWebsites (Intertext). New York: RoutledgeElling et al., (2012). Users’ Abilities to ReviewWeb Site Pages. Journal of Business andTechnical Communication Volume 26 Number2 (April 2012)Lomicka, L. (1998). "To gloss or not to gloss":An investigation of reading comprehensiononline. Language Learning & Technology, 1(2),41-50. Retrieved May 1, 2005, fromLomicka, L. (1998). "To gloss or not to gloss":An investigation of reading comprehensiononline. Language Learning & Technology, 1(2),41-50. Retrieved May 1, 2005, fromhttp://llt.msu.edu/vol1num2/article2/default.html.Hong-Nam, K., & Leavell, A. G. (2006).Language learning strategies of ESL students inan intensive English learning context.System,34(3), 399–415.University of North Texas, Denton: Departmentof Teacher Education and AdministrationKnight, J. (1994). Internationalization:Elements and checkpoints (ResearchMonograph, No. 7). Ottawa, Canada: CanadianBureau for International Education.