This document summarizes an honors thesis project that examined how website navigation design can impact cognitive load and information intake. The project involved designing an online game to test navigation of a mock university website between an experimental and control group. Key findings included the experimental group performing better on fact and menu recall questions, indicating the navigation design supported lower cognitive load. However, the study also highlighted lessons for improving the design of the online game and survey instruments to better test the hypotheses.

1. Web Site Navigation Design with Respect to Cognitive Load Aimee Maree Forsstrom Supervisor Dr Graham Cooper Honours Presentation Southern Cross University

2. Interdisciplinary research Computer Science Educational Psychology Cognitive Load and Human Computer Interaction Does web site navigation design effect how we take in information How can we apply Cognitive Load Theory to web site navigation design to assist the intake of information The story so far…

3. Timeline - Key Dates Honours Thesis started in February 2008 Original plan to run cognitive load tests on Australian Learning and Teaching Council June 2008 First Honours presentation July 2008 Honours update presentation October 2008 Plan changed to be run on a mock up University Faculty website January 2009 Plan changed to be run on a online game presenting random facts to the user May 2009 Ethics received in August 2009 Experiment ran first two weeks of September 2009 Final Honours presentation October 2009

4. Research in the Social Network era Participants informed online Tweets placed on www.twitter.com Message sent to friends on www.facebook.com Link posted on www.facebook.com profile page Email sent to all Southern Cross University staff/students Email sent to all Programming Society members (UTS) 202 participants played the game 124 meet needed requirements 118 participants in the final clean data set 59 in control group 59 in treatment group Participants in both groups balanced on age/gender/comp use

5. What we learnt Social networking is a great way to reach large number of participants Participants numbers coming to the site will peak and then slowly die off Online game was a great tool to drive people through website navigation People don’t mind playing a game and answering some short questions Expect a percentage of drop out’s and possible technical glitches (40% error rate encountered) Track mouse paths through all stages of the site to ensure proper human activity (not a robot) Data must be thoroughly cleaned and results checked to ensure there are no anomalies from technical issues or cheating

6. Experiment Design

7. Playing a Game can be fun Online fact learning game design chosen 15 questions presented in random fashion Answers where found by navigating through the web site menu 5 top levels of categories in menu 4 lower level menu options per category Using a game we can drive the participant through the menu maximum use of menu 15 clicks with short time frame 10min Engage participant in web site material and appeal to wide audience

8. Experiment Components Research online check Random fact game play 10min, 15 questions Effort survey Fill in facts, participant to fill in missing words from 10 facts presented in game Menu rebuild, participant asked to rebuild the menu to show if they acquired the menu schema Usability Survey

9. Random Game start page presented Participant asked to scroll to bottom of page and roll dice After the dice is rolled participant is given a question and asked to locate the section in the menu that will hold the answer The menu sections are named after key words in the question asked If the participant clicks on the wrong menu they are asked to re-read the question and try again Once the final answer is read the participant is directed to answer questions about the game Experiment Steps

10. Data Analysis Methods

11. Statistics Non Parametric and parametric statistics used Distribution was heavily skewed so Parametric statistics could not be used on most questions Qualitative and Quantitative data analysed

12. Survey asked on Effort Two questions Questions answered by 7 point likert scale Mann Whitney U Test Descriptive Statistics

13. Survey asked on Usability Two questions Questions answered by 7 point likert scale Mann Whitney U Test Descriptive Statistics Three open Ended questions Linear regression on positive and negative word count Linear regression and on word categories

14. Fill in Facts Questions 10 Fill in fact questions grouped by easy, medium, hard Mann Whitney U test on overall score average ANOVA Question type score and group interaction Variables Easy, Medium, Hard questions, Factor group id Chi-Square test for each individual question Group comparison of the number of participants who got the question right Wilcoxon Ranked Sum Test Difference in individual question answers between groups Group Average for Fill in facts question completion time

15. Menu Schema Rebuild Menu Rebuild, 18 menu answers captured Mann Whitney U test on overall score average ANOVA Menu top, lower menu rebuild and group interaction Variables top, lower level menu sections, Factor group id Chi-Square test for each individual menu element Group comparison of the number of participants who got the question right Wilcoxon Ranked Sum Test difference in individual menu section answers between groups Group Average for Menu rebuild time

16. Mouse path collected for each group (screen click point captured) Comparison of mouse click path used in game Overall random facts game time T-test of game time means Random Facts navigation time between each of the 15 questions T-test of individual navigation steps Mouse Paths and Game Time

17. Preliminary Results

18. Effort Survey Likert Question One Effort one question displays slight backward trend How easy was it to use the menu to play the game Treatment group mean 2.05 Control group mean 1.85 1 = very easy 2 = easy

19. Statistical Significance found for Effort question two How distracting did you find the scrolling Effort Survey Likert Question Two Treatment group mean 2.71 Control group mean 3.51 2 = very slightly distracting 3 = slightly distracting 4 = distracting

20. Usability Survey Likert Questions No difference in likert scale questions After rolling the dice for each question locating the navigation menu was easy Treatment group mean = 5.37 Control group mean = 5.34 I enjoyed playing the random facts game Treatment group mean = 4.97 Control group mean = 4.97 4 = neither agree nor disagree 5 = slightly agree 6 = agree

21. Usability Survey Open Ended Qualitative data was analyzed for positive and negative words mentioned Experiment Group had higher positive score Treatment Group had higher negative score 45% of the treatment group asked to change the scrolling in the game 12% of the experiment group asked to change the scrolling in the game

22. Fill in Facts Questions Statistical significance between groups for overall question score Treatment group outperformed the control group on overall test scores Treatment group mean 13.56 Control group mean 12.05 Maximum Score for facts = 17

23. Fill in Facts Questions Statistical difference found for overall mark on question type and group id Treatment group outperformed the control group on the medium to harder questions

24. Fill in Facts Questions Statistical difference found for individual question marks between group Treatment group outperformed control group

25. Menu Schema Rebuild Statistical significance not found on overall scores Difference in means 14.03 for treatment group and 12.02 for control group not significant Treatment group mean 14.03 Control group mean 12.02 Maximum Score for facts = 33

27. Lessons Learnt Need twice as many questions 30 instead of 15 keep participants engaged longer game play average was 5min not 10min More elements in menu schema rebuild 30 instead of 18 Need for more medium and heavy questions 10/15 instead of 6/10 easy/hard question ratio Ensure wording of questions crosses borders Include more survey questions Perform more technical checks and pilot

28. Thank you