Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Study of the usability of the WYRED Ecosystem using heuristic evaluatino


Published on

This presentation is related to this publication:

García-Peñalvo, F. J., Vázquez-Ingelmo, A., & García-Holgado, A. (2019). Study of the usability of the WYRED Ecosystem using heuristic evaluation. In P. Zaphiris & A. Ioannou (Eds.), Learning and Collaboration Technologies. Designing Learning Experiences. 6th International Conference, LCT 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings, Part I (pp. 50-63). Switzerland: Springer, Cham.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Study of the usability of the WYRED Ecosystem using heuristic evaluatino

  1. 1. Study of the usability of the WYRED Ecosystem using heuristic evaluation Francisco J. García-Peñalvo, Andrea Vázquez-Ingelmo, Alicia García-Holgado GRIAL Research Group, Computer Sciences Department, Research Institute for Educational Sciences, University of Salamanca, Salamanca, Spain,,
  2. 2. Index 1. Introduction 2. The WYRED Platform 3. Methodology: HE 4. Heuristic evaluation 5. Methodology: UX 6. User experience 7. Discussion 8. Conclusions 9. Acknowledgements HCI 2019
  3. 3. Introduction (I) The young have emerged as a distinct social group, and with it, an understanding that they have a role in social change …but half of them feel that their concerns are not taken into account HCI 2019
  4. 4. Introduction (II) WYRED project aims to provide a framework for research in which children and young people can express and explore their perspectives and interests concerning digital society HCI 2019
  5. 5. Introduction (III) WYRED provides a technological ecosystem to support the research framework and boost international conversations among young people HCI 2019 The ecosystem is composed of a set of Open Source tools and people involved in the project: partners, stakeholders and young people between 7 and 30 years
  6. 6. Introduction (IV) The main component of the ecosystem in which the interaction among young people takes place is the WYRED Platform HCI 2019 The WYRED Platform enables young people to interact inside communities focused on different aspects of Digital Society
  7. 7. Introduction (V) It is important that the ecosystem and, in particular, the Platform, is accepted by its final users HCI 2019 A preliminary study was performed to analyze the usability of the Platform from the point of view of young people
  8. 8. Introduction (VI) This work aims to describe the usability study targeting facilitators to reach insights about how these users value the WYRED Platform usability HCI 2019 A heuristic analysis following the heuristics proposed by Nielsen was executed with four experts with different profiles The heuristic analysis was complemented with the results of the Computer System Usability Questionnaire (CSUQ) (version 3) to collect the experience of real users
  9. 9. The WYRED Platform (I) The Platform is organized in multicultural and interdisciplinary communities where young people can share their ideas and opinions about Digital Society HCI 2019 The communities have different tools: • Forums to support conversations and coordinate research projects • A calendar to share dates and organize events or activities • A form to give visibility to research projects
  10. 10. The WYRED Platform (II) HCI 2019
  11. 11. The WYRED Platform (III) HCI 2019 Strong commitment to user privacy and security; it is designed as a safe space in which children and young people can express themselves freely Users need an invitation to register within the Platform and children under 14 years old need parental consent in order to have access after finishing the registration
  12. 12. Methodology: Heuristic evaluation (I) Participants • A Ph.D. student whose doctoral dissertation deals with customizable dashboards to analyze and visualize any kind of data • A web developer and researcher with ten years of experience whose main research line is the development of technological ecosystems HCI 2019 • A researcher with more than ten years of experience in multimodal human-computer interaction • A developer and researcher with more than ten years of experience focused on data visualization in different fields The heuristic evaluation was carried out by four experts, two women and two men between 25 to 39 years old
  13. 13. Methodology: Heuristic evaluation (II) Instrumentation The same template was provided to the experts in order to get the reports to perform the heuristic evaluation HCI 2019 Heuristic Points (0-10) Comments Visibility of system status Match between system and the real world User control and freedom Consistency and standards Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from errors Help and documentation
  14. 14. Methodology: Heuristic evaluation (III) Study design and data collection All experts (with the facilitator role) navigated through the application several times observing all the screens and detecting the usability problems HCI 2019
  15. 15. Heuristic evaluation (I) The heuristic that presents the largest number of usability problems was HR4 (Consistency and standards) HCI 2019
  16. 16. Heuristic evaluation (II) There are also low values associated to help users, HR9 (Help users recognize, diagnose, and recover from errors) and HR10 (Help and documentation) The WYRED Platform has a help section with videos in English, but the contents should be provided with subtitles E3 indicates that the information is not accessible on demand in text format and there is not a FAQ HCI 2019
  17. 17. Heuristic evaluation – Problems detected • “When searching for public projects or communities, there is no straightforward manner of knowing which filter is currently applied” by E3. Associated to HR1. • “The layout of the search tools is difficult to understand. The search button appears before the search field” by E2. Associated to HR4. • “When searching for public communities, there are no autocompletion options” by E3. Associated to HR5. • “It was difficult to find out how to add several tags to the different search bars shown in the Platform. The user has to manually input a comma after the first keyword to be able to insert the second one” by E1. Associated to HR7.HCI 2019
  18. 18. Methodology: UX (I) Participants A total of 30 facilitators were invited to participate in the study and 28 answered the CSUQ Most of the participants are female (64.3%), 21.4% are male, 3.6% selected another gender, and 10.7% decided not to provide this information HCI 2019 A subset of facilitators was invited to answer the CSUQ
  19. 19. Methodology: UX (II) Participants Most of the participants have used several browsers to access the Platform HCI 2019
  20. 20. Methodology: UX (III) Instrumentation Computer System Usability Questionnaire (CSUQ) version 3: 16 items rated on a 1 to 7 Likert scale A set of demographic and contextual variables were also collected by the instrument HCI 2019
  21. 21. Methodology: UX (IV) Study design and data collection Regarding CSUQ, the participants involved in the study have experience using the Platform in real scenarios (not scenario-based usability test) HCI 2019
  22. 22. User experience (I) The mean and a 95% confidence interval were computed for each CSUQ item answer and compared to the PSSUQ norms HCI 2019
  23. 23. User experience (II) To gain better understanding of these overall scores, each item mean value was also individually analyzed HCI 2019
  24. 24. User experience (III) The figure shows the distribution of values of the CSUQ Likert scale. Through this representation it is possible to understand how the participants rated each item with less granularity. HCI 2019
  25. 25. User experience (IV) Correlation among contextual variables with the CSUQ items; yellow cells represent no correlation and more blue cells represent higher correlation HCI 2019
  26. 26. User experience (V) Younger people and people using the Platform through iOS seem to rate better the Platform HCI 2019
  27. 27. Discussion (I) According to Khajouei, Ameri and Jahani, in some areas, the perception of evaluators in using this method is not consistent with the users' experience with a system HCI 2019 The results of the heuristic evaluation are useful to improve the WYRED Platform, but the results from the CSUQ questionnaire are required to get a full overview of the usability of the Platform The usability issues detected by each expert are a small number of the existed usability problem
  28. 28. Discussion (II) Regarding the user experience, one of the identified potential issues of the CSUQ questionnaire is the positive nature of the items’ statements  Acquiescence bias HCI 2019 The obtained results, compared to the PSSUQ norms, show acceptable values, although lower than the references The PSSUQ norms provide a useful reference to compare the obtained scores, and to reach insights about potential weaknesses of the tested Platform
  29. 29. Discussion (III) The information quality items are the best-rated items HCI 2019 The weakest values of the CSUQ questionnaire are related to items 14 (“I like using the interface of this application”) and 15 (“This application has all the functions and capabilities I expect it to have”). The potential cause of these lower ratings is the fact that the Platform is still in development (lack of capabilities and maturity) No strong correlations were found regarding contextual variables: only the birth date variable showed that younger people tend to rate better the Platform
  30. 30. Conclusions • A usability evaluation of the WYRED Platform has been executed through two different methods: a heuristic evaluation and usability evaluation (CSUQ questionnaire) • Having these two points of view of the Platform’s usability helped to identify weaknesses of the WYRED interface and interaction methods • The results provide relevant information to develop a new version of the WYRED Platform in order to solve the different identified problems and provide a stable final version HCI 2019
  31. 31. Conclusions It would be interesting to assess this new version through a user testing and a second heuristic evaluation HCI 2019
  32. 32. Acknowledgments This research was supported by the Spanish Ministry of Education, Culture and Sport under a FPU fellowship (FPU17/03276). With the support of the EU Horizon 2020 Programme in its “Europe in a changing world – inclusive, innovative and reflective Societies (HORIZON 2020: REV- INEQUAL-10-2016: Multi-stakeholder Platform for enhancing youth digital opportunities)” Call. Project WYRED (netWorked Youth Research for Empowerment in the Digital society) (Grant agreement No 727066). The sole responsibility for the content of this webpage lies with the authors. It does not necessarily reflect the opinion of the European Union. The European Commission is not responsible for any use that may be made of the information contained therein. HCI 2019
  33. 33. Study of the usability of the WYRED Ecosystem using heuristic evaluation Francisco J. García-Peñalvo, Andrea Vázquez-Ingelmo, Alicia García-Holgado GRIAL Research Group, Computer Sciences Department, Research Institute for Educational Sciences, University of Salamanca, Salamanca, Spain,,