ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing
•Download as PPTX, PDF•
3 likes•1,201 views
The ECOVAL framework provides guidelines and the GAMBIT tool to increase ecological validity in usability testing. ECOVAL defines attributes like objects, tasks, and environment to objectively measure ecological validity. GAMBIT allows producing low-cost, realistic prototypes for remote usability testing. A case study showed GAMBIT improved ecological validity over paper testing, enabling earlier usability problem detection and increased productivity. The authors recommend using ECOVAL and GAMBIT to improve usability evaluations and realize cost savings and other benefits from more ecologically valid testing.
Recommended
Recommended
More Related Content
Similar to ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing
Similar to ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing (20)
More from Jean Vanderdonckt
More from Jean Vanderdonckt (20)
Recently uploaded
Recently uploaded (20)
ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing
- 1. Suzanne Kieffer ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing Jean VanderdoncktUgo Braga Sangiorgi Université catholique de Louvain Louvain School of Management Louvain-la-Neuve, Belgium
- 4. Internal validity Was the study well done? External validity Can we generalize the findings to particular persons, settings and times? Ecological validity To which extent can we generalize the findings?
- 6. User interface design Ecological validity Overall validity Conclusions of usability evaluations
- 7. What’s the problem with ecological validity? No formal definition that enables To objectively quantify the ecological validity of experimental designs To capture the critical attributes of the real-world environment
- 8. LOGICAL IDITYVALECO ECOVAL A framework for increasing the ecological validity in usability testing
- 9. Outline The ECOVAL framework ECOVAL Guidelines GAMBIT Case study Operability and usefulness of ECOVAL ROI of increased ecological validity Recommendations
- 12. The guidelines
- 13. No objects
- 14. Mock objects
- 15. Real objects
- 16. Only verbalized
- 18. For real
- 19. GAMBIT – the supporting tool Production of prototypes Easy Quick Cost-effective Real user interface performances Interaction Navigation Responsiveness
- 20. GAMBIT – how to 1 Produce screens
- 21. GAMBIT – how to 2 Create behaviour
- 22. GAMBIT – how to 3 Conduct experiment
- 23. GAMBIT – how to 4 Download log file
- 24. • Q1: To which extent is ECOVAL applicable for controlling the ecological validity? • Q2: What are the possible outcomes/benefits of increased ecological validity ?
- 25. Case study
- 26. Hot-Dip Galvanizing on Continuous Lines
- 28. Monitoring report Task Control Result Task Control Result Artefact: paper checklist Section > Subsection Function – Name - Date – Shift Section > Subsection 300 items a day
- 29. Monitoring report Task Control Result Task Control Result Artefact: paper checklist Section > Subsection Function – Name - Date – Shift Section > Subsection
- 32. Usability evaluation Paper versus GAMBIT prototype
- 33. Ecological validity Ratio Paper-to-Gambit for ecological validity of 1:1.17
- 34. Balsamiq mockups
- 35. Experimentations Paper GAMBIT 5 key users x 2 10 participants (30’) 18 participants (1 hour) 5 key users x 3 + 3 user representatives V1 6 participants V2 12 participants
- 36. New version of the prototype
- 37. Apparatus
- 39. Early detection of usability problems Increased user efficiencies Increased user satisfaction
- 42. Paper GAMBIT Ecological validity 1 1.17 Data collection Manual & face-to-face Difficult & tedious Unstructured process Obtrusive Automated & remote Comfortable Structured process Unobtrusive Outcomes Compliant with the literature Early detection of usability problems Increased user productivity Decreased late design changes Increased organizational efficiency
- 43. Recommendations
- 44. Take away {ECOVAL, Guidelines, GAMBIT} Case study Operability Usefulness ROI Recommendations
Editor's Notes
- Good morning everyone! My name is Suzanne Kieffer, I’m from UCL, BELGIUM, and I’m very pleased to attend my first HICSS conference. During the next 20 minutes I will present ECOVAL, a framework built for increasing the ecological validity of experimental designs. In particular, I will focus on the applicability of the concept of increased ecological validity in usability testing.
- First of all, what is ecological validity? In experimental research, the term ecological validity refers to the correspondences between an experimental design and its related natural setting. Especially in terms of users’ environment, stimuli and tasks.
- The ecological validity of a study is high if it takes place in the participants’ natural setting or if the participants don’t know they are taking part in a research study. On the contrary, it is low if the experimental design differs from the situations, stimuli and tasks experienced in a natural setting.
- How does ecological validity relate to other types of validity? The overall validity of an experiment can be broken down into three types of validity: INTERNAL, EXTERNAL and ECOLOGICAL validity.
- And the overall validity can be compared to a puzzle. If one piece is missing or has been damaged, that will negatively impact the final result. For example, a study may have been adequately carried out, with a HIGH INTERNAL VALIDITY (instrumentation, selection, experimenter). However, if the experimental design did not sample the variables from the user’s environment, the study may suffer from problems with ecological validity. And consequently, the generalizability of the findings may be reduced. And this is exactly the reason why ecological validity is so important to anyone intending to generalize the conclusions from the particular circumstances of an experiment to wider ecological situations.
- Why is ecological validity important in HCI as well? BECAUSE it will drive user interface design By impacting the OVERALL validity of experimental studies And therefore by impacting the conclusions, outcomes and findings of usability studies
- As more or less ecologically valid
- That’s why we propose ECOVAL A framework for defining, controlling, increasing the ecological validity in usability testing.
- Which brings me to the outline of me presentation
- The ECOVAL framework builds on the combination of Schmukler’s three dimensions of ecological validity and Shackel’s components in human-machine systems. The signals and objects dimensions both refer to the stimuli used and describe how the real world environment is captured in the experimental design. Signals : sounds, smells, colors, lighting, heat, cold, dust, and pressure Objects : buildings, equipment, furniture, tools, and artifacts Test medium and user interface dimensions both refer to the computer system being tested. Task and behavior dimensions both refer to what is required of the participants.
- Similarly to wireframes, Gambit enables the EASY, QUICK and COST-EFFECTIVE production of prototypes With the advantage of performing as a real product user interface in terms of INTERACTION, NAVIGATION and RESPONSIVENESS GAMBIT displays the screens of the user interface, enables the navigation between them, and it can run on multiple devices: smartphone, tablet, laptop, monitor, TV, etc.
- The first step is the production of screens. Here they were created with Balsamiq, but they could have been created by simply talking a picture or scanning a sketch, or even by producing screenshots from a higher-fidelity prototype
- The second step is the creation of the behavior of the prototype, by simply interlinking specific areas of images
- The third step is to conduct the experiment Again, prototypes can run on multiple devices
- And the last step is the download of the log file and the analysis of data
- Framework + companion guidelines + supporting tool So the research questions are:
- We conducted a case study in a company whose core-business is hot-dip galvanizing to produce steel sheet for the automobile industry
- What’s important to know about Hot-Dip galvanizing is that the production line runs continuously through a cleaner, an annealing furnace, and then into a molten zinc bath.
- And that it requires the collaboration of 7 co-workers: one team leader, one quality technician, five operators.
- USER TASK = COMPLETION of the MONITORING REPORT FILL-IN a checklist on a daily basis 300 items a day
- USER TASK = COMPLETION of the MONITORING REPORT FILL-IN a checklist on a daily basis 300 items a day
- We identified 5 key users: the team leader, and four operators.
- And this is the timeline we followed: We conducted user profile, task analysis, usability goal setting, and screen design
- What’s important here is that that the same user interface was evaluated in both the paper and the GAMBIT evaluations And that the ecological validity was higher with the GAMBIT prototype than with the paper prototype with a ratio…
- Problem new version of the prototype
- What we believe enabled to identify the usability problems and correct them, is the fact that the GAMBIT prototype was more realistic/ecological in terms of test medium and task. The intended device The task The use of mock objects lead to the user walking from one mock object to another
- I will now present the most important results and discuss them
- First of all, Increased ECOVAL enabled the early detection of usability problems By doing so, it enabled to correct these usability problems Which led to increased user efficiencies and increased user satisfaction
- Secondly, it led to savings in terms of both usability engineers and users productivity To determine the costs we need to know the number of hours spent with the GAMBIT experiment
- The increased productivity was calculated as the operator-hours saved per year (375 hours) multiplied by the operator wage (80). The decreased late design changes were calculated as the costs of late changes minus the costs of early changes.
- Yet, there are a number of other questions beyond the scope of this paper that need to be addressed in near future. What correspondences can be made between behaviors in an experiment and behaviors in the real environment? Or how to transfer the natural/intended behaviors in an experimental context? A top-down ecological approach to tackle that relies on synthetic signals [1, 17]. Real signals are first recorded in the field. Then, participants’ behavior, efficiency and satisfaction towards the synthetic versions of these signals are studied in the lab. Relevant signals in the case study are alarms, phone calls, and GA line stop. How do signals and behavior dimensions exactly impact external validity? Finally, this raises further questions. Is there a threshold in ecological validity that, once reached, ensures external validity?