Usability Assessment of a Context-Aware and Personality-Based Mobile Recommender System

1. Usability Assessment of a Context-Aware and Personality-Based Mobile Recommender System Matthias Braunhofer, Mehdi Elahi and Francesco Ricci ! Free University of Bozen - Bolzano Piazza Domenicani 3, 39100 Bolzano, Italy {mbraunhofer,mehdi.elahi,fricci}@unibz.it EC-Web - September 2014, Munich, Germany

2. EC-Web - September 2014, Munich, Germany Outline 2 • Context-Aware Recommender Systems and their Challenges • Related Works • STS (South Tyrol Suggests) • Usability Assessment and Results • Conclusions, Lessons Learned and Future Work

3. EC-Web - September 2014, Munich, Germany Outline 2 • Context-Aware Recommender Systems and their Challenges • Related Works • STS (South Tyrol Suggests) • Usability Assessment and Results • Conclusions, Lessons Learned and

4. Context is Essential • Main idea: users can experience items differently depending on the current contextual situation (e.g., season, weather, temperature, mood) • Example: EC-Web - September 2014, Munich, Germany 3

5. Context-Aware Recommender Systems (CARSs) • CARS extend Recommender Systems (RSs) beyond users and items to the contexts in which items are experienced by users • Rating prediction function is: R: Users × Items × Context → Ratings EC-Web - September 2014, Munich, Germany 4 3 ? 4 2 5 4 ? 3 4 1 ? 1 2 5 ? 3 3 ? 5 2 5 ? 3 5 ? 5 4 5 4 ? 3 5

6. Challenges for CARSs • Identification of contextual factors (e.g., weather) that are worth considering when generating recommendations • Acquisition of a representative set of contextually-tagged ratings • Development of a predictive model for predicting the user’s ratings for items under various contextual situations • Design and implementation of a human-computer interaction (HCI) layer on top of the predictive model EC-Web - September 2014, Munich, Germany 5

7. Challenges for CARSs • Identification of contextual factors (e.g., weather) that are worth considering when generating recommendations • Acquisition of a representative set of contextually-tagged ratings • Development of a predictive model for predicting the user’s ratings for items under various contextual situations • Design and implementation of a human-computer interaction (HCI) layer on top of the predictive model EC-Web - September 2014, Munich, Germany 5 Focus of this research

8. • Context-Aware Recommender Systems and their Challenges EC-Web - September 2014, Munich, Germany Outline 6 • Related Works • STS (South Tyrol Suggests) • Usability Assessment and Results • Conclusions and Future Work

9. HCI Perspective on RSs • Effectiveness of a RS depends not only on the underlying prediction algorithm but also on the proper design of the human-computer interaction (Swearingen and Sinha, 2001) • User’s interaction with RSs: EC-Web - September 2014, Munich, Germany 7 Recommendation Algorithms Input from user (ratings) Output to user (recommendations) • No. of ratings • Time to register • Details about item to be rated • Type of rating scale • … • No. of good recs. • No. of new, unknown recs. • Information about each rec. • Confidence in prediction • Is system logic transparent? • …

10. Usability Assessment of RSs (1/2) • Evaluation of the usability of a context-aware and group-based restaurant RS using the System Usability Scale (SUS) (Park et al., 2008) • The SUS is a 10-item instrument to measure the user’s perceived usability of a system (Brooke, 1996) • Major finding: the SUS score with 13 test users was 70.58, a rating between “ok” and “good”, and corresponding to a “C” grade, which is an acceptable level of usability EC-Web - September 2014, Munich, Germany 8

11. Usability Assessment of RSs (2/2) • Usage of eye tracking, clickstream analysis and SUS to determine the usability of a constraint-based travel advisory system called VIBE (Jannach et al., 2009) • Major findings: • Average SUS score was 81.5, a rating between “good” and “excellent” and corresponding to a “B” grade, which is a very high level of usability • Identification of several usability issues: • Inadequate positioning of VIBE on the online portal • Too many recommendation results • Too little information displayed in the recommendation results EC-Web - September 2014, Munich, Germany 9

12. • Context-Aware Recommender Systems and their Challenges EC-Web - September 2014, Munich, Germany Outline 10 • Related Works • STS (South Tyrol Suggests) • Usability Assessment and Results • Conclusions, Lessons Learned and F

13. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Welcome screen

14. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Registration screen

15. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Personality questionnaire

16. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Questionnaire results

17. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Active learning

18. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Suggestions screen

19. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Context settings

20. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Details screen

21. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Rating dialog

22. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Routing screen

23. Interaction with the STS System EC-Web - September 2014, Munich, Germany 11 Bookmarked items screen

24. Software Architecture and Implementation Apache Tomcat Server EC-Web - September 2014, Munich, Germany 12 Android Client Spring Dispatcher Servlet Spring Controllers Service / Application Layer JPA Entities Hibernate Objects managed by Spring IoC Container Database JSON HTTP Web Services

29. Recommendations Computation • Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations • Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores) • Advantage: allows to model the user preferences even if no feedback is available Σ ) EC-Web - September 2014, Munich, Germany 13 kΣ ˆ ruic1,...,ck = i + bu + bicj j=1 + qi T ⋅(pu + ya a∈A(u) ī average rating for item i bu baseline for user u bicj baseline for item i and contextual condition cj qi latent factor vector of item i pu latent factor vector of user u A(u) set of user attributes ya latent factor vector of user attribute a

35. Recommendations Computation • Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF) (Baltrunas et al., 2011) to provide users with context-aware recommendations • Key difference: we incorporate additional user attributes (i.e., age, gender and Big Five personality trait scores) • Advantage: allows to model the user preferences even if no feedback is available new Σ ) EC-Web - September 2014, Munich, Germany 13 kΣ ˆ ruic1,...,ck = i + bu + bicj j=1 + qi T ⋅(pu + ya a∈A(u) ī average rating for item i bu baseline for user u bicj baseline for item i and contextual condition cj qi latent factor vector of item i pu latent factor vector of user u A(u) set of user attributes ya latent factor vector of user attribute a

38. Experimental Methodology • Live user study where we compared our system (STS) with a variant (STS-S) that has the same graphical UI but does not use the weather context when generating recommendations • We have designed a specific user task and used a questionnaire for assessing the perceived recommendation quality (Knijnenburg et al., 2012) and system usability with the System Usability Scale (SUS) (Brooke, 1996) • 30 subjects that were randomly divided in two equal groups assigned to STS and STS-S (15 each) EC-Web - September 2014, Munich, Germany 15

39. EC-Web - September 2014, Munich, Germany User Task • Users were supposed to: • have an afternoon off and to look for attractions / events in South Tyrol • consider the contextual conditions relevant for them and to specify them in the system settings • browse the attractions / events sections and check whether they could find something interesting for them • browse the system suggestions (recommendations), and select and bookmark the one that they believed fits their preferences • fill out a survey on recommendation quality and system usability 16

40. Results (1/3) Box-and-whisker plot of the SUS points for each statement given by all users EC-Web - September 2014, Munich, Germany 17 S1 I think that I would like to use this system frequently. S2 I found the system unnecessarily complex. S3 I thought the system was easy to use. S4 I think that I would need the support of a technical person to be able to use this system. S5 I found the various functions in this system were well integrated S6 I thought there was too much inconsistency in this system. S7 I would imagine that most people would learn to use this system very quickly. S8 I found the system very cumbersome to use. S9 I felt very confident using the system. S10 I needed to learn a lot of things before I could get going with this system.

41. SUS scores for all users Benchmark Average 1 2 3 4 5 6 7 8 9 10 11 12 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EC-Web - September 2014, Munich, Germany 90 85 80 75 SUS score 50 70 65 60 55 Users Results (2/3) 18

42. EC-Web - September 2014, Munich, Germany Results (3/3) Comparison of the SUS scores for STS and STS-S users 19 Statement STS STS-S p-value S1 I think that I would like to use this system frequently. 3.0 3.2 0.27 S2 I found the system unnecessarily complex. 3.2 3.5 0.16 S3 I thought the system was easy to use. 3.1 2.8 0.18 S4 I think that I would need the support of a technical person to be able to use this system. 3.3 3.4 0.40 S5 I found the various functions in this system were well integrated 3.1 2.8 0.14 S6 I thought there was too much inconsistency in this system. 3.2 2.8 0.08 S7 I would imagine that most people would learn to use this system very quickly. 2.8 3.0 0.25 S8 I found the system very cumbersome to use. 3.4 3.1 0.19 S9 I felt very confident using the system. 2.7 2.8 0.40 S10 I needed to learn a lot of things before I could get going with this system. 3.4 3.1 0.11 Overall SUS 78.8 77.0 0.19

43. Corrective Actions Based on the Results (1/3) • Five-Item Personality Inventory (FIPI) • We replaced the Ten-Item Personality Inventory (TIPI) with the Five-Item Personality Inventory (FIPI), which is less time-consuming and still provides sufficient personality data. • Built-in help • Users can click the “?” icon next to each questionnaire question to access on-screen help with term definitions. EC-Web - September 2014, Munich, Germany 20 …Before …After

48. Corrective Actions Based on the Results (2/3) • In-app notifications • Instead of forcing users to go through the lengthy AL process during registration, we give them freedom to decide when to initiate it through in-app notifications within the POI suggestions screen. • User profile page • We implemented a new user profile page, making it easier to access and change context settings, basic user information, personality information, etc. EC-Web - September 2014, Munich, Germany 21

55. Corrective Actions Based on the Results (3/3) • Many other minor UI improvements • Revised the contextual factors and contextual conditions • Improved the UI for displaying personality questionnaire results • Cleaned up the POI details screen EC-Web - September 2014, Munich, Germany 22 Before After Before After

60. Conclusions • Novel and highly usable mobile CARS called STS (South Tyrol Suggests) that offers various innovative features • Learns users’ preferences not only using their past ratings, but also exploiting their personality • Uses personality to actively acquire ratings for POIs the user has likely experienced, and to produce more accurate POI recommendations • Live user study to test the usability of STS • Results confirm high usability of the proposed system • Allowed to uncover and resolve some usability issues, such as moderate confidence in the system and poor integration of some features EC-Web - September 2014, Munich, Germany 24

61. Lessons Learned • Only ask users for the minimum required information • The more information you ask of users, the less likely they will provide it • Make the system as simple as possible to use • Keep the system as simple as possible and provide useful on-screen help or tutorials to instruct users on how to get things done • Give users control over the system • Instead of telling users how to use the user interface, give them the ability to control where they go and what they do. Moreover, always ensure that the user knows what things are and what they will do EC-Web - September 2014, Munich, Germany 25

62. EC-Web - September 2014, Munich, Germany Future Work • Evaluate the usability of the revised user interface • Provide users with proactive recommendations and rating requests • Consider additional important contextual factors in the recommendation process (e.g., parking availability, traffic conditions) • Improve explanations to make the recommendation process more transparent to users 26

63. EC-Web - September 2014, Munich, Germany Questions? Thank you.