In this paper we present STS (South Tyrol Suggests), a context-aware mobile recommender system for places of interest (POIs) that integrates some innovative components, including: a personality questionnaire, i.e., a brief and entertaining questionnaire used by the system to learn users’ personality; an active learning module that acquires ratings-in-context for POIs that users are likely to have experienced; and a matrix factorization based recommendation module that leverages the personality information and several contextual factors in order to generate more relevant recommendations. Adopting a system oriented perspective, we describe the assessment of the combination of the implemented components. We focus on usability aspects and report the end-user assessment of STS. It was obtained from a controlled live user study as well as from the log data produced by a larger sample of users that have freely downloaded and tried STS through Google Play Store. The result of the assessment showed that the overall usability of the system falls between “good” and “excellent”, it helped us to identify potential problems and it provided valuable indications for future system improvement.

1. Usability Assessment of a Context-Aware and
Personality-Based Mobile Recommender
System
Matthias Braunhofer, Mehdi Elahi and Francesco Ricci
!
Free University of Bozen - Bolzano
Piazza Domenicani 3, 39100 Bolzano, Italy
{mbraunhofer,mehdi.elahi,fricci}@unibz.it
EC-Web - September 2014, Munich, Germany

2. EC-Web - September 2014, Munich, Germany
Outline
2
• Context-Aware Recommender Systems and their Challenges
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and Future Work

3. EC-Web - September 2014, Munich, Germany
Outline
2
• Context-Aware Recommender Systems and their Challenges
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and

4. Context is Essential
• Main idea: users can experience items differently depending on the current
contextual situation (e.g., season, weather, temperature, mood)
• Example:
EC-Web - September 2014, Munich, Germany
3

5. Context-Aware Recommender Systems
(CARSs)
• CARS extend Recommender Systems (RSs) beyond users and items to the
contexts in which items are experienced by users
• Rating prediction function is: R: Users × Items × Context → Ratings
EC-Web - September 2014, Munich, Germany
4
3 ? 4
2 5 4
? 3 4
1 ? 1
2 5
? 3
3 ? 5
2 5
? 3
5 ? 5
4 5 4
? 3 5

6. Challenges for CARSs
• Identification of contextual factors (e.g., weather) that are worth considering
when generating recommendations
• Acquisition of a representative set of contextually-tagged ratings
• Development of a predictive model for predicting the user’s ratings for items
under various contextual situations
• Design and implementation of a human-computer interaction (HCI) layer
on top of the predictive model
EC-Web - September 2014, Munich, Germany
5

7. Challenges for CARSs
• Identification of contextual factors (e.g., weather) that are worth considering
when generating recommendations
• Acquisition of a representative set of contextually-tagged ratings
• Development of a predictive model for predicting the user’s ratings for items
under various contextual situations
• Design and implementation of a human-computer interaction (HCI) layer
on top of the predictive model
EC-Web - September 2014, Munich, Germany
5
Focus of this
research

8. • Context-Aware Recommender Systems and their Challenges
EC-Web - September 2014, Munich, Germany
Outline
6
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions and Future Work

9. HCI Perspective on RSs
• Effectiveness of a RS depends not only on the underlying prediction
algorithm but also on the proper design of the human-computer
interaction (Swearingen and Sinha, 2001)
• User’s interaction with RSs:
EC-Web - September 2014, Munich, Germany
7
Recommendation
Algorithms
Input from user
(ratings)
Output to user
(recommendations)
• No. of ratings
• Time to register
• Details about item
to be rated
• Type of rating scale
• …
• No. of good recs.
• No. of new, unknown recs.
• Information about each rec.
• Confidence in prediction
• Is system logic transparent?
• …

10. Usability Assessment of RSs (1/2)
• Evaluation of the usability of a context-aware and group-based
restaurant RS using the System Usability Scale (SUS) (Park et al., 2008)
• The SUS is a 10-item instrument to measure the user’s perceived usability
of a system (Brooke, 1996)
• Major finding: the SUS score with 13 test users was 70.58, a rating between
“ok” and “good”, and corresponding to a “C” grade, which is an acceptable
level of usability
EC-Web - September 2014, Munich, Germany
8

11. Usability Assessment of RSs (2/2)
• Usage of eye tracking, clickstream analysis and SUS to determine the
usability of a constraint-based travel advisory system called VIBE (Jannach
et al., 2009)
• Major findings:
• Average SUS score was 81.5, a rating between “good” and “excellent” and
corresponding to a “B” grade, which is a very high level of usability
• Identification of several usability issues:
• Inadequate positioning of VIBE on the online portal
• Too many recommendation results
• Too little information displayed in the recommendation results
EC-Web - September 2014, Munich, Germany
9

12. • Context-Aware Recommender Systems and their Challenges
EC-Web - September 2014, Munich, Germany
Outline
10
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and F

13. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Welcome screen

14. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Registration screen

15. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Personality questionnaire

16. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Questionnaire results

17. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Active learning

18. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Suggestions screen

19. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Context settings

20. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Details screen

21. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Rating dialog

22. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Routing screen

23. Interaction with the STS System
EC-Web - September 2014, Munich, Germany
11
Bookmarked items screen

24. Software Architecture and Implementation
Apache Tomcat Server
EC-Web - September 2014, Munich, Germany
12
Android Client
Spring Dispatcher
Servlet Spring Controllers
Service /
Application Layer
JPA Entities
Hibernate
Objects managed by Spring IoC Container
Database
JSON
HTTP
Web Services

25. Software Architecture and Implementation
Apache Tomcat Server
EC-Web - September 2014, Munich, Germany
12
Android Client
Spring Dispatcher
Servlet Spring Controllers
Service /
Application Layer
JPA Entities
Hibernate
Objects managed by Spring IoC Container
Database
JSON
HTTP
Web Services

26. Software Architecture and Implementation
Apache Tomcat Server
EC-Web - September 2014, Munich, Germany
12
Android Client
Spring Dispatcher
Servlet Spring Controllers
Service /
Application Layer
JPA Entities
Hibernate
Objects managed by Spring IoC Container
Database
JSON
HTTP
Web Services

27. Software Architecture and Implementation
Apache Tomcat Server
EC-Web - September 2014, Munich, Germany
12
Android Client
Spring Dispatcher
Servlet Spring Controllers
Service /
Application Layer
JPA Entities
Hibernate
Objects managed by Spring IoC Container
Database
JSON
HTTP
Web Services

28. Software Architecture and Implementation
Apache Tomcat Server
EC-Web - September 2014, Munich, Germany
12
Android Client
Spring Dispatcher
Servlet Spring Controllers
Service /
Application Layer
JPA Entities
Hibernate
Objects managed by Spring IoC Container
Database
JSON
HTTP
Web Services

29. Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF)
(Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender
and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is
available
Σ )
EC-Web - September 2014, Munich, Germany
13
kΣ
ˆ ruic1,...,ck = i + bu + bicj
j=1
+ qi
T ⋅(pu + ya
a∈A(u)
ī average rating for item i
bu baseline for user u
bicj baseline for item i and contextual condition cj
qi latent factor vector of item i
pu latent factor vector of user u
A(u) set of user attributes
ya latent factor vector of user attribute a

30. Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF)
(Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender
and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is
available
Σ )
EC-Web - September 2014, Munich, Germany
13
kΣ
ˆ ruic1,...,ck = i + bu + bicj
j=1
+ qi
T ⋅(pu + ya
a∈A(u)
ī average rating for item i
bu baseline for user u
bicj baseline for item i and contextual condition cj
qi latent factor vector of item i
pu latent factor vector of user u
A(u) set of user attributes
ya latent factor vector of user attribute a

31. Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF)
(Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender
and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is
available
Σ )
EC-Web - September 2014, Munich, Germany
13
kΣ
ˆ ruic1,...,ck = i + bu + bicj
j=1
+ qi
T ⋅(pu + ya
a∈A(u)
ī average rating for item i
bu baseline for user u
bicj baseline for item i and contextual condition cj
qi latent factor vector of item i
pu latent factor vector of user u
A(u) set of user attributes
ya latent factor vector of user attribute a

32. Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF)
(Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender
and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is
available
Σ )
EC-Web - September 2014, Munich, Germany
13
kΣ
ˆ ruic1,...,ck = i + bu + bicj
j=1
+ qi
T ⋅(pu + ya
a∈A(u)
ī average rating for item i
bu baseline for user u
bicj baseline for item i and contextual condition cj
qi latent factor vector of item i
pu latent factor vector of user u
A(u) set of user attributes
ya latent factor vector of user attribute a

33. Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF)
(Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender
and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is
available
Σ )
EC-Web - September 2014, Munich, Germany
13
kΣ
ˆ ruic1,...,ck = i + bu + bicj
j=1
+ qi
T ⋅(pu + ya
a∈A(u)
ī average rating for item i
bu baseline for user u
bicj baseline for item i and contextual condition cj
qi latent factor vector of item i
pu latent factor vector of user u
A(u) set of user attributes
ya latent factor vector of user attribute a

34. Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF)
(Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender
and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is
available
Σ )
EC-Web - September 2014, Munich, Germany
13
kΣ
ˆ ruic1,...,ck = i + bu + bicj
j=1
+ qi
T ⋅(pu + ya
a∈A(u)
ī average rating for item i
bu baseline for user u
bicj baseline for item i and contextual condition cj
qi latent factor vector of item i
pu latent factor vector of user u
A(u) set of user attributes
ya latent factor vector of user attribute a

35. Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF)
(Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender
and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is
available
new
Σ )
EC-Web - September 2014, Munich, Germany
13
kΣ
ˆ ruic1,...,ck = i + bu + bicj
j=1
+ qi
T ⋅(pu + ya
a∈A(u)
ī average rating for item i
bu baseline for user u
bicj baseline for item i and contextual condition cj
qi latent factor vector of item i
pu latent factor vector of user u
A(u) set of user attributes
ya latent factor vector of user attribute a

36. Recommendations Computation
• Main idea: use a model similar to Context-Aware Matrix Factorization (CAMF)
(Baltrunas et al., 2011) to provide users with context-aware recommendations
• Key difference: we incorporate additional user attributes (i.e., age, gender
and Big Five personality trait scores)
• Advantage: allows to model the user preferences even if no feedback is
available
Σ )
EC-Web - September 2014, Munich, Germany
13
kΣ
ˆ ruic1,...,ck = i + bu + bicj
j=1
+ qi
T ⋅(pu + ya
a∈A(u)
ī average rating for item i
bu baseline for user u
bicj baseline for item i and contextual condition cj
qi latent factor vector of item i
pu latent factor vector of user u
A(u) set of user attributes
ya latent factor vector of user attribute a

37. EC-Web - September 2014, Munich, Germany
Outline
14
• Context-Aware Recommender Systems and their Challenges
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and Future Work

38. Experimental Methodology
• Live user study where we compared our system (STS) with a variant (STS-S)
that has the same graphical UI but does not use the weather context when
generating recommendations
• We have designed a specific user task and used a questionnaire for
assessing the perceived recommendation quality (Knijnenburg et al., 2012)
and system usability with the System Usability Scale (SUS) (Brooke, 1996)
• 30 subjects that were randomly divided in two equal groups assigned to
STS and STS-S (15 each)
EC-Web - September 2014, Munich, Germany
15

39. EC-Web - September 2014, Munich, Germany
User Task
• Users were supposed to:
• have an afternoon off and to look for attractions / events in South Tyrol
• consider the contextual conditions relevant for them and to specify them
in the system settings
• browse the attractions / events sections and check whether they could
find something interesting for them
• browse the system suggestions (recommendations), and select and
bookmark the one that they believed fits their preferences
• fill out a survey on recommendation quality and system usability
16

40. Results (1/3)
Box-and-whisker plot of the SUS points for each statement given by all
users
EC-Web - September 2014, Munich, Germany
17
S1 I think that I would like to use this system
frequently.
S2 I found the system unnecessarily complex.
S3 I thought the system was easy to use.
S4 I think that I would need the support of a
technical person to be able to use this
system.
S5 I found the various functions in this system
were well integrated
S6 I thought there was too much
inconsistency in this system.
S7 I would imagine that most people would
learn to use this system very quickly.
S8 I found the system very cumbersome to
use.
S9 I felt very confident using the system.
S10 I needed to learn a lot of things before I
could get going with this system.

41. SUS scores for all users
Benchmark Average
1 2 3 4 5 6 7 8 9 10 11 12 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
EC-Web - September 2014, Munich, Germany
90
85
80
75
SUS score 50
70
65
60
55
Users
Results (2/3)
18

42. EC-Web - September 2014, Munich, Germany
Results (3/3)
Comparison of the SUS scores for STS and STS-S users
19
Statement STS STS-S p-value
S1 I think that I would like to use this system frequently. 3.0 3.2 0.27
S2 I found the system unnecessarily complex. 3.2 3.5 0.16
S3 I thought the system was easy to use. 3.1 2.8 0.18
S4 I think that I would need the support of a technical person to
be able to use this system.
3.3 3.4 0.40
S5 I found the various functions in this system were well integrated 3.1 2.8 0.14
S6 I thought there was too much inconsistency in this
system.
3.2 2.8 0.08
S7 I would imagine that most people would learn to use this
system very quickly.
2.8 3.0 0.25
S8 I found the system very cumbersome to use. 3.4 3.1 0.19
S9 I felt very confident using the system. 2.7 2.8 0.40
S10 I needed to learn a lot of things before I could get going
with this system.
3.4 3.1 0.11
Overall SUS 78.8 77.0 0.19

43. Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality
Inventory (TIPI) with the Five-Item
Personality Inventory (FIPI), which is less
time-consuming and still provides
sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each
questionnaire question to access on-screen
help with term definitions.
EC-Web - September 2014, Munich, Germany
20
…Before
…After

44. Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality
Inventory (TIPI) with the Five-Item
Personality Inventory (FIPI), which is less
time-consuming and still provides
sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each
questionnaire question to access on-screen
help with term definitions.
EC-Web - September 2014, Munich, Germany
20
…Before
…After

45. Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality
Inventory (TIPI) with the Five-Item
Personality Inventory (FIPI), which is less
time-consuming and still provides
sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each
questionnaire question to access on-screen
help with term definitions.
EC-Web - September 2014, Munich, Germany
20
…Before
…After

46. Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality
Inventory (TIPI) with the Five-Item
Personality Inventory (FIPI), which is less
time-consuming and still provides
sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each
questionnaire question to access on-screen
help with term definitions.
EC-Web - September 2014, Munich, Germany
20
…Before
…After

47. Corrective Actions Based on the Results (1/3)
• Five-Item Personality Inventory (FIPI)
• We replaced the Ten-Item Personality
Inventory (TIPI) with the Five-Item
Personality Inventory (FIPI), which is less
time-consuming and still provides
sufficient personality data.
• Built-in help
• Users can click the “?” icon next to each
questionnaire question to access on-screen
help with term definitions.
EC-Web - September 2014, Munich, Germany
20
…Before
…After

48. Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the
lengthy AL process during registration, we
give them freedom to decide when to
initiate it through in-app notifications
within the POI suggestions screen.
• User profile page
• We implemented a new user profile page,
making it easier to access and change
context settings, basic user information,
personality information, etc.
EC-Web - September 2014, Munich, Germany
21

49. Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the
lengthy AL process during registration, we
give them freedom to decide when to
initiate it through in-app notifications
within the POI suggestions screen.
• User profile page
• We implemented a new user profile page,
making it easier to access and change
context settings, basic user information,
personality information, etc.
EC-Web - September 2014, Munich, Germany
21

50. Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the
lengthy AL process during registration, we
give them freedom to decide when to
initiate it through in-app notifications
within the POI suggestions screen.
• User profile page
• We implemented a new user profile page,
making it easier to access and change
context settings, basic user information,
personality information, etc.
EC-Web - September 2014, Munich, Germany
21

51. Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the
lengthy AL process during registration, we
give them freedom to decide when to
initiate it through in-app notifications
within the POI suggestions screen.
• User profile page
• We implemented a new user profile page,
making it easier to access and change
context settings, basic user information,
personality information, etc.
EC-Web - September 2014, Munich, Germany
21

52. Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the
lengthy AL process during registration, we
give them freedom to decide when to
initiate it through in-app notifications
within the POI suggestions screen.
• User profile page
• We implemented a new user profile page,
making it easier to access and change
context settings, basic user information,
personality information, etc.
EC-Web - September 2014, Munich, Germany
21

53. Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the
lengthy AL process during registration, we
give them freedom to decide when to
initiate it through in-app notifications
within the POI suggestions screen.
• User profile page
• We implemented a new user profile page,
making it easier to access and change
context settings, basic user information,
personality information, etc.
EC-Web - September 2014, Munich, Germany
21

54. Corrective Actions Based on the Results (2/3)
• In-app notifications
• Instead of forcing users to go through the
lengthy AL process during registration, we
give them freedom to decide when to
initiate it through in-app notifications
within the POI suggestions screen.
• User profile page
• We implemented a new user profile page,
making it easier to access and change
context settings, basic user information,
personality information, etc.
EC-Web - September 2014, Munich, Germany
21

55. Corrective Actions Based on the Results (3/3)
• Many other minor UI improvements
• Revised the contextual factors and contextual conditions
• Improved the UI for displaying personality questionnaire results
• Cleaned up the POI details screen
EC-Web - September 2014, Munich, Germany
22
Before After Before After

56. Corrective Actions Based on the Results (3/3)
• Many other minor UI improvements
• Revised the contextual factors and contextual conditions
• Improved the UI for displaying personality questionnaire results
• Cleaned up the POI details screen
EC-Web - September 2014, Munich, Germany
22
Before After Before After

57. Corrective Actions Based on the Results (3/3)
• Many other minor UI improvements
• Revised the contextual factors and contextual conditions
• Improved the UI for displaying personality questionnaire results
• Cleaned up the POI details screen
EC-Web - September 2014, Munich, Germany
22
Before After Before After

58. Corrective Actions Based on the Results (3/3)
• Many other minor UI improvements
• Revised the contextual factors and contextual conditions
• Improved the UI for displaying personality questionnaire results
• Cleaned up the POI details screen
EC-Web - September 2014, Munich, Germany
22
Before After Before After

59. EC-Web - September 2014, Munich, Germany
Outline
23
• Context-Aware Recommender Systems and their Challenges
• Related Works
• STS (South Tyrol Suggests)
• Usability Assessment and Results
• Conclusions, Lessons Learned and Future Work

60. Conclusions
• Novel and highly usable mobile CARS called STS (South Tyrol Suggests)
that offers various innovative features
• Learns users’ preferences not only using their past ratings, but also
exploiting their personality
• Uses personality to actively acquire ratings for POIs the user has likely
experienced, and to produce more accurate POI recommendations
• Live user study to test the usability of STS
• Results confirm high usability of the proposed system
• Allowed to uncover and resolve some usability issues, such as moderate
confidence in the system and poor integration of some features
EC-Web - September 2014, Munich, Germany
24

61. Lessons Learned
• Only ask users for the minimum required information
• The more information you ask of users, the less likely they will provide it
• Make the system as simple as possible to use
• Keep the system as simple as possible and provide useful on-screen help
or tutorials to instruct users on how to get things done
• Give users control over the system
• Instead of telling users how to use the user interface, give them the ability
to control where they go and what they do. Moreover, always ensure that
the user knows what things are and what they will do
EC-Web - September 2014, Munich, Germany
25

62. EC-Web - September 2014, Munich, Germany
Future Work
• Evaluate the usability of the revised user interface
• Provide users with proactive recommendations and rating requests
• Consider additional important contextual factors in the recommendation
process (e.g., parking availability, traffic conditions)
• Improve explanations to make the recommendation process more transparent
to users
26

63. EC-Web - September 2014, Munich, Germany
Questions?
Thank you.