Evaluating the User Experience
Sónia Sousa & Mati Mottus
2014 Sónia Sousa & Mati Mottus 1
• Nowadays we need to more than just
– Efficient, effective and cheap products.
• We need products that stimulate pleasure;
– Address our needs for social and individual
Usability Vs UX evaluation
Usability is concerned
• with product attributes and
UX is concerned
• in build positive experiences
resulted from the
interaction with the product
2014 Sónia Sousa & Mati Mottus 2
Prevent Experience Reﬂection
• The aim is
– To ” Evaluate HOW user’s perceive a certain product“
• This should be done by comparing different
features of the product
– What would be the difference between
• using a desktop version and the mobile version..
• The product needs to be:
• should include a Desktop, mobile and or tablet version.
2014 Sónia Sousa & Mati Mottus 3
This project contain 3 main activities
• Present your project planning procedure,
– due date: 24.04
• Project implementation,
– until 08.05
• Present the results,
– due date: 09.05
• Deliver the Report,
– due date: 26.05
2014 Sónia Sousa & Mati Mottus 4
The project planning,
• Should include the following issues,
– Forming the working groups;
– Assemble the group environment;
– Define your UX evaluation target and Study it carefully;
– Plan your evaluation procedure;
– Identify the participants;
– Choosing the methods;
– Plan on How to Implement the study;
– Write in detail the evaluation procedure;
– Pilot study;
– Implement and analyze the results; and
– Write the report.
2014 Sónia Sousa & Mati Mottus 5
Planning the project
due date: 24.04
2014 Sónia Sousa & Mati Mottus 6
• Group work organization
– Forming the working groups
• Maximum number of group members is 3
• Assemble the group environment,
– Define each member shared responsibilities and
– Assemble the various tools and services you need.
2014 Sónia Sousa & Mati Mottus 7
• Define your UX evaluation target
– Study it carefully
• Write in a paper a description of the product you will
• To do it for example
– you can turn to your colleague and try (in your
own words) to
• Describe what your product represents to you in 5
2014 Sónia Sousa & Mati Mottus 8
• Plan your evaluation procedure
– Answer to the questions:
– What you need to observe
• tasks, content, layout, and so on
– that enables you (as researcher) to get
• information on HOW user's perceive your product.
– Write the main ideas on a paper.
2014 Sónia Sousa & Mati Mottus 9
• Identify the participants
– Clearly identify who is going to participate in your
study. Focus on describing
• Age, gender, technical knowledge, social status and so
• Write the main ideas on a paper.
2014 Sónia Sousa & Mati Mottus 10
• Choosing the methods
– Think on…
• What methods I am going to use and When?
• How will I do it?
• Answer to the questions
– What is my strategy to achieve my objectives?
– What is the order of my actions?
– How will I use different resources in my actions?
• Write the main ideas on a paper.
2014 Sónia Sousa & Mati Mottus 11
• Plan on How to Implement the study
– Step 1:
• Start by developing a measurable criteria to evaluate
your planned activities.
– Reflect on what tools to use; and how to design
2014 Sónia Sousa & Mati Mottus 12
• Write in detail the evaluation procedure
– Use schemas and diagrams to explain the
– Use bullet list as well
2014 Sónia Sousa & Mati Mottus 13
• Pilot study
– After designing the tools "Do a pilot-test" and
change what is needed to change.
• Ask yourself...
– before present your ideas to the group:
– How do I know that I will be successful?
– How will I use the different resources?
2014 Sónia Sousa & Mati Mottus 14
• Write down your project working plan and
• Present to the group
– please use
• the Gdocs presentation provided by the teacher.
• And the excel file to write your project working plan
2014 Sónia Sousa & Mati Mottus 15
Category of UX methods
• Study types
– Field studies
– Lab studies
– Online studies
– Questionnaires / Scales
• Studied period of
– Before usage
– An episode (of a task or
– Long-term UX
• Development phase
– Early prototypes
– Functional prototypes
– Products on market
• Evaluator / Information
– UX experts
– One user at a time
– Groups of users
– Pairs of users
Sónia Sousa & Mati Mottus, TLU, 2014
• Some Methods are transposed from Usability like…
– Cognitive Walkthrough
– Think aloud
• Others are adaptation of…
– Extended usability testing
– TRUE Tracking Realtime User Experience
– UX Expert evaluation
– Property checklists
• Others are new…
– AttrakDiff, Emocards, Emotion Cards, Emofaces
– Reaction Cards
2014 Sónia Sousa & Mati Mottus 18
How to evaluate?
2014 Sónia Sousa & Mati Mottus 19
Intentions Social Value
Efﬁciency Perceptions of
Relevance of the product
User intention to use
When to evaluate?
2014 Sónia Sousa & Mati Mottus 20
User AttitudesProduct Qualities Usage IntentionsUser Beliefs
Before interaction During interaction After use
First impressions Mood State
Why people like and use certain products What make people use a certain product
What to evaluate?
2014 Sónia Sousa & Mati Mottus 21
Layout, utility, functionalityFirst impressions
Pleasure, Affective arousal
symbolic aspects of the product
How to collect the data
2014 Sónia Sousa & Mati Mottus 22
What is the usage intention?
Granularity measure methods
Value Vs importance of use
Low level experience measure
Overall experience measure
What is the Usage attitude
• Aesthetics and interaction features
– Layout, content, functionality
• Study period
– before and during use
– Personal expectation and needs
• Record of mouse activity and mood state
• Possible methods
– Visual and verbal tests
2014 Sónia Sousa & Mati Mottus 23
Value & perceptions
• What is the intention to use?
– Judgments of the product based on
• Feelings towards the product
• Hedonic qualities and symbolic aspects of the product
– Pleasure, challenge, needs, expressions
– EMG values
2014 Sónia Sousa & Mati Mottus 24
Emotions & Mood state
• Pleasure and affective arousal
– Positive outcomes
• Emotions, joy, excitement
– Affect grid
2014 Sónia Sousa & Mati Mottus 25
2014 Sónia Sousa & Mati Mottus 26
task 1 to 3 Usability
tasks 2 to 3
tasks 4 Emotions EmoCards
– Eliciting the positive aspects of users experience
• Emotions, feelings and motivation
– Pragmatic qualities, hedonic qualities and
• Utility – perceptions of need
• Satisfaction – mood sate
2014 Sónia Sousa & Mati Mottus 27