Usability Testing Plan & Report

Usability Testing
IT’S WHAT THE
USER DOES
It’s notWhat the Software Does.

Usability Testing
Every Product Is Built
With Specific Goals AndTarget User
Group In Mind.

Usability Testing
Every Product Is Built
With Specific Goals AndTarget User
Group In Mind.
That’s Where Exist the
Core of UsabilityTesting

Usability Testing
What Does Usability Means?

Usability Testing
What Does Usability Means?
The ExtentTo Which A Product Can
Be Used By Specified UsersTo Achieve
Specified Goals With Effectiveness,
Efficiency, And Satisfaction In A
Specified Context Of Use.
ISO, the International Organization for Standardization

Usability Testing
What Does UsabilityTesting Means?

Usability Testing
What Does UsabilityTesting Means?
UsabilityTesting Is ATechnique Used
In User-Centered Interaction DesignTo
Evaluate A Product ByTesting It On
Users. It Gives Direct Input About How
Real Users UseThe System.

Usability Testing
Methods of UsabilityTesting
Moderated In-Person UsabilityTesting
Moderated Remote Usability Testing
Unmoderated Usability Testing

Usability Testing
Methods of UsabilityTesting
Moderated In-Person UsabilityTesting
• Provides most reliable insights
• It has least number of distractions
• UT can be controlled spontaneously
Supported with Retrospective Probing Moderation
Moderated Remote Usability Testing
Unmoderated Usability Testing

Usability Testing
Getting Prepared for UsabilityTesting
 Short Initerview with Stakeholders
 Hiring a Usability Lab
 Recruiting Participants
 Creating Scenarios
 Creating Tasks
 Pre-Test Questions
 Post-Test Questions
 Note Taking Template for Observers
 QUIS Questionnaire
 Creating Templates For Facilitator
 Arranging Incentive for Participants
 Conducting Test
 Analysis of QUIS, Observer’s Records
& Videos

Usability Testing
Short Interview with Stakeholders

Usability Testing
“Users are reluctant to use the DivineIT
search in place of classic search form.”

Usability Testing
“Users are reluctant to use the DivineIT
search in place of classic search form.”
“In the coming time, touch based devices will
dominate. S0, we have to be ready with a
highly usable touched based application for
our recruiters”

Usability Testing
Hiring a Usability Lab

Usability Testing
was shortlisted as a UT lab as it was equipped
with required apparatus and software.

Usability Testing
was shortlisted as a UT lab as it was equipped
with required apparatus and software.
Also, our major clientele is there in Bangalore
& we got the access to diversified user profiles

Usability Testing
UsabilityTesting Lab Included Below:
User’s Room EquippedWith
• User’s device & Morae installed on it
• A Recording Camera
• A Voice Transfer Speaker

Usability Testing
UsabilityTesting Lab Included Below:
User’s Room EquippedWith
• User’s device & Morae installed on it
• A Recording Camera
• A Voice Transfer Speaker
Observer’s Room Equipped With
• One Way Mirror
• Display screen to replicate user’s computer screen
• Voice Receiver

Usability Testing
Recruitinting Participants & Arranging
Incentives

Usability Testing
Incentives
We have recruited 9 participants among our
users.
The users were arranged by sales team.

Usability Testing
Incentives
Name Gender Age DivineIT Exp. Total Experience
Vandana Female 26 2 yrs 6 months 3 yrs 4 months
Ajay Male 25 7 months 7 months
Surya Female 31 1 year 1 yr
Prejith Male 29 2 yrs 6 months 5 yrs 7 months
Alok Male 24 6 months 1 Year
Monica Female 22 0 months 1 yrs 7 months
Prashant Male 3 months 3 years
Ishvinder Male 1 yr 5 months 6 years +
Mohit Male 1 year 5 years

Usability Testing
Incentives
Gender
Male (6)
Female (3)

Usability Testing
Incentives
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 - 2 yr 2 - 4 yr 4 - 6 yr 6 - 8 yr 8 - Above
Total Work Experience

Usability Testing
Incentives
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 - 1 yr 1 - 2 yr 2 - Above
Experience on Divine IT

Usability Testing
CreatingTest Scenarios

Usability Testing
Scenario 1:
Recruiter has been asked to shortlist a candidate from IT background. He has never
worked on that kind of position. He has very little knowledge about this functional
area. Job description is also inadequate to search the candidate.
Normally given to recruiters with less experience (0 - 1)

Usability Testing
Scenario 1:
Scenario 2:
User has worked on that kind of profile before and has some knowledge about this
functional area but the job description is inadequate to search the candidate.
Normally given to recruiters with moderate experience (1 - 3)

Usability Testing
Scenario 1:
Scenario 2:
User has worked on that kind of profile before and has some knowledge about this
functional area but the job description is inadequate to search the candidate.
Normally given to recruiters with moderate experience (1 - 3)
Scenario 3:
User has worked on that kind of profile before and the information provided in the
job description is also sufficient to search for the candidate.
Normally given to recruiters with high experience (3+ years of exp)

Usability Testing
CreatingTasks for All Scenarios

Usability Testing
CreatingTasks for All Scenarios
We Created 10 tasks for 3 different scenarios
based on the prerequisites of the each
scenario.
Each recruiter was allocated the scenario and
related task based on his recruiting
experience.

Usability Testing
Pre & PostTest Questionnaire

Usability Testing
NoteTakingTemplate for Observers

Usability Testing
Conducting the
UsabilityTest

Usability Testing
Conducting the UsabilityTest

Usability Testing
Ice Melting Session and pre-test Questionnaire

Usability Testing
Same tasks analysis on DivineIT & Classic Search

Usability Testing
Same tasks analysis on DivineIT & Classic Search
Contextual Enquiry through post-test questionnaire

Usability Testing
Analysis of Pre & Post
Test Questionnaires,
Observer’s Records
Videos & Quick Notes

Usability Testing
Before we dive
into the usability testing evaluation report

Usability Testing
Before we dive
into the usability testing evaluation report
EveryoneWho HasWorked
On DivineIt Deserves

Usability Testing
“For searching IT candidates
DivineIT is the best in Industry”

Usability Testing
“Divine IT provides the best matching resume for IT
candidates ”

Usability Testing
“Divine IT provides the best matching resume for IT
candidates ”
“ Comparing with Naukri & Monster quality of
candidates is really good on DivineIT ”

Usability Testing
Still Recruiters are
Reluctant to Use DivineIT
in Place of Classic Search?

Usability Testing
Through the Usability
Testing Analysis Report

Usability Testing
Learnability

Usability Testing
Learnability
It is the degree to which a user interface can be learned quickly and effectively.
In software testing learnability, according to ISO/IEC 9126, is the capability of a
software product to enable the user to learn how to use it.

Usability Testing
Learnability
It is the degree to which a user interface can be learned quickly and effectively.
Success Parameter:
A recruitment software should have a very high level of lernability i.e. should be
very easy to learn and intuitive to use
In software testing learnability, according to ISO/IEC 9126, is the capability of a
software product to enable the user to learn how to use it.

Usability Testing
Learnability
Vandana Ajay Mohit Surya Prejith Alok Monica Prashant Ishvinder
Score 7 8 9 6 4 3 2 3 3
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
7
8
9
6
4
3
2
3 3
0
1
2
3
4
5
6
7
8
9
10
Scale
Learnability
0 - 4 Low | 5 - 7 Moderate | 8 - 9 High

Usability Testing
Learnability
Score 7 8 9 6 4 3 2 3 3
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
7
8
9
6
4
3
2
3 3
0
1
2
3
4
5
6
7
8
9
10
Scale
Learnability
DivineIT Analysis:
6 users suffered moderate to high level of difficult in learning the usage of
DivineIT and 3 users found it easy to learn DivineIT

Usability Testing
Learnability
How a novice user interacted with the system
Monica Female 22 0 months 1 yrs 7 months

Usability Testing
Learnability

Usability Testing
Learnability
where should I
CLICK?

Usability Testing
Learnability
How to use these
SUGGESTIONS?

Usability Testing
Learnability
Prof Q was not
used by any user?

Usability Testing
Learnability
DivineIT Analysis:
6 users suffered moderate to high level of difficult in learning the usage of
DivineIT and 3 users found it easy to learn DivineIT
Conclusion:
Complexity of InformationArchitecture & User Interface is driving the recruiters
away from repeated use of DivineIT.

Usability Testing
Efficiency
Ability to perform the specific task in least possible time through optimum use of
motor activity, cognitive process & environmental elements?

Usability Testing
Efficiency
Success Parameter:
A recruiter should be able to reach desired candidate’s resume in the minimum
possible time as compared to the competition(Classic Search)

Usability Testing
Efficiency
Score
(Classic Search)
5.3 4.1 1.1 5.38 3.5 5 4.5 4.82 3
Score
(DivineIT)
8.4 10 2.52 4 8 4 2.3 3.53 5.57
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
5.3
4.1
1.1
5.38
3.5
5
4.5 4.82
3
8.4
10
2.52
4
8
4
2.3
3.53
5.57
0
2
4
6
8
10
12
TimeScale
Efficiency – Classic Search Vs DivineIT

Usability Testing
Efficiency
Score
(Classic Search)
5.3 4.1 1.1 5.38 3.5 5 4.5 4.82 3
Score
(DivineIT)
8.4 10 2.52 4 8 4 2.3 3.53 5.57
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
5.3
4.1
1.1
5.38
3.5
5 4.5 4.82
3
8.4
10
2.52
4
8
4
2.3
3.53
5.57
0
2
4
6
8
10
12
TimeScale
Efficiency – Classic Search Vs DivineIT
DivineIT Analysis:
Recruiter is taking longer time in DivineIT to search & find the right candidate.

Usability Testing
Efficiency
Role to key skills mapping is not intuitive
Vandana Female 26 2 yrs 6 months 3 yrs 4 months

Usability Testing
Efficiency
Role to key skills mapping is not intuitive

Usability Testing
Efficiency
An eye opener about exploration mode embedded in DivineIT
Ajay Male 26 7 months 7 months

Usability Testing
Efficiency
An eye opener about exploration mode embedded in DivineIT

Usability Testing
Efficiency
DivineIT Analysis:
Recruiter is taking longer time in DivineIT to search & find the right candidate.
Conclusion:
DivineIT’s Functionalities, UI elements, Interaction Design &Transitions need to be
optimized to make it the best efficient recruitment application.

Usability Testing
Satisfaction Level
Is the product generally enjoyable to operate? Is it nice to look at? Does the user
leave with a positive or negative impression after using it?

Usability Testing
Satisfaction Level
Success Parameter:
A user should feel high degree of satisfaction after using a system.

Usability Testing
Satisfaction Level
Score 8 6 8 7 8 7 6 8
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
8
6
8
7
8
7
6
8
0
1
2
3
4
5
6
7
8
9
SatisfactionScore
Satisfaction Level

Usability Testing
Satisfaction Level
Score 8 6 8 7 8 7 6 8
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
8
6
8
7
8
7
6
8
0
2
4
6
8
10
SatisfactionScore
Satisfaction Level
DivineIT Analysis:
Recruiters displayed high degree of satisfaction and interest while using it.

Usability Testing
Satisfaction Level
DivineIT Analysis:
Recruiters displayed high degree of satisfaction and interest while using it.
Conclusion:
People like the new and innovative approach towards searching and shortlisting
candidates.

Usability Testing
Speed of the System
How fast a system reacts to actions triggered by the User

Usability Testing
Speed of the System
Success Parameter:
Response to an action triggered by user should be delivered before user’s flow of
thought gets interrupted.

Usability Testing
Speed of the System
Success Parameter:
Response to an action triggered by user should be delivered before user’s flow of
thought gets interrupted.
 Response within 0.1 second: User feels that the response is instant.
 Response within 1 second: User doesn’t feel distracted.
 Response within 10 second: User’s focus moves to the dialogue box
By Jokob Nielsen
Source: https://www.nngroup.com/articles/response-times-3-important-limits/

Usability Testing
Speed of the System
Score 5 8 9 9 5 8 7 8
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
5
8
9 9
5
8
7
8
0
1
2
3
4
5
6
7
8
9
10
SpeedScore
Speed of DivineIT

Usability Testing
Speed of the System
Score 5 8 9 9 5 8 7 8
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
5
8
9 9
5
8
7
8
0
1
2
3
4
5
6
7
8
9
10
SpeedScore
Speed of DivineIT
DivineIT Analysis:
Most of the users felt that DivineIT gave the response instantaneously.

Usability Testing
Speed of the System
DivineIT Analysis:
Most of the users felt that DivineIT gave the response instantaneously
Conclusion:
Speed of the system is as per user’s expectations

Usability Testing
User Interface Noise
A lot of information is presented in a small space which makes it difficult for user
to find, classify, comprehend and operation upon them.
Success Parameter:
User should be able to comprehend the page information with minimum
cognitive load.

Usability Testing
Score 8 5 5 5 6 5 5 8
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
8
5 5 5
6
5 5
8
0
1
2
3
4
5
6
7
8
9
UINoiseScore
UI Noise

Usability Testing
Score 8 5 5 5 6 5 5 8
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
8
5 5 5
6
5 5
8
0
2
4
6
8
10
UINoiseScore
UI Noise
DivineIT Analysis:
Most of the users felt that the interface is workable but it takes a bit more effort
to work around all the functionalities.

Usability Testing
How much efforts a user has to put in to select education?
Prejith Male 29 2 yrs 6 months 5 yrs 7 months

Usability Testing
How much efforts a user has to put in to select education?

Usability Testing

Usability Testing
No one was able to
notice this section

Usability Testing
No one noticed the
writing pad feature
of view source

Usability Testing
No one noticed
these accelerator
actions

Usability Testing
Interestingly, some
users used it to
change the pane size

Usability Testing
No one used this
mechanism to
browse within and
among pages

Usability Testing
Again, no user
noticed Prof. Q, even
if they require it.

Usability Testing
DivineIT Analysis:
Most of the users felt that the interface is workable but it takes a bit more effort
to work around all the functionalities.
Conclusion:
A lot of functionalities have been cramped into a small space which makes it
difficult for the user to find and work on them for refining the search result.

Usability Testing
Relevance
Are recruiters able to find out the candidates they are searching for?

Usability Testing
Relevance
Success Parameter:
Application should build the confidence in user that it is intelligent enough to
understand & follow their instructions.

Usability Testing
Relevance
Score 7 7 8 8 9 7 5 8
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
7 7
8 8
9
7
5
8
0
1
2
3
4
5
6
7
8
9
10
RelevanceScore
Relevance

Usability Testing
Relevance
Score 7 7 8 8 9 7 5 8
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
7 7
8 8
9
7
5
8
0
1
2
3
4
5
6
7
8
9
10
RelevanceScore
Relevance
DivineIT Analysis:
DivineIT helped recruiters in finding the right candidate with high degree of
relevance.The relevance is far better than classic search.

Usability Testing
Relevance
DivineIT Analysis:
DivineIT helped recruiters in finding the right candidate with high degree of
relevance.The relevance is far better than classic search.
Conclusion:
Recruiters are highly satisfied with the application intelligence.

Usability Testing
Ease of Use
The overall perception of the user about using an application. Is it difficult, easy
or not usable at all

Usability Testing
Ease of Use
Success Parameter:
The application should be as near to the user’s mental model as possible.

Usability Testing
Ease of Use
Score 7 6 9 8 5 8 3 7
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
7
6
9
8
5
8
3
7
0
1
2
3
4
5
6
7
8
9
10
EaseofUseScore
Ease of Use

Usability Testing
Ease of Use
Score 7 6 9 8 5 8 3 7
Total Exp 3.4 0.7 5 1 5.7 1 1.7 3 6
DivineIT Exp 2.6 0.6 1 1 2.6 0.6 0.1 0.3 1.5
7
6
9
8
5
8
3
7
0
2
4
6
8
10
EaseofUseScore
Ease of Use
DivineIT Analysis:
Recruiters struggled to find and use various functionalities.

Usability Testing
Ease of Use
Trying to find and use the “and, or, not functionality”
Surya Female 31 1 year 1 yr

Usability Testing
Ease of Use
Trying to find and use the “and, or, not functionality”

Usability Testing
Ease of Use
Use of different
behavior for same
mechanism is
confusing the user

Usability Testing
Ease of Use
Inconsistent behavior of
drag and drop component
makes it very difficult to
create boolean criteria

Usability Testing
Ease of Use
Recruiters took a lot of
time to select parameters
on these slide bars

Usability Testing
Ease of Use
Why there are
49 similar
resume with
every candidate

Usability Testing
Ease of Use
It is very difficult
to operate upon
these parameters
in a small space

Usability Testing
Ease of Use
Selecting
education is one
of the parameter

Usability Testing
DivineIT Analysis:
Recruiters struggled to find and use various functionalities.
Conclusion:
Recruiters are reluctant to use DivineIT because they find it difficult to create their
required search criteria on it.
Ease of Use

Usability Testing
Cross Analysis: Learnability & Ease of Use

Usability Testing
An application which can be learnt quickly and used easily has high propensity of
getting repeat users and being in the favorite list.
Success Parameter:
Application should be high on learnability as well as ease of use scale.

Usability Testing
An application which can be learnt quickly and used easily has high propensity of
getting repeat users and being in the favorite list.
Name Vandana Ajay Mohit Surya Prejith Alok Monica Prashant Ishvinder
Learnability Ok High High Ok Low Low Low Low Low
Ease of Use Low Ok Easy Low Ok Easy Low Low Ok

Usability Testing
What Actually Happened?

Usability Testing
An innovative approach of candidate search

Usability Testing
An innovative approach of candidate search got
wrapped in a traditionally featured and complex looking
user interface

Usability Testing
user interface further supported by the error prone &
inconsistently responding scripting structure

Usability Testing
inconsistently responding scripting structure along with
some good work in creating the high degree relevance
to recruiter’s requirements

Usability Testing
inconsistently responding scripting structure along with
some good work in creating the high degree relevance
to recruiter’s requirements put forward for the usage of
our recruiters without any kind of help & motivational
mechanism available at their end.

Usability Testing
TheWay Ahead

Usability Testing
TheWay Ahead
Embedding User Centered
Design Process in the core of
Product Development Life Cycle

Usability Testing
TheWay Ahead
Creating an Inline
help module and feedback
mechanism for recruiters.

Usability Testing
TheWay Ahead
Establishing better
communication between
recruiter and the system, i.e.
when no results are found

Usability Testing
TheWay Ahead
Building a selective,
supportive and motivational
model of introducing this
innovation approach
into the market

Usability Testing
TheWay Ahead
Using above usability testing
scales as guiding principles for
the revamp of DivineIT

Usability Testing
Next Lined Up Activity

Usability Testing
Next Lined Up Activity
FOR DivineIT Revamp

Usability Testing Plan & Report

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Usability Testing Plan & Report

Similar to Usability Testing Plan & Report (20)

Recently uploaded

Recently uploaded (20)

Usability Testing Plan & Report