SlideShare a Scribd company logo
1 of 73
Download to read offline
@amir_ansari @dius_au
1
UX: How to measure more
than a gut feel
Amir Ansari – DiUS
@amir_ansari @dius_au
2
6’3”
@amir_ansari @dius_au
3
@amir_ansari @dius_au
4
@amir_ansari @dius_au
5
product and
technology strategy
application-specific
devices
web applications
mobile applications
@amir_ansari @dius_au
6
What is UX?
@amir_ansari @dius_au
7
Study and Design of ALL aspects
of a person’s EXPERIENCE with a
product or service
@amir_ansari @dius_au
8
@amir_ansari @dius_au
9
http://www.uxisnotui.com/
@amir_ansari @dius_au
10
Pain points
Usability
Likelihood to recommend
Customer loyalty
Current behaviours Right direction
Alternate designs
Mental model
Why measure?
@amir_ansari @dius_au
11 www.theimprovegroup.com
@amir_ansari @dius_au
12
@amir_ansari @dius_au
13
Quantitative Qualitative
What? Why?
• Data is solid and repeatable
• Confirm and deduce
• Result-oriented
• Large sample sizes
• Data is rich and deep
• Explore and describe
• Process-oriented
• Small sample sizes
• A/B or multi-variant testing
• Analytics
• Remote un-moderated
usability testing
• Surveys
• Focus groups
• Cognitive walkthroughs
• Moderated usability
testing
• Card sorting
@amir_ansari @dius_au
14
@amir_ansari @dius_au
15
@amir_ansari @dius_au
16 http://lyndabuitrago.com/tag/leaky-gut/
@amir_ansari @dius_au
17
• Picture of me measuring UX
@amir_ansari @dius_au
18
@amir_ansari @dius_au
19
@amir_ansari @dius_au
20
www.r-66.info
Completion rate
Task time
Task level satisfaction
Conversions
Page views
System Usability Scale (SUS)
Net Promoter Score (NPS)
@amir_ansari @dius_au
21
Qualitative
Why?
• Data is rich and deep
• Explore and describe
• Process-oriented
• Small sample sizes
• Focus groups
• Cognitive walkthroughs
• Moderated usability
testing
• Card sorting
@amir_ansari @dius_au
22
http://dralexkent.files.wordpress.com/2011/12/gut.jpg
We feel we know…
• who our customers are
• what their pain points & barriers are
• what will trigger them to use our product
• this product will be a good idea
@amir_ansari @dius_au
23
How do we
measure and
validate these?
And When?
@amir_ansari @dius_au
24
Visioning Iterations Iterations Iterations Iterations
MVP
Launch
Inception &
Planning
1 2 3
@amir_ansari @dius_au
25
Technique 1
@amir_ansari @dius_au
26
The sooner we can find which features
are worth investing in, the sooner we
can focus our limited resources on the
best solution to our business problems.
Jeff Gothelf – LEAN UX
“
@amir_ansari @dius_au
27
@amir_ansari @dius_au
28
Visioning Iterations Iterations Iterations Iterations
MVP
Launch
Inception &
Planning
1 2 3
@amir_ansari @dius_au
29
30 – 60 mins 3 One on one interviews
@amir_ansari @dius_au
30
• Concerns
• Fears
• Pain points
• Ideal experience
• Ideal channel
@amir_ansari @dius_au
31
• Stakeholders observing
• Call centre Staff felt engaged
and included
• More insights than we had
anticipated
@amir_ansari @dius_au
32
• Some had not come prepared
• Stakeholders wanted to ask
‘speculative questions’
@amir_ansari @dius_au
33
Technique 2
@amir_ansari @dius_au
34
@amir_ansari @dius_au
35
Visioning
Inception &
Planning
1 2 3
Iterations Iterations Iterations Iterations
MVP
Launch
@amir_ansari @dius_au
36
Focus group 3
2 hours
Focus group 1
Focus group 2
24
@amir_ansari @dius_au
37
@amir_ansari @dius_au
38
Feature 1 Feature 2 Feature 3
@amir_ansari @dius_au
39
• Stakeholders buy-in through
observation
• Validate assumptions
• Client was anonymous
@amir_ansari @dius_au
40
• 2 hours wasn’t quite enough
• Running sessions back to
back in the same evening!
@amir_ansari @dius_au
41
Technique 3
@amir_ansari @dius_au
42
Visioning
Inception &
Planning
1 2 3
Iterations Iterations Iterations Iterations
MVP
Launch
@amir_ansari @dius_au
43
1 hour
One on one testing
6
@amir_ansari @dius_au
44
http://www.nngroup.com/articles/how-many-test-users/
@amir_ansari @dius_au
45
@amir_ansari @dius_au
46
Lean Stack framework by Spark 59
(www.spark59.com/leanstack)
@amir_ansari @dius_au
47
1. Background
2. Falsifiable Hypotheses
3. Details
6. Next Actions
5. Validated Learning
4. Results
User 1 User 2 User 3 etc.
Hypothesis 1   
Hypothesis 2  
Etc.   
• What feature or assumption to test
• What questions to ask
• How well does X perform
• Etc. • What to test next
• What hypothesis do we want to validate
1. That majority of users think X
2. That majority of users perform Y
3. That majority of users need Z
4. Etc.
• Previous experiment taught us …
• We want to find out…
• We want to test our solutions…
1. We can proceed with X
2. We need to tweak Y
3. We need to pivot and get rid of Z
Title: Experiment 2: XXX Author: Created:
@amir_ansari @dius_au
48
1. Background
2. Falsifiable Hypotheses
3. Details
6. Next Actions
5. Validated Learning
4. Results
User 1 User 2 User 3 etc.
Hypothesis 1   
Hypothesis 2  
Etc.   
• What feature or assumption to test
• What questions to ask
• How well does X perform
• Etc. • What to test next
• What hypothesis do we want to validate
1. That majority of users think X
2. That majority of users perform Y
3. That majority of users need Z
4. Etc.
• Previous experiment taught us …
• We want to find out…
• We want to test our solutions…
1. We can proceed with X
2. We need to tweak Y
3. We need to pivot and get rid of Z
Title: Experiment 2: XXX Author: Created:
• From focus groups, it was clear that one barrier
with other online providers has been the lack of
trainer support
• That providing access to trainers in the classroom
and telling students upfront about trainer
availability will eliminate barriers and increase
conversion
• Show users the marketing pages that talk about
trainer availability
• Show users the classroom with the trainer being
online
• Users indeed look for trainer availability and are
comforted by knowing OTI provides access to
them
• Test something else! ; )
@amir_ansari @dius_au
49
bit.ly/rapid-debrief
@amir_ansari @dius_au
50
@amir_ansari @dius_au
51
@amir_ansari @dius_au
52
@amir_ansari @dius_au
53
@amir_ansari @dius_au
54
• Speed of testing
• The feedback loop
• Stakeholder buy-in
(live stream)
• Rapid de-brief
• One designs and one tests
@amir_ansari @dius_au
55
• Refinements sometimes not
subtle enough
• Design ping pong
@amir_ansari @dius_au
56
@amir_ansari @dius_au
57
@amir_ansari @dius_au
58
5
One on one testing
Focus groups
= $800
2 FTEs
X $160
X $375*
* Daily gross equivalent based on a gross annual salary of $90k
= $750
$1,550
16 = $3,200
3 FTEs
X $200
X $375* = $1,125
$4,325
If doing this internally…
@amir_ansari @dius_au
59
@amir_ansari @dius_au
60
http://stamfordinteractive.com.au/conversations/benefits-of-guerrilla-testing/
@amir_ansari @dius_au
61
You can quantify qualitative data!
bit.ly/quantifying-qualitative bit.ly/t-distribution
@amir_ansari @dius_au
62
Completion rate
Task time
Task level satisfaction
Conversions
Page views
System Usability Scale (SUS)
Net Promoter Score (NPS)
{
Visioning
Inception &
Planning
Iterations Iterations Iterations Iterations
MVP
Launch
@amir_ansari @dius_au
63
@amir_ansari @dius_au
64
Define why, what, when
who, where
how, how often
1
@amir_ansari @dius_au
65
Become friends with the PM
(or the person with the budget)
2
@amir_ansari @dius_au
66
Keep it lean and measure often
3
@amir_ansari @dius_au
67
Get stakeholder buy-in
(Get them to observe)
4
@amir_ansari @dius_au
68
Be consistent
in how you measure and capture measurements
5
@amir_ansari @dius_au
69
Define when to Quant and when
to Qual
6
@amir_ansari @dius_au
70
https://www.measuringusability.com/topics/UX
@amir_ansari @dius_au
71
Thanks to
@amir_ansari @dius_au
72
http://dralexkent.files.wordpress.com/2011/12/gut.jpg
We feel we know…
• who our customers are
• what their pain points & barriers are
• what will trigger them to use our product
• this product will be a good idea
@amir_ansari @dius_au
73
Thank you
Qualitative data can in fact be
converted into quantitative
measures even if it doesn't come
from an experiment or from a large
sample size.
Jeff Sauro – Measuring Usability
“

More Related Content

Viewers also liked

7 สามัญ ฟิสิกส์ เฉลย (2)
7 สามัญ ฟิสิกส์ เฉลย (2)7 สามัญ ฟิสิกส์ เฉลย (2)
7 สามัญ ฟิสิกส์ เฉลย (2)warangnan
 
JR's Lifetime Advanced Analytics
JR's Lifetime Advanced AnalyticsJR's Lifetime Advanced Analytics
JR's Lifetime Advanced AnalyticsChase Hamilton
 
о методобъединении
о методобъединениио методобъединении
о методобъединенииMurat77
 
Fashion Influencer Research May'15
Fashion Influencer Research May'15Fashion Influencer Research May'15
Fashion Influencer Research May'15Adila Harsono
 
о доп общеразвив программе
о доп общеразвив программео доп общеразвив программе
о доп общеразвив программеMurat77
 
Government’s Challenge - Solving Today’s Problems with Yesterday’s Tools - Pa...
Government’s Challenge - Solving Today’s Problems with Yesterday’s Tools - Pa...Government’s Challenge - Solving Today’s Problems with Yesterday’s Tools - Pa...
Government’s Challenge - Solving Today’s Problems with Yesterday’s Tools - Pa...Software AG South Africa
 
7 สามัญ อังกฤษ เฉลย
7 สามัญ อังกฤษ เฉลย7 สามัญ อังกฤษ เฉลย
7 สามัญ อังกฤษ เฉลยwarangnan
 
о порядке формирования
о порядке формированияо порядке формирования
о порядке формированияMurat77
 
о правилах приема и отбора детей
о правилах приема и отбора детейо правилах приема и отбора детей
о правилах приема и отбора детейMurat77
 
DAFTAR PUSTAKA Kajian Kecepatan Penarikan Purse Line dan Waktu Penangkapan Te...
DAFTAR PUSTAKA Kajian Kecepatan Penarikan Purse Line dan Waktu Penangkapan Te...DAFTAR PUSTAKA Kajian Kecepatan Penarikan Purse Line dan Waktu Penangkapan Te...
DAFTAR PUSTAKA Kajian Kecepatan Penarikan Purse Line dan Waktu Penangkapan Te...Andi Mahardika
 
об орг внеаудиторной раб
об орг внеаудиторной рабоб орг внеаудиторной раб
об орг внеаудиторной рабMurat77
 
родит комитет
родит комитетродит комитет
родит комитетMurat77
 
ใบงานที่5
ใบงานที่5ใบงานที่5
ใบงานที่5warangnan
 
Digital access at wvss – parent information evening
Digital access at wvss – parent information eveningDigital access at wvss – parent information evening
Digital access at wvss – parent information eveningSteveRauh
 
о сайте
о сайтео сайте
о сайтеMurat77
 

Viewers also liked (17)

7 สามัญ ฟิสิกส์ เฉลย (2)
7 สามัญ ฟิสิกส์ เฉลย (2)7 สามัญ ฟิสิกส์ เฉลย (2)
7 สามัญ ฟิสิกส์ เฉลย (2)
 
JR's Lifetime Advanced Analytics
JR's Lifetime Advanced AnalyticsJR's Lifetime Advanced Analytics
JR's Lifetime Advanced Analytics
 
Parramatta's Heritage
Parramatta's HeritageParramatta's Heritage
Parramatta's Heritage
 
о методобъединении
о методобъединениио методобъединении
о методобъединении
 
math
mathmath
math
 
Fashion Influencer Research May'15
Fashion Influencer Research May'15Fashion Influencer Research May'15
Fashion Influencer Research May'15
 
о доп общеразвив программе
о доп общеразвив программео доп общеразвив программе
о доп общеразвив программе
 
Government’s Challenge - Solving Today’s Problems with Yesterday’s Tools - Pa...
Government’s Challenge - Solving Today’s Problems with Yesterday’s Tools - Pa...Government’s Challenge - Solving Today’s Problems with Yesterday’s Tools - Pa...
Government’s Challenge - Solving Today’s Problems with Yesterday’s Tools - Pa...
 
7 สามัญ อังกฤษ เฉลย
7 สามัญ อังกฤษ เฉลย7 สามัญ อังกฤษ เฉลย
7 สามัญ อังกฤษ เฉลย
 
о порядке формирования
о порядке формированияо порядке формирования
о порядке формирования
 
о правилах приема и отбора детей
о правилах приема и отбора детейо правилах приема и отбора детей
о правилах приема и отбора детей
 
DAFTAR PUSTAKA Kajian Kecepatan Penarikan Purse Line dan Waktu Penangkapan Te...
DAFTAR PUSTAKA Kajian Kecepatan Penarikan Purse Line dan Waktu Penangkapan Te...DAFTAR PUSTAKA Kajian Kecepatan Penarikan Purse Line dan Waktu Penangkapan Te...
DAFTAR PUSTAKA Kajian Kecepatan Penarikan Purse Line dan Waktu Penangkapan Te...
 
об орг внеаудиторной раб
об орг внеаудиторной рабоб орг внеаудиторной раб
об орг внеаудиторной раб
 
родит комитет
родит комитетродит комитет
родит комитет
 
ใบงานที่5
ใบงานที่5ใบงานที่5
ใบงานที่5
 
Digital access at wvss – parent information evening
Digital access at wvss – parent information eveningDigital access at wvss – parent information evening
Digital access at wvss – parent information evening
 
о сайте
о сайтео сайте
о сайте
 

Similar to Agile Australia 2014 | UX: How to measure more than a gut feel by Amir Ansari

Agile Australia 2014
Agile Australia 2014Agile Australia 2014
Agile Australia 2014Amir Ansari
 
20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion ConferenceCraig Sullivan
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meCraig Sullivan
 
John lewis lego craig sullivan 1335
John lewis lego craig sullivan  1335John lewis lego craig sullivan  1335
John lewis lego craig sullivan 1335Charlie Lines
 
Design sprint talk - april 2017
Design sprint talk - april 2017Design sprint talk - april 2017
Design sprint talk - april 2017Jeffrey Mikres
 
20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid themCraig Sullivan
 
Conversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteConversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteWebanalisten .nl
 
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Marketing Festival
 
Detecting Good Abandonment in Mobile Search
Detecting Good Abandonment in Mobile SearchDetecting Good Abandonment in Mobile Search
Detecting Good Abandonment in Mobile SearchJulia Kiseleva
 
It's time to research our designs better. Here's how. UIUX Conference 2018 - ...
It's time to research our designs better. Here's how. UIUX Conference 2018 - ...It's time to research our designs better. Here's how. UIUX Conference 2018 - ...
It's time to research our designs better. Here's how. UIUX Conference 2018 - ...Sophie Freiermuth
 
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Myths and Illusions of Cross Device Testing - Elite Camp June 2015Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Myths and Illusions of Cross Device Testing - Elite Camp June 2015Craig Sullivan
 
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...UXPA International
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9Craig Sullivan
 
Customer Feedback: the missing piece of the Agile puzzle
Customer Feedback: the missing piece of the Agile puzzleCustomer Feedback: the missing piece of the Agile puzzle
Customer Feedback: the missing piece of the Agile puzzleskierkowski
 
Sourcing The Right Participants For Your UX Research & Testing
Sourcing The Right Participants For Your UX Research & TestingSourcing The Right Participants For Your UX Research & Testing
Sourcing The Right Participants For Your UX Research & TestingUserZoom
 
Validating hypotheses with user research
Validating hypotheses with user researchValidating hypotheses with user research
Validating hypotheses with user researchgabrieljonsson
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeCraig Sullivan
 

Similar to Agile Australia 2014 | UX: How to measure more than a gut feel by Amir Ansari (20)

Agile Australia 2014
Agile Australia 2014Agile Australia 2014
Agile Australia 2014
 
20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference20 Ways to Shaft your Split Tesring : Conversion Conference
20 Ways to Shaft your Split Tesring : Conversion Conference
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
 
John lewis lego craig sullivan 1335
John lewis lego craig sullivan  1335John lewis lego craig sullivan  1335
John lewis lego craig sullivan 1335
 
Design sprint talk - april 2017
Design sprint talk - april 2017Design sprint talk - april 2017
Design sprint talk - april 2017
 
20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them
 
Conversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynoteConversion Hotel 2014: Craig Sullivan (UK) keynote
Conversion Hotel 2014: Craig Sullivan (UK) keynote
 
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
 
Detecting Good Abandonment in Mobile Search
Detecting Good Abandonment in Mobile SearchDetecting Good Abandonment in Mobile Search
Detecting Good Abandonment in Mobile Search
 
It's time to research our designs better. Here's how. UIUX Conference 2018 - ...
It's time to research our designs better. Here's how. UIUX Conference 2018 - ...It's time to research our designs better. Here's how. UIUX Conference 2018 - ...
It's time to research our designs better. Here's how. UIUX Conference 2018 - ...
 
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Myths and Illusions of Cross Device Testing - Elite Camp June 2015Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
 
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9Confessions of an uber optimiser   conversion summit - craig sullivan - v 1.9
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
 
Designing Mobile UX
Designing Mobile UXDesigning Mobile UX
Designing Mobile UX
 
Designing the Right Product
Designing the Right ProductDesigning the Right Product
Designing the Right Product
 
Customer Feedback: the missing piece of the Agile puzzle
Customer Feedback: the missing piece of the Agile puzzleCustomer Feedback: the missing piece of the Agile puzzle
Customer Feedback: the missing piece of the Agile puzzle
 
Sourcing The Right Participants For Your UX Research & Testing
Sourcing The Right Participants For Your UX Research & TestingSourcing The Right Participants For Your UX Research & Testing
Sourcing The Right Participants For Your UX Research & Testing
 
Validating hypotheses with user research
Validating hypotheses with user researchValidating hypotheses with user research
Validating hypotheses with user research
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
 
How to Do Better Market Research
How to Do Better Market ResearchHow to Do Better Market Research
How to Do Better Market Research
 

More from DiUS

Lunch and Learn: You have the data, now what?
Lunch and Learn: You have the data, now what?Lunch and Learn: You have the data, now what?
Lunch and Learn: You have the data, now what?DiUS
 
How to build confidence in your release cycle
How to build confidence in your release cycleHow to build confidence in your release cycle
How to build confidence in your release cycleDiUS
 
Serverless microservices: Test smarter, not harder
Serverless microservices: Test smarter, not harderServerless microservices: Test smarter, not harder
Serverless microservices: Test smarter, not harderDiUS
 
Test Smart, not hard
Test Smart, not hardTest Smart, not hard
Test Smart, not hardDiUS
 
10 things-to-inspire-in-10-mins
10 things-to-inspire-in-10-mins10 things-to-inspire-in-10-mins
10 things-to-inspire-in-10-minsDiUS
 
Trends and development practices in Serverless architectures
Trends and development practices in Serverless architecturesTrends and development practices in Serverless architectures
Trends and development practices in Serverless architecturesDiUS
 
Deploying large-scale, serverless and asynchronous systems - without integrat...
Deploying large-scale, serverless and asynchronous systems - without integrat...Deploying large-scale, serverless and asynchronous systems - without integrat...
Deploying large-scale, serverless and asynchronous systems - without integrat...DiUS
 
The Diversity Dilemma - Supporting our Sisters in STEM
The Diversity Dilemma - Supporting our Sisters in STEMThe Diversity Dilemma - Supporting our Sisters in STEM
The Diversity Dilemma - Supporting our Sisters in STEMDiUS
 
GameDay - Achieving resilience through Chaos Engineering
GameDay - Achieving resilience through Chaos EngineeringGameDay - Achieving resilience through Chaos Engineering
GameDay - Achieving resilience through Chaos EngineeringDiUS
 
The case for consumer-driven contracts
The case for consumer-driven contractsThe case for consumer-driven contracts
The case for consumer-driven contractsDiUS
 
Deploy with Confidence using Pact Go!
Deploy with Confidence using Pact Go!Deploy with Confidence using Pact Go!
Deploy with Confidence using Pact Go!DiUS
 
Crafting Quality Software
Crafting Quality SoftwareCrafting Quality Software
Crafting Quality SoftwareDiUS
 
Metrics on the front, data in the back
Metrics on the front, data in the backMetrics on the front, data in the back
Metrics on the front, data in the backDiUS
 
Antifragility and testing for distributed systems failure
Antifragility and testing for distributed systems failureAntifragility and testing for distributed systems failure
Antifragility and testing for distributed systems failureDiUS
 
DIY IoT Backend
DIY IoT BackendDIY IoT Backend
DIY IoT BackendDiUS
 
How to Build Hardware Lean
How to Build Hardware LeanHow to Build Hardware Lean
How to Build Hardware LeanDiUS
 
Behaviour Change and Coaching: What we can learn from BJ Fogg
Behaviour Change and Coaching: What we can learn from BJ FoggBehaviour Change and Coaching: What we can learn from BJ Fogg
Behaviour Change and Coaching: What we can learn from BJ FoggDiUS
 
Power in Agile Teams
Power in Agile Teams Power in Agile Teams
Power in Agile Teams DiUS
 
The Diversity Dilemma: Attracting and Retaining Talented Women in Technology-...
The Diversity Dilemma: Attracting and Retaining Talented Women in Technology-...The Diversity Dilemma: Attracting and Retaining Talented Women in Technology-...
The Diversity Dilemma: Attracting and Retaining Talented Women in Technology-...DiUS
 
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary SlidesRise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary SlidesDiUS
 

More from DiUS (20)

Lunch and Learn: You have the data, now what?
Lunch and Learn: You have the data, now what?Lunch and Learn: You have the data, now what?
Lunch and Learn: You have the data, now what?
 
How to build confidence in your release cycle
How to build confidence in your release cycleHow to build confidence in your release cycle
How to build confidence in your release cycle
 
Serverless microservices: Test smarter, not harder
Serverless microservices: Test smarter, not harderServerless microservices: Test smarter, not harder
Serverless microservices: Test smarter, not harder
 
Test Smart, not hard
Test Smart, not hardTest Smart, not hard
Test Smart, not hard
 
10 things-to-inspire-in-10-mins
10 things-to-inspire-in-10-mins10 things-to-inspire-in-10-mins
10 things-to-inspire-in-10-mins
 
Trends and development practices in Serverless architectures
Trends and development practices in Serverless architecturesTrends and development practices in Serverless architectures
Trends and development practices in Serverless architectures
 
Deploying large-scale, serverless and asynchronous systems - without integrat...
Deploying large-scale, serverless and asynchronous systems - without integrat...Deploying large-scale, serverless and asynchronous systems - without integrat...
Deploying large-scale, serverless and asynchronous systems - without integrat...
 
The Diversity Dilemma - Supporting our Sisters in STEM
The Diversity Dilemma - Supporting our Sisters in STEMThe Diversity Dilemma - Supporting our Sisters in STEM
The Diversity Dilemma - Supporting our Sisters in STEM
 
GameDay - Achieving resilience through Chaos Engineering
GameDay - Achieving resilience through Chaos EngineeringGameDay - Achieving resilience through Chaos Engineering
GameDay - Achieving resilience through Chaos Engineering
 
The case for consumer-driven contracts
The case for consumer-driven contractsThe case for consumer-driven contracts
The case for consumer-driven contracts
 
Deploy with Confidence using Pact Go!
Deploy with Confidence using Pact Go!Deploy with Confidence using Pact Go!
Deploy with Confidence using Pact Go!
 
Crafting Quality Software
Crafting Quality SoftwareCrafting Quality Software
Crafting Quality Software
 
Metrics on the front, data in the back
Metrics on the front, data in the backMetrics on the front, data in the back
Metrics on the front, data in the back
 
Antifragility and testing for distributed systems failure
Antifragility and testing for distributed systems failureAntifragility and testing for distributed systems failure
Antifragility and testing for distributed systems failure
 
DIY IoT Backend
DIY IoT BackendDIY IoT Backend
DIY IoT Backend
 
How to Build Hardware Lean
How to Build Hardware LeanHow to Build Hardware Lean
How to Build Hardware Lean
 
Behaviour Change and Coaching: What we can learn from BJ Fogg
Behaviour Change and Coaching: What we can learn from BJ FoggBehaviour Change and Coaching: What we can learn from BJ Fogg
Behaviour Change and Coaching: What we can learn from BJ Fogg
 
Power in Agile Teams
Power in Agile Teams Power in Agile Teams
Power in Agile Teams
 
The Diversity Dilemma: Attracting and Retaining Talented Women in Technology-...
The Diversity Dilemma: Attracting and Retaining Talented Women in Technology-...The Diversity Dilemma: Attracting and Retaining Talented Women in Technology-...
The Diversity Dilemma: Attracting and Retaining Talented Women in Technology-...
 
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary SlidesRise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
Rise of the machines: Continuous Delivery at SEEK - YOW! Night Summary Slides
 

Recently uploaded

Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Muhammad Tiham Siddiqui
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 

Recently uploaded (20)

Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 

Agile Australia 2014 | UX: How to measure more than a gut feel by Amir Ansari

Editor's Notes

  1. 15 years in the usability and UX space, most of it at the management level of some sort. Last stint before DiUS was 8 years at Stamford (Aus’s largest specialised UX agency). I’ve done my 10000+ hours.
  2. Who knows DiUS?
  3. Privately owned. 100 staff in both Melb & Syd. In May this year (2014) we turned 10 years old. SW engineers at heart, BA, hardware engineers, UX. Agile and nimble. People drive our culture to innovate. We believe in a strong and vibrant community. We sponsor conferences and meetups. Our employees give their time and skills to solve community problems.
  4. Product and technology strategy: what product should be built, how to get it to market. We do this for our own products too. Web applications: QANTAS-Jetstar accommodation booking, Vodafone self-service. Application-specific devices: a fridge magnet for displaying in-home energy usage. Mobile applications: the iPad app for Australia Post Digital Mailbox.
  5. Any uxer’s out there? Hands up. Who can confidently define what UX means?
  6. Firstly, most people still don’t know the breadth of user experience skillset (uxisnotui).
  7. Sometimes confusion between qualitative and quantitative.
  8. Two approaches to measuring UX.
  9. As different as the two approaches are, we both need each other – so let’s all be friends and hug.
  10. My talk - So why the title?
  11. I'm obsessed with measuring, learning and iterating when it comes to UX.
  12. UX has traditionally been waterfall. With Agile and Lean, Lean UX has taken traction. Build, measure, learn is very appropriate to the UX discipline.
  13. I’m going to talk about some the harder measurements – some of the qualitative techniques and prove how the can provide value through some example case studies. I'm a bit tired of having to sell the importance of UX... Not so much that’s it’s important, but more that it’s not fluffy, design only or that it’s not meant for measuring and driving/impacting decisions. It’s not ALL about SAMPLE SIZE. IT’s qualitative – about the Why, not the what! Most recently I've been trying to bring the Research and Insights team of one of my clients on board. Still get the good old argument of 'why Such a small sample size? And that it's not significant. It's not about sample size and you can still measure UX and give great value back to the project.
  14. UX metrics are relatively easy and black and white, so I won’t be covering them today. You can google and find out all about them.
  15. In this presentation, I’m going to cover the qualitative aspects.
  16. So you’re probably thinking to yourself: what does Qualitative measurement look like in your agile project?
  17. When you don’t have access to users. I used this when working for an enterprise startup – OTI.
  18. Enterprise startup using Lean Startup approach.
  19. Next best thing to users. Managed to capture and count the number of times the following themes occurred: Motivations and expectations to study; Concerns/fears/barriers/pain points; Ideal experience and channel. Outcome: Personas; Feature sets; Hypotheses.
  20. What worked well.
  21. Watch out for anecdotes. Plural of anecdote is anecdotes, not data!
  22. Qualitative, small size but to go deep and understand sentiments, reasons for behaviours and perceptions etc. As we had planned to run experiments throughout the project, keeping it lean and light was important.
  23. Things to look out for when running group-based sessions: poor facilitation, group think. In quant, each data is seen as being independent. In a focus group, the entire session = 1 data point, due to Group Think effect. We split users up and do activities to increase our data points.
  24. Outcomes.
  25. Stakeholders got total buy-in – no barriers to convince re findings.
  26. What didn’t work so well.
  27. Moderated usability testing.
  28. As we had planned to run experiments throughout the project, keeping it lean and light was important.
  29. How many know about the Nielsen Norman Group? Godfathers of usability and UX, especially when it comes to research and field studies.
  30. Lean experiment report.
  31. Lean experiment report.
  32. Results annotated, tallied up, prioritised, recommendations made and fed back at the end of that very day.
  33. Online classroom was changed to Online study centre. Seeing two modes side by side preferred over tab. Seeing full course price was still important.
  34. What worked well.
  35. What didn’t work so well.
  36. Hand’s up if you’re a product owner, project manager, responsible for the project’s budget?
  37. So how much do these activities cost? UX REDUCES COST, INCREASES PRODUCTIVITY, SALES, BRAND LOYALTY AND ADVOCACY. Just to name a few. Research was done by Forrester Research that suggested the cost of fixing code was 10 times the cost of the design activities if done upfront , and up to 100 x when the product was launched.
  38. We have trusted partners / market research recruiters that find our users form a large database.
  39. PM happy.
  40. If too expensive, to Guerrilla Usability Testing. Shorter, often 15 – 30 mins. Few features tested. Often, on the road or at the participant’s location, and with some training developers can do it (Steve Krug’s book – This is not Rocket Surgery). Book by Steve Krug – Rocket Surgery Made Easy provides tips on how anybody can conduct usability testing.
  41. You can convert a usability problem into a frequency, using confidence intervals. Categorizing and counting issues. N= + 30 = good estimate for standard deviations, otherwise stick with t-distribution.
  42. As the product matures, you can introduce more UX metrics and quantitative measurements.
  43. Picking the right technique is very important. And these questions can help you choose and narrow down on the technique you need.
  44. For the OTI project, I spent 1 hour very early on defined the research/experiment approach, costed it and put it to the PM. It was included in the budget and signed off. Now some of you may not have access to the person with the wallet, or the budget may have already been allocated prior. In this case, you may have to take a more guerrilla approach to your experiments, OR find an advocate who can play in your team to try and convince a minimum number of user sessions.
  45. Keep it lean, quick and fast. 5 – 6 users per experiment. Prioritise issues. Feedback quickly. You don’t want to be seen as been the bottleneck – pragmatism is important.
  46. Get them to observe. If not, make sure you run through your issues. Agree on pivots and recommendations.
  47. Use consistent approaches to test and report. Consider using templates. Benchmark as design matures for future measurability.
  48. Use consistent approaches to test and report. Consider using templates. Benchmark as design matures for future measurability.
  49. Measuring the User Experience is more statistical, and assumes you have an existing product or service.