How to Carry Out a Design Due Diligence? | Seattle Interactive 2019

How to carry out a design due
diligence?
Boeckle Martin, Seattle Interactive Conference SIC - 2019

Data-Driven UX
PURE (Pragmatic Usability Rating by Experts)
Heuristic Evaluation and Classification Framework
Methods, Clusters, Dimensions
DDD Cluster
Evaluation Framework DDD
Introduction Design Due Diligence

Typically four or more parallel due diligences are
combined in a target assessment
Financial due diligence Legal due diligence Strategic/commercial
due diligenceTechnology due diligence
• Certification of accounts
• Deal structure
• Determination of
offer price
• Draft of investment
agreement
• Legal go-ahead
for deal
• Commercial assessment of
growth and profitability
potential (target
attractiveness)
• Assessment of technology/
platform that support the business
• Evaluation of technological risks
• Accountants
• I-banks
• (Strategy consultant)
• Lawyer • Strategy consultant• Specialized technology
consultant
• Financial analysis
• Various valuation models
• In-depth review of
accounting information
• Scrutiny of
contractual risks
• Assessment of
regulatory risks
• Formulation of investment
agreement
• Evaluation of
market dynamics
– Sustainable growth
– Customer needs
– Segmentation
• Assessment of competitive
positioning
• Evaluation of target
historical performance
• Business plan assessment
• Evaluation of quality of technology in
light of deal rationale (e.g., grow
business) by assessing the target
technology on dimensions like
maturity, scalability, performance,
maintainability
• Assessment of impact of technology on
deal, including strategic technology fit,
risk, required investments, future
running costs

Conversational UX aims to
support seamless voice-based
interfaces for the interaction
between customer and the
service
Good UX finds the balance
between challenging tasks
and the ones which are too
difficult to manage
Which parts should be considered within a
Design Due Diligence?
Financial DD
Legal DD
Strategic / Commercial DD
Technology DD
1 2

Design DD
Framework
Team and Organization
• Design Culture
• Value of Design
• Collaboration between
Designers/Developers/Product
Developers
• Diversity of the team
UX of product or service
• Ease of Use
• Efficiency
• Effectiveness
• Task success
• Customer satisfaction
• Accessibility
Process & Tools
• Design Methods (e.g., Design
Thinking, UX Design process)
• Tools (Usability Testing etc.)
• Decision making process

DD
Dependencies
DD Assessment
Preparation
DD Evaluation
DD Final
report
Understand key client and
key system dependencies
in order to select the
appropriate evaluation
method
Select the to be assessed
cluster and dimension
Select the right method
Prepare the assessment
Carry out the assessment
Calibration of maturity
levels
Link it to a maturity level
Classification of identified
usability issues (e.g., severity,
importance)
Detailed recommendations
about how to improve the
identified issues including risk
and the effort to fix
Analysis focuses on assessment whether IT of target
company meets the buyers objectives
3-6 weeks

• Industry and product
• Target group
• Solution purpose
Phase 1: DD Dependencies
Setup of key meetings, communication to experts, preparation
of surveys and tools
System access
Data access
Availability of target users
Availability of experts
Keysystem
dependencies
Keyclient
dependencies
Client goals (e.g., increase
of retention rate or perform
competitive analysis)
Client needs (e.g., involve-
ment of target users, key
stakeholders)
First client contact and
baselining on
Understand system and client dependencies

Phase 2: DD Assessment Preparation
Selection of to be assessed cluster and dimensions
• UX within customer lifecycle (e.g., adoption, enrichment)
• Usability dimensions (e.g., efficiency, effectiveness)
• UI perception (e.g., accessibility)
• Customer engagement
Prepare for the assessment
• Setup the testing environment
• Survey-Tool (e.g., for assessment through
survey)
• Cognitive walkthrough preparations
Select the right method which covers the dimensions
to be assessed
• Survey-based evaluations
• System walkthroughs and interviews
• Data-driven analysis

UX within
Customer
lifecycle (B2C)
Customer
engagement
UI perception
UI operations
Usability
Dimensions
Design Due Diligence Clusters

Customer lifecycle
Ensuring user retention by
providing appropriate UX
along the full customer
lifecycle, including different
stages when a customer uses
the application (e.g., buying a
product)
Introduction
First phase of user lifecycle is establishing a
link to the user (e.g., registration)
Dimension: Acquisition, activation
Growth
End users are on-boarded, learn and explore the offered services
of the application Dimension: Retention
Maturity
Users are familiar with all offered features and can fully leverage
them Dimension: Referral, advocacy, revenue
Saturation
End-user activities are declining as engagement incentives and
sensation of new impulses slowly wear off
Possible development
• Reinforcement necessary to renew the growth
• Contraction
• End-of-life (e.g., account deletion)

Usability Dimensions
Enabling efficient and
successful operational usage
by focusing mainly on
behavioral metrics like the
measurement of efficiency,
effectiveness, etc.
Error rates
Measuring number of errors the customer made by completing a
task (or errors caused by the system)
Number of completed tasks
Tracking number of successfully completed tasks within a given
timeframe to make efficiency levels comparable
Number of clicks
Counting the number of clicks on a specific site or module
of the product to identify hot spots or complexity drivers
Ease of use
Understanding end users perception on how easy it is to use the
product to identify preferred improvement areas

UI perception
This cluster mainly covers
attitudinal metrics, for
example “what the end-users
say about the front-end of
the application”
Satisfaction towards the interface
Measuring end-user satisfaction while using the interface (e.g.,
booking a flight) to identify
levers which matter
Usefulness
Rating the usefulness of the interface (e.g., support of booking
form) while completing a process
Quality of feedback messages
Evaluating provided feedback messages regarding their
appropriateness to support error resolution and learning
Content
Assess the structure and quality of the provided content to identify
improvement opportunities and needs

Customer engagement
This cluster contains metrics,
which aim to understand the
target groups emotions while
using the application
Engagement
Tracking degree of engagement while using the product
or service
Happiness
Measuring factor of happiness while using the product or
service
Emotion
Assessing level of emotion at certain steps of completing a
task (e.g., positive or negative emotions while using the
payment function)

Select the right method based on coverage of required assessment
dimensions
Cluster Dimension Heuristic Evaluation PURE
Domain Expert
Interviews AARRR Framework
Google Heart
Framework
Usability Metrics
(Data)
Usability Metrics
(Survey)
Cognitive
Walkthrough
Customer
Experience Index
Customer
lifecycle
Adoption ✓ ✓
Enrichment ✓ ✓
Activation ✓
Acquisition ✓
Advocacy ✓
Referral ✓ ✓
Retention ✓ ✓ ✓ ✓ ✓
Revenue ✓ ✓ ✓
Usability
Dimensions
Efficiency ✓
Ease of Use ✓ ✓ ✓ ✓ ✓ ✓
Effectiveness ✓ ✓
Operability ✓ ✓ ✓ ✓
Task success ✓
UI perception
Accessibility ✓ ✓ ✓ ✓
Satisfaction ✓ ✓ ✓ ✓
Usefulness ✓ ✓ ✓ ✓
Universality ✓ ✓ ✓ ✓
Customer
Engagement
Happiness ✓ ✓
Engagement ✓ ✓
Emotion ✓ ✓
Methods

Each assessment method covers certain dimensions
within (multiple) clusters
Methods
Customer
lifecycle
UI operations UI perception
Customer
excitement
Heuristic Evaluation
UI gets reviewed by experts focusing on different UX aspects
PURE
Application’s design assessment (ease of use) based on UX/UI expert ratings
Expert Interviews
Structured qualitative interviews to capture expert UI views/knowledge
AARRR Framework
Framework for growth to evaluate performance of startups
Google’s Heart Framework
Predefined metrics (e.g., Adoption, Retention etc.) to measure UX on a large scale
Usability Metrics (Data)
Data driven usability metrics are a method to evaluate usability quality
Usability Metrics (Survey)
Method for setting up usability survey’s to be filled out by usability testers
Cognitive Walkthrough
Task-specific method for expert examination from the user’s perspective
Customer Experience Index
Growth evaluation method focusing on CX quality & customer loyalty
Focus on
method
Partial
focus
No
focus

Crosscheck the selected methods with system and client
dependencies
Methods Analysis Data System access App user Domain expert Estimated effort
Prerequisites
Execution
Cognitive
Walkthrough
qualitative Swalkthrough RequiredRequired
Usability Metrics
(Survey – driven)
quantitative +
qualitative
M
survey based
evaluation
Required
Google’s Hearts
Framework
quantitative +
qualitative
data driven and
survey based
Required L
AAARR Framework quantitative L
data driven
analysis
RequiredRequired
Expert Interviews qualitative Mwalkthrough Required Required
SPURE
quantitative +
qualitativewalkthrough RequiredRequired
Heuristic evaluation qualitative Swalkthrough Required
Usability Metrics
(Data – driven)
data-driven
analysis
quantitative MRequired
Customer Experience
Index
quantitative +
qualitative
L
data driven and
survey based
Required Required

Assessment approaches
System walkthroughs and Interviews
• Breadth-first discovery and evaluation of
status-quo
• Uncovers pain points
Survey-based evaluation
• Development of first hypotheses
• Gathering of strategic inputs for assessments
Data-driven analysis
• Deep-dive on systems and applications
• Uncovers capability gaps
Breadth of topics covered
Wide assessment
Deep subsequent analysis
Depth of assessment
C
B
A
C
B
A
Depending on the chosen assessment approach, each
methodology provides a certain breadth and depth

Carry out the assessment
• Depending on the selected assessment typology, consider
each step of the assessment carefully
• Follow best practices when carrying out any evaluations
(e.g., for experts interviews)
Calibration of maturity levels regarding baseline (industry,
product, target group, solution purpose)
Link it to a maturity level
• Break down the analysis into individual issues or assessed
tasks
• Link the results a pre-defined scale and provide a maturity
level for the overall system
Phase 3: DD evaluation

Identified issues Maturity assessment
Main menu reveals inconsistent colors and is linked
with the wrong page. It therefore conflicts with the
rule “Consistency of Standards”
Filter function is hidden and at the wrong place and
therefore conflicts with the rule “Consistency of
Standards”
Severity 1 Severity 2
Severity 3 Severity 4
Usability catastrophe imperative to fix this before
product can be released
Major usability problem important to fix, so
should be given high priority
Minor usability problem fixing this should be
given low priority
Cosmetic problem only need not be fixed unless
extra time is available on project
Level1Level4
Completion of the assessment usually leads to a list of
issues including the usability maturity level

01234
Flexibility and Efficiency of use
Aesthetic and Minimalist
Design
Help Users Recognize,
Diagnose, and Recover from
Errors
Help and Documentation
Recognition Rather than Recall
The selection don’t show
details at all
Countries in the selection are
missing
The end-user needs to
manually zoom-back to read
the details
Details are displayed with a
non readable color
Contact details are displayed
below the country selection
To select a seat the end-user
jumps between different pages
Several elements follow
minimalistic approaches
Long vertical selection also
does not solve the problem
Scrollable (right – left) seat
selection is not user friendly
All design elements of the
application follow a
minimalistic design
The error message is too
general
No error message at all (few
places only in a certain
language)
The error message does not
make sense
The error message is barely
readable
No help links at all
Information are hidden in
the burger menu and links
not clickable
No onboarding at all
The app should provide simple
onboarding functionalities
Onboarding is executed in a
bad way
No issues identified The system should not force the
end-user to memorize items
The application provide
meaningful error messages
Ambition levelMaturity level
Severity
Observations
Ambition level for each observation

Analyse the results
• Aim for a proper categorization of the identified results
(e.g., severity level, importance)
Based on the identified issues the design due diligence should
provide
• Detailed recommendations on how to improve the
identified issues, including the effort to fix
• Risk analysis which help the evaluators in their decision-
making process of replacing, maintaining or modernizing
the system in question
Phase 4: Recommendations

Following dimension have been considered within the
design due diligence
Efficiency
Adoption
Enrichment
Acquisition
Universality
Ease of Use
Accessibility
Task success
Retention
Advocacy
Referral
Customer
Engagement
Effectiveness
Operability
Emotion
Activation
Satisfaction
Usefulness
Happiness
Engagement
UX within
Customer Life CycleUI Perception
Usability
Dimensions
Revenue

DD
Dependencies
DD Assessment
Preparation
DD Evaluation
DD Final
report
Dependencies have been
clarified
Heuristic evaluation
method has been selected
Maturity assessment of a
product within the
financial domain
Final report
Example Design Due Diligence within the financial domain
by following the four phases
3-6 weeks

Define scope
Focus on the most crucial user flows
and functionalities of the product,
which will be evaluated
Choose heuristics
Literature provides several lists of
heuristics, which represent common
user experience guidelines
Analyze and present
Severity rating is added to each of the identified issue. The
list of issues is organized by the issues containing the highest
deviance against the selected heuristics
Evaluate and identify
Experts evaluate against the selected set of heuristics. The
final results will be aggregated and discussed. Findings
should describe the issues specific and clearly
Start End
Deep-dive
Understand requirements and users
The experts who evaluate the product should understand the
business needs of the product as well as the target group (e.g.,
accessibility for an older audience)

Heuristic evaluation uses 10 heuristics for user
interface design
Visibility of system
status
The system informs the user
about the current system
status (e.g., signup process,
file upload, booking a
flight).
Match between
system and the real
world
The system design should
follow the characteristics of
the real world.
User control
and freedom
Users should move freely
between the different
stages of a process and
often require a
“emergency” exit.
Consistency and
standards
The overall design should
follow interface standards.
Error prevention
Good design prevents
problems from occurring
before the next step.
Recognition rather
than recall
The memory load of the
user should be minimized.
Flexibility and
efficiency of use
The system should be
designed for experienced
and inexperienced users. Aesthetic and
minimalistic design
Designs should not contain
irrelevant information.
Every irrelevant information
competes with relevant
pieces of information.
Help and
documentation
Help information should be
structured and easy to find.
Help users recognize,
diagnose and recover
from errors
Error messages should
express the problem
precisely.

Visibility of system status
The system informs the user about the
current system status (e.g., signup
process, file upload, booking a flight).

Match between system
and the real world
The system design should follow the
characteristics of the real world.

User control and
freedom
Users should move freely between the
different stages of a process and often
require a “emergency” exit.

Consistency and
standards
The overall design should follow interface
standards.

Error prevention
Good design prevents problems from
occurring before the next step.

Recognition rather than
recall
The memory load of the user
should be minimized.

Flexibility and efficiency
of use
The system should be designed for
experienced and inexperienced users.

Aesthetic and minimalist
design
Designs should not contain irrelevant
information. Every irrelevant information
competes with relevant pieces of
information.

Help users recognize,
diagnose, and recover
from errors
Error messages should express the
problem precisely.

Help and documentation
Help information should be structured and
easy to find.

Two usability experts conducted a heuristic evaluation of two
applications within the finance domain by following the guidelines of
Nielsen Norman Group (NNG).
Comparison of two two systems with similar customer journeys
System A = 10 Usability Issues
System B = 24 Usability Issues
Results have been clustered and structured
Iterative process with several software developers in order to
come up with four categories
Development of a novel framework for the classification of usability
issues

Usability Heuristic Severity
rating
Identified conflicts with UX/UI best practice
Consistency and
standards
Aesthetic and minimalistic
design
#1 - Bad arrangement of buttons
#2 - Software version expires
#3 - Inconsistent Buttons
#4 - Display of last forms is missing
#5 - incorrect use of icon
#15 - Summary of Calculate and Deduct
#17 - Certain options (e.g. export posting for billing, archive export etc.) must be defined before
billing
1
3Consistency and standards
1
Consistency and
standards
Consistency and
standards
3
Visibility of system status
1
Consistency and
standards
3
Consistency and
standards
4Flexibility and efficiency of use
Different levels of severity reveal a certain type of complexity
(Identified Usability Issues System B)

Developed
Framework
Categories
I Represent minor improvements on the UI ‒ level like colour
codes, inconsistent buttons, menus, warnings and feedback
UI - Optimizations
II
Weak System Support
The system does not support the end-users in their customer
journey. For instance, the accomplishment of tasks (e.g., complete
booking process) is connected with considerable effort
III
Structural adjustments within the system architecture
These issues are connected to a higher effort to fix (e.g., changes in
the business logic)
I
V
High usability conflict with intended customer journey
These issues prevent the end-user to accomplish their goals and
need to be fixed immediately. While end-users will probably drop
out of the process, the effort to fix as well as the severity rating is
relatively high

Different levels of severity ratings reveal a certain
effort to fix
Technical effort to fix
Severityofusabilityissue
II
III
IV
I
I
Represent minor improvements on the UI ‒
level like color codes, inconsistent buttons,
menus, warnings and feedback
UI - Optimizations (0.2)
II
Weak System Support (0.4)
End user needs to memorize configurations
and the system does not support the
accomplish of tasks
III
Structural adjustments within the system
architecture (0.6)
Issues prevent the end-user to accomplish
task and need to be addressed
IV
High usability conflict with intended customer
journey (0.8)
Issues are connected to changes within the
business logic

Identified issues
have been
categorized within
the proposed
framework
#9
#12
#11
#10
#15
#13
#14
#22
#17#16
#19
#18
#23
#21
#20
#5
#3
#8
#24
#4
#2
#7
#1
#6
Effort to fixSeverityRating
III
I
II
Structural adjustmentsUI optimization
Weak system support
High usability conflict with intended user
journey
IV

Number of issues in each System
Issue
Q1 - UI optimization
Q2 - Weak system support
Q3 - Structural adjustments
Q4 - High usability conflict with
the intended user journey
Overall score 281 76
System A System B
11 4
5 1
4 5
4 0
24 10
w1 = factor 0.2 (for issues with severity 1)
X1 = number of identified issues with low severity (cosmetic problem only)
281
Low UX
Maturity
Competitive analysis of two systems, which share a similar
user journey

“Component based architectures foster reusable components and therefore aim to handle changes more quickly.”
“UI libraries of modern web-based architectures can be updated much more flexible than older systems with less
capabilities.”
“Modern user interfaces solve problems in different ways and therefore a comparison of identified issues between two system
could be unrealistic.”
“Generally, current tech stacks have a higher focus on maintainability.”
“Especially for legacy systems, the effort to fix evolves around the replacement, maintenance or the modernization of
functionalities or the entire system. Consequently, this requires extensive expertise, preparation and planning as well.
Therefore, the decision-making process is also more time consuming.”
Technological Dependencies and Limitations

A minimum of eight steps are required to conduct a PURE evaluation
Rate and review
steps
Calculate inter-rater
score reliability
Set step
boundaries
Identify fundamental
tasks
Compare
scores*
Add qualitative
notes*
Summarize issues
Discuss scoresDefine happy paths
Start End
Identify user
types

Usability ratings for steps are used to generate an overall
PURE score
1
The step can be accomplished by the target user, due to low cognitive load or
because it‘s a known pattern, such as the acceptance of a terms-of-service
agreement
3
The step is difficult for the target user, due to significant cognitive load or
confusion, some target users would likely fail or abandon the task at this point
2
The step requires a notable degree of cognitive load (or physical effort) by the
target user, but can generally be accomplished with some effort

Usability ratings for steps are used to generate an overall PURE+
score
Individual step ratings PURE scoreTask
8
Search for a car and
reserve it
3
Walk to the car and
open it through the
application
Drive to the
destination and
finish the booking
2
1
1
1
1
3
2
1
2
1
1. Open the application
2. Click share (or default)
3. Decide which type of car to take 4.
Select a car from multiple offers
5. Press Reserve

Application screenshots show the identified issues in the search for a
car task
The app does not automatically show the closest car from the user’s location and
how to get there (e.g. minutes to walk there), which is an important feature
supporting the user in the decision making process of selecting a car
The app informs the user about the car’s
location and how to get there only after
the reservation is completed
3

The first phase includes the general product introduction, problem
definition, customer journey selection and definition
Customer journey selection and task definition
Enter
passenger
details
Receive check-in
confirmation and
Ticket
Enter
payment
details
Tom wants
to fly
Open the
app
Select booking
flights
Enter flight
details
Search and
select flights
FlightbookingCheck-in
Open the
App
Select
check-in and
flight
Select
seat
Enter
passport
details
Receive
ticket
Product:
Flight App (iOS)
Number of users decreased
Number of flight bookings
reduced
Number of online check-in
decreased
Identified issues which need
to be clarified within the DDD
Defined for major assessment

Six steps are required to prepare and conduct a data driven
assessment
Set goals & define
questions to answer
Set clear measurement priorities
(What to measure & how to measure
it)
Analyze the data
Collect the right
data
Interpret results
Provide recommendations for
improvements
Start End

A data driven assessment can be performed via several different
usability metrics
Dimension Description
Task completion time The amount of time users take to complete the intended task
Number of clicks The amount of clicks which are necessary to complete the intended task
Time in mode The amount of time used inside the help functionality
Duration of use of particular functions within the interface
Time spent on particular parts of the task
Mean duration of pauses between actions
Time until event The amount of time elapsed until users employ a specific feature or perform a particular action
The amount of time before a secondary task is solved
Rate of input by the user, using keyboard The number of words entered by minute in comparison to the necessary amount of input
Use frequency The frequency of function use or action
The number of functions used
The number of interface actions
The number of times help is consulted
Information accessed The amount of information that users access or employ
The number of web pages visits
… …

Thank you for your attention
boeckle.martin@bcgplatinion.com

Detailed description of Dimensions 1/2
Cluster Dimension Description
Customer lifecycle
(Commercial)
Adoption Describes the process of beginning to use something new (e.g., feature, product, service)
Enrichment Focuses on likelihood that end-users are buying additional products and services within the customer life cycle
Activation Describes the process of introducing the service to the end-users, which leads them towards the "aha moment"
Acquisition Shows through which channels the end-users are coming from
Advocacy Reveals the likelihood of recommending the product or service in question to others
Referral Focuses on the recommendations of end-users through social media shares, email
Retention Shows how many end-users are kept within a predefined amount of time
Revenue Highlights the number of paying end-users, how much they spend on average and how to incentivize to buy again
UI operations
Efficiency Reveals how well the products or services do what they should do
Ease of Use Highlights how easy the product or service is used by it's intended users
Effectiveness Reveals how useful the product or service is by considering accuracy and completeness
Operability Focuses on the degree to which a product or system has attributes that makes it easy to operate and control
Task success Highlights the number of completed tasks within a product or service
Backup

Detailed description of Dimensions 2/2
Cluster Dimension Description
UI perception
Accessibility Focuses on making a product accessible for a broad range of people, specifically people with disabilities
Satisfaction Refers to the subjective response from users about their feelings when using the software
Usefulness
Reveals to which degree a user is satisfied with the perceived achievement of pragmatic goals, including the results of
use and the consequence of use
Universality
Considers whether a software product accommodates a diversity of users with different cultural backgrounds (e.g., local
is considered)
Customer
Excitement
Happiness Reveals if the end-user has a pleasurable or satisfying user experience
Engagement Highlights and measures how involved the end-users are with the new product or service
Emotion Highlights if customers feel good about their experience
Backup

How to Carry Out a Design Due Diligence? | Seattle Interactive 2019

Recommended

Recommended

More Related Content

Similar to How to Carry Out a Design Due Diligence? | Seattle Interactive 2019

Similar to How to Carry Out a Design Due Diligence? | Seattle Interactive 2019 (20)

More from Seattle Interactive Conference

More from Seattle Interactive Conference (20)

Recently uploaded

Recently uploaded (20)

How to Carry Out a Design Due Diligence? | Seattle Interactive 2019