A brief talk we gave in the Switzerland-based Ethics Matters group.

1. Studio Llama
Ethics Matters., 06.07.2021
Scarlett Eisenhauer, Benedikt Merz & Andreas Wolters
Designing for
I
mpact
Thoughts on metrics and goals for responsible
products

2. Studio Llama
What to expect.
We want to show you that deliberate goals & metrics are crucial to responsible
products.
We want you to be comfortable to approach metrics & goals.
We’ll have to be high-level (a bit).

3. Gross
Domestic
Product
But
fi
rst:
Let’s talk about the

4. Gross
Domestic
Product
What is it?
“GDP measures the monetary value
of
fi
nal goods and services—that is,
those that are bought by the
fi
nal
user—produced in a country in a
given period of time (say a quarter or
a year). It counts all of the output
generated within the borders of a
country. GDP is composed of goods
and services produced for sale in the
market and also includes some
nonmarket production, such as
defense or education services
provided by the government.”
I
nternational Monetary Fund
But
fi
rst:
Let’s talk about the

5. Just how important is the GDP?
Promises & campaigning
“We’re bringing it (the GDP) from 1 percent
up to 4 percent. And I actually think we
can go higher than 4 percent. I think you
can go to 5 percent or 6 percent.”
Donald J. Trump
Basis for policy making
“The ability to accurately predict the e
ff
ect
of such (social welfare interventions) is a
powerful tool and keenly sought by all
involved. Without an agreed upon method
of measuring welfare within the
fi
eld of
economics, GDP is widely used.”
Reference
De-facto economic jargon
“Furthermore, the GDP and its various
proxies—rates of growth, expansion,
recovery—have become the very language
of the nation's economic reportage and
debate. We literally cannot think about
economics without them.”
Reference

6. What the
Gross
Domestic
Product
was made
to measure
Let’s talk about

8. Studio Llama
« »
Richard T. Froyen
One reads with dismay of Presidents Hoover and then
Roosevelt designing policies to combat the Great
Depression of the 1930's on the basis of such sketchy
data as stock price indices, freight car loadings, and
incomplete indices of industrial production.The fact
was that comprehensive measures of national
income and output did not exist at the time.The
Depression, and with it the growing role of
government in the economy, emphasized the need
for such measures.

9. The GDP was
developed as a
set of metrics
to measure
how e
ff
ective
policies were
to li
ft
the U.S.
out of the
Great
Depression.
INTENTION

10. Studio Llama
« »
Clifford Cobb, Ted Halstead, and Jonathan Rowe
The Clinton Administration waited expectantly, but
the applause never came. Voters didn't feel better,
even though economists said they should.The
economy as economists define it was booming, but
the individuals were not. President Bill Clinton
actually sent his economic advisers on the road to
persuade Americans that their experience was
wrong and the indicators were right.
I
ndicators over voters

11. The GDP was
developed as a
set of metrics
to measure
how e
ff
ective
policies were
to li
ft
the U.S.
out of the
Great
Depression.
The GDP
became the
metric for the
economy as a
whole — and
over time a
proxy for
things like
welfare and
health.
EFFECT
INTENTION

12. Studio Llama
« »
Simon Kuznets
or: the guy who developed the GDP.
National Income, 1929-1933
The welfare of a nation can, therefore, scarcely be
inferred from a measurement of national income.

13. Measuring something makes it important.
Not measuring something makes it
unimportant.
WHAT CAN WE LEARN FROM THIS?
We have the tendency
to lose sight of what a
measure represents.
Psychologists call this
surrogation.
SURROGATION
We loose sights of
variability and
alternatives through a
narrowing of options
and processes of
naturalization.
BLINDNESS TO ALTERNATIVES

14. Studio Llama
What does all that have to do
with design?

15. I
n its most fundamental form, design is a set
of methods to solve a problem. By de
fi
ning the
problem we set an agenda and also implicitly
say what makes a solution "good" — o
ft
en as a
set of metrics.
DEFINING DESIGN & METRICS

16. Studio Llama
Let’s look at three points.
1
I
n
fi
nite Scroll
How user-focussed metrics shaped a
harmful interaction pa
tt
ern
2 3
Dark pa
tt
erns
How biased metrics can give rise to
deception.
Design & communication
How design has communicated in the
past to get the “seat on the table”.

17. 1
I
nfinite Scroll
The “Infinite Scroll” interaction
pattern automatically loads new
items in the background — gone
are the pages to click through.
It was developed by Aza Raskin
in 2006.
Illustration by
Augusto Zambonato

18. Studio Llama
« »
Aza Raskin
Coming up with the infinite scroll was just the joy of
creating something that felt better. It was really
simple. It was just like, hey: here is one way in which
we don’t have to ask the user to make a choice that
they don’t care about.
I
ntention behind the infinite scroll

19. Studio Llama
« »
Kate Lucey, Hearst UK
When the infinite scroll was first brought into media
publishing companies, prosecco bottles popped on
every office floor. (…).The replacement of homepage
widgets and carousels with a fast-loading, infinitely
scrolling experience saw every brand’s engagement
metrics go through the roof.
I
nfinite scroll was applied everywhere.

20. Aza Raskin now estimates that his invention, in
fi
nite scroll, wastes
around 200,000 human lifetimes — each day.
UNINTENDED CONSEQUENCE OF INFINITE SCROLL
I
n
fi
nite scroll was initially well-intentioned
and a
tt
empted to solve a user need. But it
aimed at a single goal — and was then applied
to drive business numbers.
WHAT CAN WE LEARN FROM THIS?

21. 2 Dark Patterns
There are many types of dark
patterns. What they all have in
common is that they are tricks
used in websites and apps that
make you do things that you
didn't mean to.
Illustration by TechCrunch

22. I
t’s unfortunately still a relevant topic.

23. Studio Llama
« »
Hoa Loranger, Nielsen Norman Group
I think there’s a pressure nowadays on teams to meet
or exceed their business goals, to constantly grow at
a faster and probably unsustainable rate.This
pressure ends up making even good companies
adopt dark patterns to satisfy very short-sighted
aims.
How do dark patterns happen?

24. Dark pa
tt
erns are not bad design.
Dark pa
tt
erns are good design aimed at a sh*t
set of metrics.
HOW DO DARK PATTERNS COME ABOUT?

25. Some of them have become illegal.

26. Dark pa
tt
erns occur with an imbalanced set
of metrics and / or goals — when business
outcomes ma
tt
er without regard for much
else.
WHAT CAN WE LEARN FROM THIS?

27. 3 Design & communication
User Experience is a relatively
new discipline that still is finding
its place in the corporate world.
Part of that is the goal was to get
“a seat on the table”: by
showing that design is important
we can bring it into the realms of
corporate leadership.
So design was simplified to be
measured by business value.

28. Design has that seat by now.
What do we use it for?
WHAT NOW?

29. Studio Llama
What can we do differently?

30. Don't miss the
storm while
you’re looking
in the hay.

31. We have to
care about and
have a say on
the goals and
metrics.
THOUGHT 1
I
t’s part of our
job.
I
t’s also
one of our best
tools to shape
the
conversation.

32. Studio Llama
Goals & metrics are your responsibility.
Goals and metrics are one of our most important tools.
Let‘s learn to be comfortable with them.
Good design needs to shape the conversation towards a coherent vision that brings all
different needs — business needs, user needs, wider (societal/global) needs — together.
Goals and metrics are a crucial tool to shape that conversation.
You’re not a designer / researcher?
Make goals (& metrics) your responsibility, too. Bring in your perspective on goals: what’s
good, what’s bad, what we should avoid. Do research. Growth & business goals will likely
be part of that — we need to take that into account.

33. Studio Llama
Goals & metrics are designers’ responsibilities.
Balance goals & metrics to the different perspectives, well.
Dark patterns occur when business goals reign above all else. Bring all relevant
perspectives to the table; discuss, what an appropriate balance is.

34. I
f there are
only metrics,
talk about
goals.
THOUGHT 2
De
fi
ne the
metrics, if
there are only
goals.

35. Studio Llama
On goals & metrics.
« We want to improve click-through rate by 8%. »
That’s a metric.
« We want to improve our online store and also aim to empower our
customers with a more transparent interaction with our offerings
in order to help them reflect on their needs and our products. »
That’s a goal.

36. Studio Llama
On goals & metrics.
Goals and metrics go hand-in-hand.
You won’t always be part of a project from the very beginning, so some idea of what’s to
achieve is likely already there. It might be an idea of a goal, it might be a sketch of how the
goal should be measured. Let‘s make this discussion explicit.
Use that and shape the discussion.
Goals and metrics may not be defined early in the project — which is your chance to open
up that discussion. The goal is defined? Challenge them, ask about the metrics and
redefine the goal after, if needed. The metrics are there? Start a discussion on what the
goal is.

37. Metrics and
goals can be
diverse.
THOUGHT 3
Let di
ff
erent
elements
complement
each other to
get a more
holistic view.

38. Studio Llama
Working with goals
There are many ways of de
fi
ning a goal.
There’s not one optimal way to define a goal, but many options: Empathy Maps, Problem
Statements or Vision Statements.
You should
fi
nd the "ultimate" root(s).
A classic method here are the Six Why’s - ask why six times and you will discover the
ultimate root problem or goal you are after (or be closer to it).
As much detail as needed.
Be comfortable with complexity. It doesn’t have to be one statement that sums up all
perspectives in one goal — why not use a set of four statements / areas to intertwine
business goals, technological development goals, user goals and societal goals?

39. Studio Llama
Types of metrics
There are many types of metrics — some easier to use, some harder.
Metrics shouldn’t be chosen just because they’re easy to collect (eg. Analytics only).
Be informed about what we are really measuring
What does a metric really mean? Avoid biased or implied assumptions whenever possible.
As an example: scroll depth does not necessarily mean that somebody shows interest —
unless we can prove that it does.

40. Studio Llama
Types of „metrics“ (and methods)
Some examples
I
nterviews Surveys
Usability tests
Standardized usability metrics
(eg SUS)
Experience metrics
(eg A
tt
rakDi
ff
)
Societal-level metrics
Web analytics Expert reviews
Thematic reviews
(eg Humane Design Guide)

41. Studio Llama
I
ntegration of different inputs is important.
Optimising for one single metric
or goal is very likely missing the
wider picture. Seek different
inputs (qualitative and
quantitative), integrate these
different findings, understand
what you can and cannot state
based on this. Make dashboards
and overviews.

42. Studio Llama
« »
Erika Hall
Offscreen 18
The best way to assess a functional design is through
a combination of quantitative and qualitative
methods.

43. Agreeing on
balanced goals
& metrics is the
foundation to a
good design
process.
THOUGHT 4
But it’s also a
moving target.

44. Studio Llama
Working with goals & metrics.
Goals & metrics should be worked with and evolve over time.
Make goals & metrics something that evolves as you learn more about the project. It should
be a living document: re-asses, redefine, re-tweak, and do that often.
Remember surrogation & blindness to alternatives.
Try not to get hung up too much on what we have chosen in the beginning — and don’t
make your metrics more important than they should really be.

45. Goals &
metrics don’t
have to be
positive. They
can also be
risks to be
monitored.
THOUGHT 5

46. Studio Llama
Working with risk monitoring.
Avoiding a risk is an important goal.
Goals shouldn’t just be about gains to be made — but also risks to be avoided. This is
especially important when designing in sensitive areas or for “vulnerable groups” (e.g.
children).

47. The process
might be
di
ffi
cult. But
remember:
li
tt
le strokes
fell big oaks.
THOUGHT 6
Be patient,
accept this as
an ongoing
process and
think in small
steps.

48. Studio Llama
What to expect from this process
Convincing others can be tiring.
When bringing in this new perspective, it can appear as if you are overcomplicating things
to others. You’ll find arguments and strategies to convince them over time and bring this
perspective to the table.
Finding the right metric may not be easy.
Even metrics happening within an app or a service are not always easy to interpret. Tracking
goals that happen on a wider, societal level may be practically impossible. Setting up the
goal will already help in the design process. Qualitative methods may at least offer you
some way to learn about the goals.

49. Studio Llama
What to expect from this process
Think in small steps.
You may find many goals or risks — with potentially more metrics — involved in your
design. Start small and pick only a few, the most important to start with. Try to be balanced
with your goals from the start.

50. Studio Llama
Questions, so far? :-)

51. Studio Llama
Workshop part.
Let’s try this!