Most of us start projects with good intentions—we want to make things welcoming, seamless, and maybe even fun to use. But too often, toxic cultures within tech result in products that have all sorts of biases embedded in them: “smart scales” that assume everyone wants to lose weight, form fields that fail for queer people, résumé-reading algorithms that are biased against women, image-recognition software that doesn’t work for people of color. As tech becomes increasingly central to users' days—and intertwined with their most intimate lives—UX folks have more responsibility than ever to consider who could be harmed or left out by their decisions. In this talk, we’ll take a hard look at how our industry’s culture—its lack of diversity, its “fail fast” ethos, its obsession with engagement, and its chronic underinvestment in understanding the humans it’s designing for—creates products that perpetuate bias, manipulate and harm users, undermine democracy, and ultimately wreak havoc.

1. 8:00 p.m.

3. 9:03 p.m.

4. To recap:
• There’s no way to turn it off
• This is dangerous for people with eating disorders
• This feels shamey
• “Average” calorie counts are wildly inaccurate
• Not all calories are equal
• A cupcake is not a useful metric
• Pink cupcakes are not neutral—they have social and culture
encoding (feminine, white, middle class)
• This perpetuates diet culture

5. It took one hour to
document these flaws.

6. It took three hours to
shut the feature down.

7. How many hours did it
take to build?

8. flickr.com/photos/3059349393/8000156483/ (CC BY 2.0)

9. SMALL CHOICES

10. From Dan Hon

11. From Dan Hon

13. Patrick Gerard

14. From @sallyrooney

15. ‘‘We talked about getting rid of it but it performs
kinda great :/
—Tag Savage, lead writer at Tumblr

16. BIG CONSEQUENCES

17. Screensh0t of a knockoff Peppa Pig video: James Bridle

18. • Cartoons about torture and suicide
• Sexualized characters and themes
• Violence and weapons
• Kids being tied up and hurt
• Kids vomiting and writhing in pain

20. ‘‘
Peterson is not an alt-right figure and cannot be held
responsible for the “recommended” content that his
viewers come across on YouTube. But YouTube, and its
parent company, Google, should be…
Within just a few clicks of supposedly “related”
content, a viewer watching a Peterson lecture can end
up on a video entitled “How Savage Are Blacks In
America & Why Is Everyone Afraid To Discuss It?”
—Sam Levin in The Guardian

21. Altena, Germany
Photo of Altena: Ksenia Kuleshova for The New York Times
x

22. ‘‘
Towns where Facebook use was higher than average, like
Altena, reliably experienced more attacks on refugees.
That held true in virtually any sort of community—big
city or small town; a!luent or struggling; liberal haven or
far-right stronghold… Wherever per-person Facebook use
rose to one standard deviation above the national average,
attacks on refugees increased by about 50 percent.
—Amanda Taub & Max Fisher,
The New York Times

23. ‘‘
Towns where Facebook use was higher than average, like
Altena, reliably experienced more attacks on refugees.
That held true in virtually any sort of community—big
city or small town; a!luent or struggling; liberal haven or
far-right stronghold… Wherever per-person Facebook use
rose to one standard deviation above the national average,
attacks on refugees increased by about 50 percent.
—Amanda Taub & Max Fisher,
The New York Times

24. ‘‘
[The] algorithm is built around a core mission: promote
content that will maximize user engagement. Posts that
tap into negative, primal emotions like anger or fear,
studies have found, perform best and so proliferate.
That is how anti-refugee sentiment… can seem
unusually common on Facebook, even in a pro-refugee
town like Altena.
—Amanda Taub & Max Fisher,
The New York Times

26. Our systems aren’t just
vulnerable to abuse.
They’re optimized for it.

27. EMBEDDED
BIASES

28. Video: Chukwuemeka Afigbo

29. From Jacky Alciné

30. From Jacky Alciné

31. ‘‘We’re also working on longer-term fixes around both
linguistics (words to be careful about in photos of
people)… and image recognition itself (e.g., better
recognition of dark-skinned faces).
—Yonatan Zunger, former Googler

32. ‘‘We’re also working on longer-term fixes around both
linguistics (words to be careful about in photos of
people)… and image recognition itself (e.g., better
recognition of dark-skinned faces).
—Yonatan Zunger, former Googler

33. ‘‘When the person in the photo is a white man, the
software is right 99 percent of the time. But the darker
the skin, the more errors arise — up to nearly 35
percent for images of darker skinned women.
—“Facial Recognition Is Accurate, if You’re a White Guy”
by Steve Lohr, The New York Times

34. ‘‘When the person in the photo is a white man, the
software is right 99 percent of the time. But the darker
the skin, the more errors arise —up to nearly 35
percent for images of darker skinned women.
—“Facial Recognition Is Accurate, if You’re a White Guy”
by Steve Lohr, The New York Times

36. • Word2vec: Natural language processing
tool trained using 3 million words from
Google News articles.
• Can complete analogies: Paris is to
France as Tokyo is to _______.
• Thinks man is to computer programmer
as woman is to homemaker.

38. • COCO: More than 100,000 images from
the web, labeled.
• AI trained on it can identify objects in
those images.

39. • COCO: More than 100,000 images from
the web, labeled.
• AI trained on it can identify objects in
those images.
• It associates kitchen implements with
women.

40. If we’re not explicitly
removing bias, we’re
reinforcing it.

41. A WAY FORWARD

42. Practice
Process
Priorities

43. Practice
Process
Priorities

44. ASSUME

46. • The user’s weight is “extra pounds.”
• The user is trying to lose weight.
• The user feels bad about having
gained weight.
• The user wants help losing weight.

47. • Identity
• Location
• Emotional state
• Physical state
• Personal history
• Lifestyle
• Goals
• Pain points

49. ‘‘The challenge is that government, and local government
is no exception, was designed by old, white guys….
Some things have changed. Oakland in particular is
led by badass women and people of color who are
trying to change an organization that wasn’t designed
for them.
—Mai-Ling Garcia, City of Oakland

50. Why it matters:
• Redlining (denying home loans in “risky” areas) led to
segregation, low home ownership rates, and lack of
community investment.
• 19% of English-speaking adults lack basic literacy
skills. 22% speak English less than “very well.”
• Average reading level on public info sites was post-grad.
• Residents speak over 100 languages.
• Oakland is a sanctuary city.

51. ‘‘We would talk about ability and accessibility instead
of handicaps. We would capitalize the word Black or
Brown. We would strip pronouns from our examples.
The word citizen was reframed strictly as a legal
requirement, not an alternative term for resident.
—Mai-Ling Garcia, City of Oakland

52. Practice
Process
Priorities

53. If it’s important,
it’s planned.

54. If it’s important,
it’s resourced.

55. If it’s important,
it’s evaluated.

56. ‘‘Lean methodology and the Minimum Viable Product
technique are supposed to help reduce waste and
increase the timely flow of useful feedback. In
practice, they are used as cover for rushing to a less
thoughtful solution without considering the context
or the long-term implications.
—Erika Hall, “Thinking in Triplicate”

57. Make inclusive design and worst-case scenario
planning an explicit part of:
• Project roles and responsibilities
• Roadmapping and project planning
• Use cases and scenarios
• Content crits and editing cycles
• Project postmortems
• Employee evaluations

58. Practice
Process
Priorities

59. Image: Pixabay (CC0)

60. Image: Pixabay (CC0)

61. ‘‘We’re an idealistic and optimistic company. For the
first decade, we really focused on all the good that
connecting people brings… But it’s clear now that we
didn’t do enough. We didn’t focus enough on
preventing abuse and thinking through how people
could use these tools to do harm as well.
—Mark Zuckerberg, April 2018

62. ‘‘The ugly truth is that we believe in connecting people
so deeply that anything that allows us to connect more
people more often is *de facto* good. That’s why all the
work we do in growth is justified. All the questionable
contact importing practices. All the subtle language
that helps people stay searchable by friends… All of it.
—Facebook VP Andrew “Boz” Bosworth, 2016

63. ‘‘The ugly truth is that we believe in connecting people
so deeply that anything that allows us to connect more
people more often is *de facto* good. That’s why all the
work we do in growth is justified. All the questionable
contact importing practices. All the subtle language
that helps people stay searchable by friends… All of it.
—Facebook VP Andrew “Boz” Bosworth, 2016

64. Image: Josh Mattson (CC - BY NC)
$40 billion

65. ‘‘The problems the company is facing today are due to
tens of thousands of small decisions made over the
last decade within an incentive structure that was not
predicated on our 2018 threat profile.
—Alex Stamos, departing Facebook CSO

66. ‘‘The problems the company is facing today are due to
tens of thousands of small decisions made over the
last decade within an incentive structure that was not
predicated on our 2018 threat profile.
—Alex Stamos, departing Facebook CSO

70. If we can only discuss ethics
when it’s revenue-neutral,
we don’t have ethics.

71. IT’S TIME
TO LEAD

72. Our work is entangled
with people’s most
intimate lives.

73. We have a responsibility
to slow down, take stock,
and do better.

74. What are you willing to risk
to make that happen?

75. Books/Speaking
sarawb.com
Consulting
rareunion.com
Podcast
noyougoshow.com
Thank you.