Google created a new discipline called Decision Intelligence Engineering to help humans make better decisions at scale. It combines data science, psychology, neuroscience and other fields. Google's chief decision officer Cassie Kozyrkov developed a framework for decision making that involves defining metrics and acceptable risks before looking at data. This helps reduce biases. The framework involves determining a default choice with no data, defining ideal metrics to make the decision, and then determining if the available data meets those metrics while accounting for uncertainty. Applying this process can improve decisions, especially for complex problems or automated decisions made at scale.

1. Why Google defined a new discipline to help humans make
decisions
Machine-learning systems are only as smart as their training
data. So Google formalized the marshaling of hard and soft
sciences that go into its decisions.
By Ciara Byrne
Cassie Kozyrkov is Google’s first-ever chief decision officer.
She has already trained 17,000 Googlers to make better
decisions by augmenting data science with psychology,
neuroscience, economics, and managerial science. Now Google
wants to share this new discipline–which it calls Decision
Intelligence Engineering–with the world.
At Google, the need for someone like Kozyrkov stemmed from
the company’s adoption of machine-learning technology across
an array of products and services to make decisions reliably and
at massive scale. A Machine Learning model which decides if a
photo of an animal is a cat, can trigger actions accordingly: If
it’s a cat, do A. If it’s not a cat, do B. And it can do it over and
over without ongoing human involvement.
The problem is that an algorithm which learns from examples–
in this case, photos of animals which are labeled as cats or not
cats–is only as good as the examples it’s trained on. If the
human being training the algorithm sometimes labels rabbits as
cats, the algorithm will make bad decisions as efficiently as it
does good ones. And the more sophisticated machine-learning
applications get, the more opportunity there is for humans to
introduce subtle problems into the final results.
Google needed a decision-making framework which enabled
individual humans, groups of humans, and machines to make
wise decisions. Such a process didn’t yet exist. So the company
decided to build it.
DECISION INTELLIGENCE ENGINEERING
The well-established academic field of decision science covers
the psychology, neuroscience, and economics of how human

2. beings make decisions–but it doesn’t encompass the engineering
perspective and the scale of automated decision-making.
Likewise, data science doesn’t cover how humans think through
a decision.
“A lot of the training that data scientists have assumes that the
decision maker knows exactly what they need and the question
and problem are framed perfectly,” says Kozyrkov. “The data
scientist goes off and collects the data in service of that
question, and answers it, or builds the machine learning system
to implement it.”
That ideal scenario is all too rare in the real world. While
working with Google’s data-science consulting arm, Kozyrkov
often saw executives make decisions that were steered by
unconscious bias rather than by the data itself.
Kozyrkov’s graduate training spans psychology, neuroscience,
and statistics. Instead of just training decision-makers in data
science, she set out to draw on the behavioral sciences to help
them to make truly data-driven decisions. This means framing a
decision effectively–often before looking at any data at all.
HOW TO DECIDE
The first step in Google’s framework asks decision-makers to
determine how they will make the decision with no additional
information. What would the default choice be? Let’s say you
have to decide whether to stay at a hotel. You have photos of
the hotel but no guest reviews. Based on the photos alone,
would you stay there?
“We pretend that we don’t have a preference, but we’re really
lying to ourselves,” says Kozyrkov. “We do have some kind of
innate feeling about what seems to be the safer, better option
under ignorance.”
The second step is to define how you would make the decision
if you had access to any information you wanted. What would it
take to convince you to stay at the hotel? Would you want to
read every review or just see the average review score? If you
use average review score only, does that number need to be 4.2
or 4.5 or something else to convince you to stay at the hotel?

3. This soul-searching exercise determines the metrics you need to
make the decision and the cut-off point for each metric. In the
hotel example, this could be end up being an average review
score of more than 4.2 or a more complex, compound bottom
line using the average rating weighted with one star reviews and
the possibility of bedbugs.
“That step is actually skipped a lot in industry, where people
will use fuzzy concepts, they’ll never make them concrete, and
they won’t own up to the fact that they’re using them,” says
Kozyrkov. “They’ll think that putting a bunch of mathematics
near it fixes it.”
In the final step of this process, you should look at whether you
can get access to all the data you ideally want to make the
decision. If you’ve decided that you’ll stay at any hotel that has
an average star rating of over 4.2, and you have access to hotel
ratings, then you’re good to go. But if you’re factoring in
bedbugs–but don’t have the full reviews which might mention
them–then you have to make your decision under uncertainty.
That introduces the potential for mistakes–and in automated
decision-making, you might make mistakes many times.
As the decision maker, you must therefore consider which
mistakes you can live with. Is it worse if you end up in a hotel
with bedbugs or if you miss out on a hotel which would have
best met all your criteria? How costly is one mistake versus the
other? In the case of automated decision making, only when you
have figured out which risks you are willing to accept can a
data scientist gather relevant data and apply statistical analysis
to help you to make a decision.
BEYOND DATA SCIENCE
Actually, Kozyrkov says, social scientists are often better
equipped than data scientists to translate the intuitions and
intentions of a decision-maker into concrete metrics. And
ideally, data scientists and social scientists work together to
define metrics, collect appropriate data, and apply it to
automated decision-making.
“I think that we don’t realize how valuable social scientists

4. are,” she says. “A data scientist might think that they’re
qualified to create a survey (e.g. to measure user satisfaction),
and then analyze data from it, but what happens if your users
simply ignore your survey? We call that non-response bias.
What if there’s some incentive for your users to lie to you?
That’s not something that a data scientist is trained in handling,
whereas if you worked in a social psychology lab, you deal with
that day in and day out.”
Making decisions the Google way isn’t necessarily easy.
Applying Decision Intelligence Engineering to an important,
complex decision can take weeks, or, if multiple stakeholders
are involved, even months. But with any luck, the result should
be better, wiser decisions, especially at scale.
“We also figure out the right approach based on the importance
of the decision,” says Kozyrkov. “A lot of the training
prioritizes the most important decisions, but there are also
approaches to taking decisions based on … no information at
all, if the decision is not that important.”
Kozyrkov argues that Decision Intelligence Engineering is not
just for experts like data scientists and social scientists.
Everyone in a company can be involved in decision making
using data. The trick is figuring out how each person can best
contribute.
“Decision-making is something that our species does,” says
Kozyrkov. “Everyone knows something about it, and I would
say that everyone is an expert in at least some piece of the
process.”
From the instructor: Google has a special website to teach AI.
Check it out here: https://ai.google/education/
Reference
Byrne, C. (2018). Why Google defined a new discipline to help
humans make decisions, retrieved from
https://www.fastcompany.com/90203073/why-google-defined-a-
new-discipline-to-help-humans-make-decisions

5. Let Curiosity Drive: Fostering Innovation in Data Science
ERIC COLSON, Chief Algorithms Officer, Stitch Fix
January 18, 2019 - San Francisco, CA
Source:
https://multithreaded.stitchfix.com/blog/2019/01/18/fostering-
innovation-in-data-science/
The real value of data science lies not in making existing
processes incrementally more efficient but rather in the creation
of new algorithmic capabilities that enable step-function
changes in value. However, such capabilities are rarely asked
for in a top-down fashion. Instead, they are discovered and
revealed through curiosity-driven tinkering by data scientists.
For companies ready to jump on the data science bandwagon I
offer this advice: think less about how data science will support
and execute your plans and think more about how to create an
environment to empower your data scientists to come up with
ideas you’ve never dreamed of.
At Stitch Fix, we have more than 100 data scientists who have
created several dozens of algorithmic capabilities, generating
100s of millions of dollars in benefits. We have algorithms for
recommender systems, merchandise buying, inventory
management, client relationship management, logistics,
operations—we even have algorithms for designing clothes!
Each provides material and measurable returns, enabling us to
better serve our clients, while providing a protective barrier
against competition. Yet, virtually none of these capabilities
were asked for—not by executives, product managers, or
domain experts, and not even by a data science manager.
Instead, they were born out of curiosity and extracurricular
tinkering by data scientists.
Data scientists are a curious bunch, especially the talented ones.
They work towards stated goals, and they are focused on and
accountable to achieving certain performance metrics. But they

6. are also easily distracted—in a good way. In the course of doing
their work they stumble on various patterns, phenomena, and
anomalies that are unearthed during their data sleuthing. This
goads the data scientist’s curiosity: “Is there a latent dimension
that can characterize a client’s style?” “If we modeled clothing
fit as a distance measure could we improve client feedback?”
“Can successful features from existing styles be systematically
re-combined to create better ones?” Such curiosity questions
can be insatiable and the data scientists knows the answers lie
hidden in the reams of historical data. Tinkering ensues. They
don’t ask permission (eafp). In some cases, explanations can be
found quickly, in only a few hours or so. Other times, it takes
longer because each answer evokes new questions and
hypotheses, leading to more tinkering. But the work is
contained to non-sanctioned side-work, at least for now. They’ll
tinker on their own time if they need to—evening and weekend
if they must. Yet, no one asked them to; curiosity is a powerful
force.
Are they wasting their time? No! Data science tinkering is
typically accompanied by evidence for the merits of the
exploration. Statistical measures like AUC, RMSE, and R-
squared quantify the amount of predictive power the data
scientist’s exploration is adding. They are also equipped with
the business context to allow them to assess viability and
potential impact of a solution that leverages their new insights.
If there is no “there” there, they stop. But, when compelling
evidence is found and coupled with big potential, the data
scientist is emboldened. The exploration flips from being
curiosity-driven to impact-driven. “If we incorporate this latent
style space into our styling algorithms we can better recommend
products.” “This fit feature will materially increase client
satisfaction.” “These new designs will do very well with this
client segment.” Note the difference in tone. Much of the
uncertainty has been allayed and replaced with potential impact.
No longer satisfied with mere historical data, the data scientist

7. is driven to more rigorous methods—randomized controlled
trials or “AB Testing,” which can provide true causal impact.
She wants to see how her insights perform in real life. She
cobbles together a new algorithm based on the newly revealed
insights and exposes it to a sample of clients in an experiment.
She’s already confident it will improve the client experience
and business metrics, but she needs to know by how much. If
the experiment yields a big enough win, she’ll roll it out to all
clients. In some cases, it may require additional work to build a
robust capability around her new insights. This will almost
surely go beyond what can be considered “side work” and she’ll
need to collaborate with others for engineering and process
changes. But she will have already validated her hypothesis and
quantified the impact, giving her a clear case for its
prioritization within the business.
The essential thing to note here is that no one asked the data
scientists to explore. Managers, PMs, domain experts—none of
them saw the unexplained phenomenon that the data scientist
stumbled upon. This is what tipped her off to start tinkering.
And, the data scientist didn’t have to ask permission to explore
because it’s low-cost enough that it just happens fluidly in the
course of their work, or they are compelled by curiosity to flesh
it out on their own time. In fact, if they had asked permission to
explore their initial itch, managers and stakeholders probably
would have said “no.” The insights and resulting capabilities
are often so unintuitive and/or esoteric that, without the
evidence to support it, they don’t seem like a good use of time
or resources.
These two things—low cost exploration and empirical
evidence—set data science apart from other business functions.
Sure, other departments are curious too: “I wonder if clients
would respond better to this this type of creative?” a marketer
might ponder. “Would a new user interface be more intuitive?”
a product manager inquires, etc. But those questions can’t be

8. answered with historical data. Exploring those ideas requires
actually building something, which is costly. And justifying the
cost is often difficult since there’s no evidence that suggests the
ideas will work. But with data science’s low-cost exploration
and risk-reducing evidence, more ideas are explored which, in
turn, leads to more innovation.
Sounds great, right? It is! But this doesn’t happen by will alone.
You can’t just declare as an organization that “we’ll do this
too.” This is a very different way of doing things. Many
established organizations are set up to resist change. Such a new
approach can create so much friction with the existing processes
that the organization rejects it in the same way antibodies attack
a foreign substance entering the body. It’s going to require
fundamental changes to the organization that extend beyond the
addition of a data science team. You need to create an
environment in which it can thrive.
First, you have to position data science as its own entity. Don’t
bury it under another department like marketing, product,
engineering, etc. Instead, make it its own department, reporting
to the CEO. In some cases the data science team can be
completely autonomous in producing value for the company. In
other cases, it will need to collaborate with other departments to
provide solutions. Yet, it will do so as equal partners—not as a
support staff that merely executes on what is asked of them.
Recall that most algorithmic capabilities won’t be asked for;
they are discovered through exploration. So, instead of
positioning data science as a supportive team in service to other
departments, make it responsible for business goals. Then, hold
it accountable to hitting those goals—but enable the data
scientists to come up with the solutions.
Next, you need to equip the data scientists with all the technical
resources they need to be autonomous. They’ll need full access
to data as well as the compute resources to process their

9. explorations. Requiring them to ask permission or request
resources will impose a cost and less exploration will occur. My
recommendation is to leverage a cloud architecture where the
compute resources are elastic and nearly infinite.
The data scientists will also need to have the skills to provision
their own processors and conduct their own exploration. They
will have to be great generalists. Most companies divide their
data scientists into teams of functional specialists—say,
Modelers, Machine Learning Engineers, Data Engineers, Causal
Inference Analysts, etc. While this may provide greater focus, it
also necessitates coordination among many specialists to pursue
any exploration. This increases costs and fewer explorations
will be conducted. Instead, leverage “full-stack data scientists”
that possess varied skills to do all the specialty functions. Of
course, data scientists can’t be experts in everything. Providing
a robust data platform can help abstract them from the
intricacies of distributed processing, auto-scaling, graceful
degradation, etc. This way the data scientist focuses more on
driving business value through testing and learning, and less on
technical specialty. The cost of exploration is lowered and
therefore more things are tried, leading to more innovation.
Finally, you need a culture that will support a steady process of
learning and experimentation. This means the entire company
must have common values for things like learning by doing,
being comfortable with ambiguity, and balancing long- and
short-term returns. These values need to be shared across the
entire organization as they cannot survive in isolation.
Before you jump in and implement this at your company, be
aware that it will be hard if not impossible to implement at an
older, more established company. I’m not sure it could have
worked, even at Stitch Fix, if we hadn’t enabled data science to
be successful from the very beginning. Data Science was not
“inserted” into the organization. Rather, data science was native

10. to us even in the formative years, and hence, the necessary
ways-of-working are more natural.
This is not to say data science is necessarily destined for failure
at older, more mature companies, although it is certainly harder
than starting from scratch. Some companies have been able to
pull off miraculous changes. It’s too important not to try. The
benefits of this model are substantial, and for companies that
have the data assets to create a sustaining competitive
advantage through algorithmic capabilities, it’s worth
considering whether this approach can work for you.
Postscript
People often ask me, “Why not provide the time for data
scientists to be creative like Google’s 20 percent time?” We’ve
considered this several times after seeing many successful
innovations emerge from data science tinkering. In spirit, it’s a
great idea. Yet, we have concerns that a structured program for
innovation may have unintended consequences.
Such programs may be too open-ended and lead to research
where there is no actual problem to solve. The real examples I
depicted above all stemmed out of observation—patterns,
anomalies, an unexplained phenomenon, etc. They were
observed first and then researched. It’s less likely to lead to
impact the other way around.
Structured programs may also set expectations too high. I
suspect there would be a tendency to think of the creative time
as a PhD dissertation, requiring novelty and a material
contribution to the community (e.g., “I’d better consult with my
manager on what to spend my 20 percent time on”). I’d prefer a
more organic process that is driven from observation. The data
scientists should feel no shame in switching topics often or
modifying their hypotheses. They may even find that their
stated priorities are the most important thing they can be doing.

11. So, instead of a structured program, let curiosity drive. By
providing ownership of business goals and generalized roles,
tinkering and exploration become a natural and fluid part of the
role. In fact, it’s hard to quell curiosity. Even if one were to
explicitly ban curiosity projects, data scientists would simply
conduct their explorations surreptitiously. Itches must be
scratched!
Reference
Colson, E. (2019). Let Curiosity Drive: Fostering Innovation in
Data Science, retrieved from
https://multithreaded.stitchfix.com/blog/2019/01/18/fostering-
innovation-in-data-science/