The document provides guidance on effectively defining questions, doing the work to answer questions, and drawing conclusions from data analysis projects. It emphasizes clarifying the question being asked, exploring all factors that could influence results, and acknowledging limitations to avoid making unjustified claims from the data. An example analysis is included that shows households who used remote deposit generated higher revenue but also deposited more checks and were generally more engaged, so more factors contributed to the relationship. The analysis concludes RDC may slightly reduce retention based on satisfaction scores.

1. Setting Expectations on
Data Science Projects:
What am I doing and why am I doing it?
March 21, 2019

2. 2
“Hey Kathleen,
Can you get me X data?
It should be really simple.
I just need the data.”

3. The game changer…
“I need the data that shows that client experience
increased because of the new Preferred Call Center.”
Historical Approach Better Approach
3

4. The game changer…
“I need the data that shows that client experience
increased because of the new Preferred Call Center.”
“Great! Let’s spend
another $7MM to
double capacity.”
“Here’s your data.”
Historical Approach Better Approach
4

5. The game changer…
“I need the data that shows that client experience
increased because of the new Preferred Call Center.”
“Great! Let’s spend
another $7MM to
double capacity.”
“Benefit was driven by
your talent selection,
not the $7MM.”
“Let’s expand with
the right people.”
“Here’s your data.”
Historical Approach Better Approach
5

6. 1. Define the
question
2. Do the work
(and draw a
conclusion)
3. Answer the
question
“Hey Kathleen,
Can you get me X data?
It should be really simple. I
just need the data.”
6

7. Hear the question differently, and trust yourself
“Hey Kathleen,
I need your help with
something.”
7
Part 1: Define the question

8. Don’t say “yes,” but definitely don’t say “no”
“Help me understand
what you are trying to
achieve so I can better
meet your needs.”
8
Part 1: Define the question

9. Ask open-ended questions, clarify as you go, and trust your
business partner
“Ok here’s what’s really going on… But don’t say
anything because it’s confidential…”
Who else is working on this?
What other theories are there?
What’s different about this?
What did you learn last time?
What level of precision?
If we do X, couldn’t it impact Y?
Is it possible that…?
What is your hypothesis?
What is your concern?
What changed?
What did you expect?
Who asked the question?
What prompted it?
What do you think?
9
Part 1: Define the question

10. Work backward and focus on the story
I ask two questions…
1. What’s my conclusion?
2. How do I know?
10
Part 3: Answer the question

11. Talk it out, and use conversational language for your first draft
11
Part 3: Answer the question

12. Give yourself plenty of time, and plan for multiple iterations
“I have made this letter longer than
usual, only because I have not had the
time to make it shorter.”
-- Blaise Pascal
12
Part 3: Answer the question

13. 13
Iteration #1
Objective – Understand difference in revenue based on RDC use
Methodology/Period – Test vs. pseudo-control (Q2 to Q3)
Note – This is a historical “statement of fact”. It does not imply
causality, other confounding factors could also contribute
Test/Pseudo-Control Groups – Both test (N=27k) and pseudo-
control groups (N=40k) are households that were open, mobile
active, and depositing checks during both periods. Clients in the
test group started using RDC in the post period; clients in the
pseudo-control group never utilized RDC. Test and pseudo-controls
are balanced by demographics based on pre-period data
Result –
• Revenue – Households that started utilizing RDC have an
increase in revenue. The increase in revenue could be driven by
the increasing check deposits for the households who started
utilizing RDC
• RDC Usage – Households that deposit checks through RDC as
their main channel deposit fewer checks overall. No correlation
between change in revenue and RDC usage is observed.
Application – Of n retail households open, mobile active, as well
as depositing checks during Q3, m of them were using RDC at the
time. The “value” of households who utilized remote deposit during
Q3 would roughly translate to $x +/- $y
Define the question

14. 14
Iteration #1
Objective – Understand difference in revenue based on RDC use
Methodology/Period – Test vs. pseudo-control (Q2 to Q3)
Note – This is a historical “statement of fact”. It does not imply
causality, other confounding factors could also contribute
Test/Pseudo-Control Groups – Both test (N=27k) and pseudo-
control groups (N=40k) are households that were open, mobile
active, and depositing checks during both periods. Clients in the
test group started using RDC in the post period; clients in the
pseudo-control group never utilized RDC. Test and pseudo-controls
are balanced by demographics based on pre-period data
Result –
• Revenue – Households that started utilizing RDC have an
increase in revenue. The increase in revenue could be driven by
the increasing check deposits for the households who started
utilizing RDC
• RDC Usage – Households that deposit checks through RDC as
their main channel deposit fewer checks overall. No correlation
between change in revenue and RDC usage is observed.
Application – Of n retail households open, mobile active, as well
as depositing checks during Q3, m of them were using RDC at the
time. The “value” of households who utilized remote deposit during
Q3 would roughly translate to $x +/- $y
Define the question
Do the work

15. 15
Iteration #1
Objective – Understand difference in revenue based on RDC use
Methodology/Period – Test vs. pseudo-control (Q2 to Q3)
Note – This is a historical “statement of fact”. It does not imply
causality, other confounding factors could also contribute
Test/Pseudo-Control Groups – Both test (N=27k) and pseudo-
control groups (N=40k) are households that were open, mobile
active, and depositing checks during both periods. Clients in the
test group started using RDC in the post period; clients in the
pseudo-control group never utilized RDC. Test and pseudo-controls
are balanced by demographics based on pre-period data
Result –
• Revenue – Households that started utilizing RDC have an
increase in revenue. The increase in revenue could be driven by
the increasing check deposits for the households who started
utilizing RDC
• RDC Usage – Households that deposit checks through RDC as
their main channel deposit fewer checks overall. No correlation
between change in revenue and RDC usage is observed.
Application – Of n retail households open, mobile active, as well
as depositing checks during Q3, m of them were using RDC at the
time. The “value” of households who utilized remote deposit during
Q3 would roughly translate to $x +/- $y
Define the question
Do the work
Draw a conclusion

16. 16
Iteration #1
Objective – Understand difference in revenue based on RDC use
Methodology/Period – Test vs. pseudo-control (Q2 to Q3)
Note – This is a historical “statement of fact”. It does not imply
causality, other confounding factors could also contribute
Test/Pseudo-Control Groups – Both test (N=27k) and pseudo-
control groups (N=40k) are households that were open, mobile
active, and depositing checks during both periods. Clients in the
test group started using RDC in the post period; clients in the
pseudo-control group never utilized RDC. Test and pseudo-controls
are balanced by demographics based on pre-period data
Result –
• Revenue – Households that started utilizing RDC have an
increase in revenue. The increase in revenue could be driven by
the increasing check deposits for the households who started
utilizing RDC
• RDC Usage – Households that deposit checks through RDC as
their main channel deposit fewer checks overall. No correlation
between change in revenue and RDC usage is observed.
Application – Of n retail households open, mobile active, as well
as depositing checks during Q3, m of them were using RDC at the
time. The “value” of households who utilized remote deposit during
Q3 would roughly translate to $x +/- $y
Define the question
Do the work
Draw a conclusion
Answer the question

17. 17
Final Iteration
Households who started using RDC generated
$x more revenue during Q3 compared to
households that didn’t.
This is not necessarily a causal relationship –
households that started using RDC also…
• Deposited more check overall
• Had a higher level of engagement in general
RDC appears to have a slight negative impact on
retention
• Lower CSAT observed in households that used
RDC suggests a poor RDC experience may have
contributed to client attrition
Households that utilized RDC as their main
channel for check deposit had the highest
satisfaction score in RDC
• These households, however, also deposit fewer
checks on average
Answer the question

18. 18
Final Iteration
Households who started using RDC generated
$x more revenue during Q3 compared to
households that didn’t.
This is not necessarily a causal relationship –
households that started using RDC also…
• Deposited more check overall
• Had a higher level of engagement in general
RDC appears to have a slight negative impact on
retention
• Lower CSAT observed in households that used
RDC suggests a poor RDC experience may have
contributed to client attrition
Households that utilized RDC as their main
channel for check deposit had the highest
satisfaction score in RDC
• These households, however, also deposit fewer
checks on average
1. What’s my conclusion?
Answer the question
2. How do I know?

19. 19
Final Iteration
Households who started using RDC generated
$x more revenue during Q3 compared to
households that didn’t.
This is not necessarily a causal relationship –
households that started using RDC also…
• Deposited more check overall
• Had a higher level of engagement in general
RDC appears to have a slight negative impact on
retention
• Lower CSAT observed in households that used
RDC suggests a poor RDC experience may have
contributed to client attrition
Households that utilized RDC as their main
channel for check deposit had the highest
satisfaction score in RDC
• These households, however, also deposit fewer
checks on average
1. What’s my conclusion?
2. How do I know?
1. What’s my conclusion?
2. How do I know?
Answer the question

20. 20
Final Iteration
Households who started using RDC generated
$x more revenue during Q3 compared to
households that didn’t.
This is not necessarily a causal relationship –
households that started using RDC also…
• Deposited more check overall
• Had a higher level of engagement in general
RDC appears to have a slight negative impact on
retention
• Lower CSAT observed in households that used
RDC suggests a poor RDC experience may have
contributed to client attrition
Households that utilized RDC as their main
channel for check deposit had the highest
satisfaction score in RDC
• These households, however, also deposit fewer
checks on average
1. What’s my conclusion?
Answer the question
2. How do I know?
1. What’s my conclusion?
2. How do I know?
1. What’s my conclusion?
2. How do I know?

21. 1. Hear the question differently, and trust
yourself
2. Don’t say “yes,” but definitely don’t say
“no”
3. Ask open-ended questions, clarify as you
go, and trust your business partner
1. Define the question
1. Work backward and focus on the story
2. Talk it out, and use conversational
language for your first draft
3. Give yourself plenty of time, and plan for
multiple iterations
3. Answer the question
21
In summary…