Copyright © 2014 EMC Corporation. All Rights Reserved..docx

Copyright ©
2014 EM
C Corporation. All Rights Reserved.
Data Analytics Lifecycle
1
M
odule 2: Data Analytics Lifecycle
Copyright ©
2014 EM
U
pon com
pletion of this lesson, you should be able to:
•
Apply the Data Analytics Lifecycle to a case study scenario
•
Fram
e a business problem

as an analytics problem
•
Identify the four m
ain deliverables in an analytics project
M
2
Copyright ©
2014 EM
How
to Approach Your Analytics Problem
s
•
How
do you currently approach
your analytics problem
s?
•
Do you follow

a m
ethodology or
som
e kind of fram
ew
ork?
•
How
do you plan for an analytic
project?
3
M
Your Thoughts?
Copyright ©
2014 EM
•
Focus your tim
e
•

Ensure rigor and com
pleteness
•
Enable better transition to m
em
bers of the cross-functional
analytic team
s
Repeatable
Scale to additional analysts
Support validity of findings
4
“A journey of a thousand m
iles begins w
ith a single step“ (Lao Tzu)
M
Value of U
sing the Data Analytics Lifecycle
Copyright ©
2014 EM

5
1.
W
ell-defined processes
can help guide any analytic
project
2.
Focus of Data Analytics
Lifecycle is on Data Science
projects, not business
intelligence
3.
Data Science projects tend to require a m
ore consultative
approach, and differ in a few
w
ays
M
ore due diligence in Discovery phase
M
ore projects w
hich lack shape or structure
Less predictable data
N
eed For a Process to Guide Data Science Projects

5
M
Copyright ©
2014 EM
Key Roles for a Successful Analytic Project
M
6
Role
Description
Business U
ser
Som
eone w
ho benefits from
the end results
and can consult and advise project team
on
value of end results and how
these w

ill be operationalized
Project Sponsor
Person responsible for the genesis of the project, providing the
im
petus for the project and
core business problem
, generally provides the funding
and w
ill gauge the degree of value
from
the final outputs of the w
orking team
Project M
anager
Ensure key m
ilestonesand objectives are m
et on tim
e and at expected quality.
BusinessIntelligence
Analyst
Businessdom
ain expertise w
ith deep understanding of the data,KPIs, key m

etrics and
business intelligence from
a reporting perspective
Data Engineer
Deep technical skills to assist w
ith tuning SQ
L queries for data m
anagem
ent, extraction and
support data ingest to analytic sandbox
Database
Adm
inistrator (DBA)
Database Adm
inistratorw
ho provisions and configures database environm
ent to support
the analytical needs of the w
orking team
Data Scientist
Provide subject m
atter expertise
for analytical techniques, data m

odeling, applying
valid
analytical techniques to given business problem
s and ensuring overall analytical objectives
are m
et
Copyright ©
2014 EM
M
7
D
iscovery
O
perationalize
M
odel
P
lanning
D

ata P
rep
M
odel
B
uilding
C
om
m
unicate
R
esults
Do I have enough
inform
ation to draft an
analytic plan and share for
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?

Do I have a good idea
about the type of m
odel
to try? Can I refine the
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
1
23
4
65
Copyright ©
2014 EM
Phase 1: Discovery
M

8
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P
rep
M
odel
B
uilding
C
om
m
unicate
R
esults

Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?
about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
•
Learn the Business Dom

ain
Determ
ine am
ount of dom
ain know
ledge needed to orient you to the data and
interpret results dow
nstream
Determ
ine the general analytic problem
type (such as clustering, classification)
If you don’t know, then conduct initial research to learn about
the dom
ain area
you’ll be analyzing
•
Learn from
the past
Have there been previous attem
pts in the organization to solve this problem
?
If so, w
hy did they fail? W
hy are w

e trying again? How
have things changed?
1
Copyright ©
2014 EM
Phase 1: Discovery
M
9
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P

rep
M
odel
B
uilding
C
om
m
unicate
R
esults
Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?

about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
•
Resources
Assess available technology
Available data –
sufficient to m
eet your needs
People for the w
orking team
Assess scope of tim
e for the project in calendar tim
e and person-hours
Do you have sufficient resources to attem
pt the project? If not, can you get
m

ore?
1
Copyright ©
2014 EM
Phase 1: Discovery
M
10
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P
rep

M
odel
B
uilding
C
om
m
unicate
R
esults
Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?
about the type of m

odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
•
Fram
e the problem
…
..Fram
ing is the process of stating the analytics problem
to be solved
State the analytics problem
, w
hy it is im
portant, and to w
hom
Identify key stakeholders and their interests in the project
Clearly articulate the current situation and pain points
O

bjectives –
identify w
hat needs to be achieved in business term
s and w
hat needs
to be done to m
eet the needs
W
hat is the goal? W
hat are the criteria for success? W
hat’s “good enough”?
W
hat is the failure criterion (w
hen do w
e just stop trying or settle for w
hat w
e
have)?
Identify the success criteria, key risks, and stakeholders (such
as RACI)
1
Copyright ©
2014 EM

Tips for Interview
ing the Analytics Sponsor
•
Even if you are “given” an analytic problem
you should w
ork w
ith clients to
clarify and fram
e the problem
You’re typically handed solutions, you need to
identify the problem
and their desired outcom
e
Sponsor Interview
Tips
•
Prepare for the interview
–
draft your questions, review
w
ith colleague, team
•
U

se open-ended questions, don’t ask leading questions
•
Probe for details, follow
-up
•
Don’t fill every silence –
give them
tim
e to think
•
Let them
express their ideas, don’t put w
ords in their m
outh, let them
share their feelings
•
Ask clarifying questions, ask w
hy –
is that correct? Am
I on target? Is there anything else?
•
U
se active listening –
repeat it back to m

ake sure you heard it correctly
•
Don’t express your opinions
•
Be m
indful of your body language and theirs –
use eye contact, be attentive
•
M
inim
ize distractions
•
Docum
ent w
hat you heard and review
it back w
ith the sponsor
1111
M
Copyright ©
2014 EM

Tips for Interview
ing the Analytics Sponsor
Interview
Q
uestions
•
W
hat is the business problem
you’re trying to solve?
•
W
hat is your desired outcom
e?
•
W
ill the focus and scope of the problem
change if the follow
ing dim
ensions
change:
•
Tim

e –
analyzing 1 year or 10 years w
orth of data?
•
People –
how
w
ould this project change this?
•
Risk –
conservative to aggressive
•
Resources –
none to unlim
ited (tools, tech, …
..)
•
Size and attributes of Data
•
W
hat data sources do you have?
•
W
hat industry issues m

ay im
pact the analysis?
•
W
hat tim
elines are you up against?
•
W
ho could provide insight into the project? Consulted?
•
W
ho has final say on the project?
1212
M
Copyright ©
2014 EM
Phase 1: Discovery
M

13
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P
rep
M
odel
B
uilding
C
om
m
unicate
R
esults
Do I have enough

inform
ation to draft an
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?
about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
•
Form

ulate Initial Hypotheses
IH, H
1 , H
2, H
3 , …
H
n
Gather and assess hypotheses from
stakeholders and
dom
ain experts
Prelim
inary data exploration to inform
discussions w
ith
stakeholders during the hypothesis form
ing stage
•
Identify Data Sources –
Begin Learning the Data
Aggregate sources for preview
ing the data and provide
high-level understanding
Review

the raw
data
Determ
ine the structures and tools needed
Scope the kind of data needed for this kind of problem
1
Copyright ©
2014 EM
U
sing a Sam
ple Case Study to Track the Phases in the
Situation Synopsis
•
Retail Bank, Yoyodyne Bank w
ants to im
prove the N
et Present Value
(N
PV) and retention rate of custom
ers

•
They w
ant to establish an effective m
arketing cam
paign targeting
custom
ers to reduce the churn rate by at least five percent
•
The bank w
ants to determ
ine w
hether those custom
ers are w
orth
retaining. In addition, the bank also w
ants to analyze reasons for
custom
er attrition and w
hat they can do to keep them
•
The bank w
ants to build a data w
arehouse to support M

arketing
and other related custom
er care groups
14
M
ini C
ase Study: C
hurn Prediction for
Yoyodyne B
ank
14
M
Copyright ©
2014 EM
How
to Fram
e an Analytics Problem
Sam
ple

Business Problem
s
Q
ualifiers
Analytical
Approach
•
How
can w
e im
prove on x?
•
W
hat’s happening real-tim
e?
Trends?
•
How
can w
e use analytics
differentiate ourselves
•
How
can w

e use analytics to
innovate?
•
How
can w
e stay ahead of our
biggest com
petitor?
W
ill the focus and scope of the problem
change if
the follow
ing dim
ensions change:
•
Tim
e
•
People
–
how
w
ould x change this?
•

Risk –
conservative/aggressive
•
Resources –
none/unlim
ited
•
Size of Data?
Define an analytical
approach, including
key term
s, m
etrics, and
data needed.
Yoyodyne Bank
How
can
w
e im
prove
N
et Present Value (N
PV) and
retention rate of the custom

ers?
•
Tim
e: Trailing 5 m
onths
•
People: W
orking team
and business users
from
the
Bank
•
Risk: the projectw
ill fail if w
e cannot
determ
ine valid predictors of churn
•
Resources: EDW
, analytic
sandbox, O
LTP

system
•
Data:U
se 24 m
onths for the training set, then
analyze 5 m
onths of historical data for those
custom
ers w
ho churned
How
do w
e identify
churn/no churn for a
custom
er?
Pilot study follow
ed
full scale analytical
m
odel
1515
M

C
hurn Prediction for
Yoyodyne B
ank
M
ini C
ase
Study
Copyright ©
2014 EM
Phase 2: Data Preparation
M
16
D
iscovery
O
perationalize
M
odel

P
lanning
D
ata P
rep
M
odel
B
uilding
C
om
m
unicate
R
esults
Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good

quality data to
start building
the m
odel?
about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
•
Prepare Analytic Sandbox
W
ork space for the analytic team
10x+ vs. EDW
•
Perform
ELT
Determ

ine needed transform
ations
Assess data quality and structuring
Derive statistically useful m
easures
Determ
ine and establish data connections
for raw
data
Execute Big ELT and/or Big ETL
•
U
seful Tools for this phase:
•
For D
ata Transform
ation &
C
leansing: S
Q
L, H
adoop, M
apR
educe, A

lpine M
iner
2
Copyright ©
2014 EM
M
17
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P

about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
•
Fam
iliarize yourself w
ith the data thoroughly
List your data sources
W
hat’s needed vs. w
hat’s available
•
Data Conditioning
Clean and norm
alize data
Discern w
hat you keep vs. w

hat you discard
•
Survey &
Visualize
O
verview, zoom
&
filter, details-on-dem
and
Descriptive Statistics
Data Q
uality
•
U
seful Tools for this phase:
•
D
escriptive S
tatistics on candidate variables for diagnostics &
quality
•
Visualization: R
(base package, ggplot and lattice), G
nuP

lot, G
gobi/R
ggobi, S
potfire,
Tableau
2
Copyright ©
2014 EM
Phase 3: M
odel Planning
M
18
D
iscovery
O
perationalize
M
odel

P
lanning
D
ata P
rep
M
odel
B
uilding
C
om
m
unicate
R
esults
Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good
quality data to

start building
the m
odel?
about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
•
Determ
ine M
ethods
Select m
ethods based on hypotheses, data
structure and volum
e
Ensure techniques and approach w
ill m

eet
business objectives
•
Techniques &
W
orkflow
Candidate tests and sequence
Identify and docum
ent m
odeling
assum
ptions
•
U
seful Tools for this phase: R
/P
ostgresS
Q
L, S
Q
L
A
nalytics, A
lpine M

iner, S
A
S
/A
C
C
E
S
S
, S
P
S
S
/O
B
D
C
3
Copyright ©
2014 EM

Phase 3: M
odel Planning
M
19
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P
rep
M
odel
B
uilding
C
om
m

unicate
R
esults
Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?
about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e

failed for sure?
•
Data Exploration
•
Variable Selection
Inputs from
stakeholders and dom
ain
experts
Capture essence of the predictors, leverage
a technique for dim
ensionality reduction
Iterative testing to confirm
the m
ost
significant variables
•
M
odel Selection
Conversion to SQ
L or database language for
best perform
ance
Choose technique based on the end goal

3
Copyright ©
2014 EM
Sam
ple Research: Churn Prediction in O
ther Verticals
M
arket Sector
Analytic Techniques/M
ethods U
sed
W
ireless Telecom
DM
EL m
ethod (data m
ining by evolutionary learning)
Retail Business
Logistic regression, ARD (autom
atic

relevance determ
ination), decision tree
Daily Grocery
M
LR (m
ultiple linear regression), ARD, and decision tree
W
ireless Telecom
N
eural netw
ork, decision tree, hierarchical neurofuzzy system
s, rule evolver
Retail Banking
M
ultiple regression
W
ireless Telecom
Logistic regression, neural netw
ork, decision tree
2020
M
M

ini C
ase Study:
C
hurn Prediction for
Yoyodyne B
ank
•
After conducting research on churn prediction, you have
identified m
any
m
ethods for analyzing custom
er churn across m
ultiple verticals (those in
bold
are taught in this course)
•
At this point, a Data Scientist w
ould assess the m
ethods and select the best
m
odel for the situation

Copyright ©
2014 EM
Phase 4: M
odel Building
M
21
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P
rep
M
odel
B

uilding
C
om
m
unicate
R
esults
Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?
about the type of m
odel
analytic plan?

Is the m
odel robust
enough? Have w
e
failed for sure?
•
Develop data sets for testing, training, and production purposes
N
eed to ensure that the m
odel data is sufficiently robust for the m
odel
and analytical techniques
Sm
aller, test sets for validating approach, training set for initial
experim
ents
•
G
et the best environm
ent you can for building m
odels and
w
orkflow

s…
fast hardw
are, parallel processing
•
U
seful Tools for this phase: R
, P
L/R
, S
Q
L, A
lpine M
iner, S
A
S
E
nterprise M
iner
4
Copyright ©
2014 EM

Phase 5: Com
m
unicate Results
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P
rep
M
odel
B
uilding
C
om
m
unicate

R
esults
Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?
about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?

Did w
e succeed? Did w
e fail?
•
Interpret the results
•
Com
pare to IH’s from
Phase 1
•
Identify key findings
•
Q
uantify business value
•
Sum
m
arizing findings, depending on
audience
5
For the YoyoD
yne C
ase S

tudy,
w
hat w
ould be som
e possible results and key findings?
M
ini C
ase Study:
C
hurn Prediction for
Yoyodyne B
ank
M
22
Copyright ©
2014 EM
Phase 6: O
perationalize

M
23
D
iscovery
O
perationalize
M
odel
P
lanning
D
ata P
rep
M
odel
B
uilding
C
om
m
unicate
R
esults

Do I have enough
inform
ation to draft an
peer review
?
Do I have
enough good
quality data to
start building
the m
odel?
about the type of m
odel
analytic plan?
Is the m
odel robust
enough? Have w
e
failed for sure?
•

Run a pilot
•
Assess the benefits
•
Provide final deliverables
•
Im
plem
ent the m
odel in the
production environm
ent
•
Define process to update, retrain,
and retire the m
odel, as needed
6
Copyright ©
2014 EM
Analytic Plan

24
Com
ponentsof
Analytic Plan
RetailBanking: Yoyodyne
Bank
Phase
1: Discovery
Business Problem
Fram
ed
How
do w
e identify churn/no churn for a custom
er?
InitialHypotheses
Transaction volum
e and type
are key predictors of churn rates.
Data
5 m
onths of custom
er account history.

Phase
3: M
odel Planning
-Analytic
Technique
Logistic regression to identify m
ost influentialfactors predicting churn.
Phase 5:
Result&
Key Findings
O
nce custom
ers stop using their accounts for gas and groceries,they w
ill
soon erode their accounts and churn.
If custom
ers use their debitcard few
er than 5 tim
es per m
onth, they w
ill
leave the bank w
ithin 60 days.

BusinessIm
pact
If w
e can target custom
ers w
ho are high-risk for churn, w
e can reduce
custom
er attrition by 25%
. This w
ould save $3 m
illion in lost of
custom
er revenue and avoid $1.5 m
illion in new
custom
er acquisition
costs each year.
24
M
M
ini C

ase Study:
C
hurn Prediction for
R
etail B
anking
Copyright ©
2014 EM
Key O
utputs from
a Successful Analytic Project, by RoleM
25
Role
Description
W
hat the Role N
eeds in the Final Deliverables
Business U
ser
Som

eone w
ho benefits from
the end resultsand can consult
and advise project team
on value of end results and how
these w
ill be operationalized
•
SponsorPresentation addressing:
•
Are the results good for m
e?
•
W
hatare the benefits of the findings?
•
W
hat are the im
plicationsof this for m
e?
Project
Sponsor
Person responsible for the genesis of the project, providing

the im
petus for the project and
core business problem
,
generally provides the funding
and w
ill gauge the degree of
value from
the final outputs of the w
orking team
•
SponsorPresentation addressing:
•
W
hat’s the business im
pact of doing this?
•
W
hat are the risks? RO
I?
•
How
can this be evangelized w

ithin the
organization (and beyond)?
Project
M
anager
Ensure key m
ilestonesand objectives are m
et on tim
e and at
expected quality.
Business
Intelligence
Analyst
Businessdom
ain expertise w
ith deep understanding of the
data,KPIs, key m
etrics and business intelligence from
a
reporting perspective
•
Show
the analyst presentation
•

Determ
ine if the reports w
ill change
Data Engineer
Deep technical skills to assist w
ith tuning SQ
L queries for
data m
anagem
ent, extraction and support data ingest to
analytic sandbox
•
Share the code
from
the analytical project
•
Create technicaldocum
ent on how
to
im
plem
ent it.
Database
Adm

inistrator
(DBA)
Database Adm
inistratorw
ho provisions and configures
database environm
ent to support the analytical needs of the
w
orking team
•
Share the code
from
the analytical project
•
Create technicaldocum
ent on how
to
im
plem
ent it.
Data Scientist
Provide subject m
atter expertise

for analytical techniques,
data m
odeling, applying
valid analytical techniques to given
business problem
s and ensuring overall analytical objectives
are m
et
•
Show
the analyst presentation
•
Share the code
Copyright ©
2014 EM
4 Core Deliverables to M
eet M
ost Stakeholder N
eeds
1.
Presentation for Project Sponsors

•
“Big picture" takeaw
ays for executive level stakeholders
•
Determ
ine key m
essages to aid their decision-m
aking process
•
Focus on clean, easy visuals for the presenter to explain and for
the
view
er to grasp
2.
Presentation for Analysts
•
Business process changes
•
Reporting changes
•
Fellow
Data Scientists w
ill w
ant the details and are com

fortable w
ith
technical graphs (such as RO
C curves, density plots, histogram
s)
3.
Code
for technical people
4.
Technical specs of im
plem
enting the code
26
M
Copyright ©
2014 EM
Analyst W
ish List for a Successful Analytics
Project

Data &
W
orkspaces
•
Access to all the data, including aggregated O
LAP data, BI tools, raw
data, structured
and various states of unstructured data as needed
•
U
p-to-date data dictionary to describe the data
•
Area for staging and production data sets
•
Ability to m
ove data back and forth
betw
een w
orkspaces and staging areas
•
Analytic sandbox w
ith strong com
pute pow
er to experim

ent and play w
ith the data
Tools
•
Statistical/m
athem
atical/visual softw
are of choice for a given situation and problem
set,
such as SAS, M
atlab, R, java tools, Tableau, Spotfire
•
Collaboration: an online platform
or environm
ent for collaboration and com
m
unicating
w
ith team
m
em
bers
•
Tool or place to log errorsw

ith system
s, environm
ents or data sets
2727
M
Copyright ©
2014 EM
Concepts in Practice
Greenplum
’s Approach to Analytics
M
28
ED
C PLATFO
RM
D
ata
Analytics

A
nalyze
data
R
estructure
data
R
epeat
Past Future
Facts
Interpretation
W
hat w
ill
happen?
How
can w
e
do better?
W
hat
happened
w

here and
w
hen?
How
and
w
hy did it
happen?
M
agnetic
Attract all kinds of data
Agile
Flexible and elastic data structures
Deep
Rich data repository and
algorithm
ic engine
S
ource: M
A
D
S

kills: N
ew
A
nalysis P
ractices for B
ig D
ata, M
arch 2009
Copyright ©
2014 EM
“The pessim
ist –
com
plains about the w
ind
The optim
ist –
expects it to change
The leader–
adjusts the sails
John M

axw
ell
(Leadership Author)
29
M
Copyright ©
2014 EM
Check Your Know
ledge
•
In w
hich phase w
ould you expect to invest m
ost of your project tim
e and
w
hy? W
here w
ould expect to spend the least tim
e?

•
W
hat are the benefits of doing a pilot program
before a full scale rollout of a
new
analytical m
ethodology? Discuss this in the context of the m
ini case
study.
•
W
hat kinds of tools w
ould be used in the follow
ing phases, and for w
hich
kinds of use scenarios?
Phase 4: M
odel Execution
•
N
ow
that you have com
pleted the analytical project at Yoyodyne, you have an

opportunity to repurpose this approach for an online eCom
m
erce com
pany.
W
hat phases of the lifecycle do you need to focus on to identify w
ays to do
this?
30
M
Your Thoughts?
Copyright ©
2014 EM
Sum
m
ary
Key points covered in this lesson:
•
The Data Analytics Lifecycle w

as applied to a case study
scenario
•
A business problem
w
as fram
ed as an analytics problem
•
The four m
ain deliverables in an analytics project w
ere
identified
M
31

Copyright © 2014 EMC Corporation. All Rights Reserved..docx

Recommended

Recommended

More Related Content

Similar to Copyright © 2014 EMC Corporation. All Rights Reserved..docx

Similar to Copyright © 2014 EMC Corporation. All Rights Reserved..docx (20)

More from dickonsondorris

More from dickonsondorris (20)

Recently uploaded

Recently uploaded (20)

Copyright © 2014 EMC Corporation. All Rights Reserved..docx