The document discusses using data science for lead generation analytics. It describes how a data scientist would analyze a company's customer relationship management (CRM) data to build a model that can more accurately predict which sales leads are most likely to convert into customers. Features like customer reputation, past purchase history, and offer amounts are engineered from the CRM data to train machine learning models. The models' performance is evaluated based on metrics like accuracy, precision, and recall compared to baseline models. Costs and potential profits from leads that convert are also considered to help determine if the data science approach will be financially beneficial.

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Data Science in the Silicon Valley
Stefano Brunelli
Datum
Ansicht > Kopf und Fusszeile
1

2. Agenda
Ansicht > Kopf und Fusszeile2 Datum
1. Who is a Data Scientist?
2. Can we use Data Science for our everyday business?
3. Lead Generation Analytics
4. HR Analytics
5. Examples from the Silicon Valley

3. Ansicht > Kopf und Fusszeile3 Datum
Who is a Data Scientist?

4. Meet the Characters
Ansicht > Kopf und Fusszeile4 Datum
Darren, 32 Software developer
He knows Python and R inside-out
He also has academic experience with Java and C/C++
which is good only for his resume
He can use any database, relational or NoSQL and
has great experience working with data.

5. Meet the Characters
Ansicht > Kopf und Fusszeile5 Datum
Michail, 48 Statistician, Mathematician
He is an expert in Statistics, Machine Learning and
Artificial Intelligence.
He knows a great deal of Calculus and Linear
Algrebra.

6. Meet the Characters
Ansicht > Kopf und Fusszeile6 Datum
Larry, 45 Business Man.
His life is defined in terms of revenues, profits, costs.
He loves to tie anything he does professionally to well-
defined business goals.
He is an expert in data visualization and has excellent
presentation skills.

7. Ansicht > Kopf und Fusszeile7 Datum
Can we „mere mortals“ use Data Science?

8. Data Science Applied
Ansicht > Kopf und Fusszeile8 Datum
• We’re having troubles identifying customers. How can we more efficiently find them?
• Do we have the right competencies the market is looking for?
• How much money will I make over the next 3 months?

9. Data Science Applied
Ansicht > Kopf und Fusszeile9 Datum
• We’re having troubles identifying customers. How can we more efficiently find them?
Lead Generation Analytics
• Do we have the right competencies the market is looking for?
• How much money will I make over the next 3 months?

10. Data Science Applied
Ansicht > Kopf und Fusszeile10 Datum
• We’re having troubles identifying customers. How can we more efficiently find them?
Lead Generation Analytics
• Do we have the right competencies the market is looking for?
Internal vs Market Skills Analytics
• How much money will I make over the next 3 months?

11. Data Science Applied
Ansicht > Kopf und Fusszeile11 Datum
• We’re having troubles identifying customers. How can we more efficiently find them?
Lead Generation Analytics
• Do we have the right competencies the market is looking for?
Internal vs Market Skills Analytics
• How much money will I make over the next 3 months?
Sales Forecast Analytics

12. Ansicht > Kopf und Fusszeile12 Datum
Lead Generation Analytics

13. Ansicht > Kopf und Fusszeile13 Datum
Lead Generation Analytics
(sort-of)

14. Lead Generation Analytics
Ansicht > Kopf und Fusszeile14 Datum
• What do I need to make money with Data Science:
 A real Business Problem
 Data
 Algorithms and Tools
 Code
 Infrastructure
 Business Buy-In
 An Analytically Driven Mentality

15. Lead Generation Analytics - Data
Ansicht > Kopf und Fusszeile15 Datum

16. Lead Generation Analytics - Data
Ansicht > Kopf und Fusszeile16 Datum
Lead /
Opportunity
Date
Organization
Customer
StatusOffer Amount
CHF

17. Lead Generation Analytics – The Importance of the NULL Model
Ansicht > Kopf und Fusszeile17 Datum
System
CRM Data Is this a good lead?

18. Lead Generation Analytics – The Importance of the NULL Model
Ansicht > Kopf und Fusszeile18 Datum
System
CRM Data Is this a good lead?
Data Hygiene
Feature
Engineering
Model
Selection

19. Lead Generation Analytics – The Importance of the NULL Model:
DataHygiene
Ansicht > Kopf und Fusszeile19 Datum

20. Lead Generation Analytics – The Importance of the NULL Model: Feature
Engineering
Ansicht > Kopf und Fusszeile20 Datum

21. Lead Generation Analytics – The Importance of the NULL Model: Model
Selection
Ansicht > Kopf und Fusszeile21 Datum

22. Lead Generation Analytics – The Importance of the NULL Model: Putting it
all together
Ansicht > Kopf und Fusszeile22 Datum

23. Lead Generation Analytics – 1 Feature on Steroids
Ansicht > Kopf und Fusszeile23 Datum
Customer
Reputation
Money

24. Lead Generation Analytics – 1 Feature on Steroids
Ansicht > Kopf und Fusszeile24 Datum
Customer
Reputation
 What does the customer think of me?
 Am I doing more or less business with
him over time?
Money

25. Lead Generation Analytics – 1 Feature on Steroids
Ansicht > Kopf und Fusszeile25 Datum
Customer
Reputation
 What does the customer think of me?
 Am I doing more or less business with
him over time?
Money
 Is my current offer in line with the average
business I make with this customer?
 Are the prices the customer is willing to
pay me going up or down over time?

26. Lead Generation Analytics – 1 Feature on Steroids
Ansicht > Kopf und Fusszeile26 Datum
1. Cumulative number of contacts
2. Cumulative number of wins
3. Cumulative convertion ratio
4. Cumulative number of contacts over rolling last year
5. Cumulative number of wins over rolling last year
6. Cumulative convertion ratio over rolling last year
7. Cumulative number of contacts over rolling last semester
8. Cumulative number of wins over rolling last semester
9. Cumulative convertion ratio over rolling last semester
10. Cumulative number of contacts over rolling last quarter
11. Cumulative number of wins over rolling last quarter
12. Cumulative convertion ratio over rolling last quarter
13. Cumulative number of contacts over rolling last month
14. Cumulative number of wins over rolling last month
15. Cumulative convertion ratio over rolling last month
16. Offer amount versus overall rolling average
17. Offer amount versus rolling average of last year
18. Offer amount versus rolling average of last semester
19. Offer amount versus rolling average of last quarter
20. Offer amount versus rolling average of last month
Reputation
Money

27. Lead Generation Analytics – How to Evaluate the Model?
Ansicht > Kopf und Fusszeile28 Datum
 Accuracy: fraction of correct classifications

28. Lead Generation Analytics – How to Evaluate the Model?
Ansicht > Kopf und Fusszeile29 Datum
 Accuracy: fraction of correct classifications
 Recall: how good am I at identifying when I am going to win?

29. Lead Generation Analytics – How to Evaluate the Model?
Ansicht > Kopf und Fusszeile30 Datum
 Accuracy: fraction of correct classifications
 Recall: how good am I at identifying when I am going to win?
 Precision: when I say I win, how confident can I be about it? How many false
positive am I going to generate?

30. Lead Generation Analytics – How to Evaluate the Model?
Ansicht > Kopf und Fusszeile31 Datum
 Accuracy: fraction of correct classifications
 Recall: how good am I at identifying when I am going to win?
 Precision: when I say I win, how confident can I be about it? How many false
positive am I going to generate?
 F1 Score: balance between Precision and Recall

31. Lead Generation Analytics – How to Evaluate the Model? Think like a Pro
Ansicht > Kopf und Fusszeile32 Datum
 Scenario 1 - Flip a coin, Head I win, Tail I lose.
 E[Accuracy] = 0.5
 Scenario 2 – Always predict the Majority Class. Business-wise it would mean I
aways chase an opportunity / lead
 E[Accuracy] = ProportionMAJORITY CLASS= 0.61
 Our Model – Is my model improving over both baselines?
 E[Accuracy] = Cross-Validation Score of Best Model
BASELINE

32. Lead Generation Analytics – How to Evaluate the Model? Think like a Pro
Ansicht > Kopf und Fusszeile33 Datum
Scenario Accuracy Precision Recall F1-Score
Flip-a Coin 0.50 - - -

33. Lead Generation Analytics – How to Evaluate the Model? Think like a Pro
Ansicht > Kopf und Fusszeile34 Datum
Scenario Accuracy Precision Recall F1-Score
Flip-a Coin 0.50 - - -
Majority Class 0.61 0.61 1.00 0.76

34. Lead Generation Analytics – How to Evaluate the Model? Think like a Pro
Ansicht > Kopf und Fusszeile35 Datum
Scenario Accuracy Precision Recall F1-Score
Flip-a Coin 0.50 - - -
Majority Class 0.61 0.61 1.00 0.76
Null Model 0.66 0.66 0.93 0.77

35. Lead Generation Analytics – How to Evaluate the Model? Think like a Pro
Ansicht > Kopf und Fusszeile36 Datum
Scenario Accuracy Precision Recall F1-Score
Flip-a Coin 0.50 - - -
Majority Class 0.61 0.61 1.00 0.76
Null Model 0.66 0.66 0.93 0.77
Customer 0.69 0.70 0.89 0.78

36. Lead Generation Analytics – How to Evaluate the Model? Think like a Pro
Ansicht > Kopf und Fusszeile37 Datum
Scenario Accuracy Precision Recall F1-Score
Flip-a Coin 0.50 - - -
Majority Class 0.61 0.61 1.00 0.76
Null Model 0.66 0.66 0.93 0.77
Customer 0.69 0.70 0.89 0.78
Many metrics 0.72 0.74 0.86 0.80

37. Lead Generation Analytics – How to Evaluate the Model? Think like a Pro
Ansicht > Kopf und Fusszeile38 Datum
Scenario Accuracy Precision Recall F1-Score
Flip-a Coin 0.50 - - -
Majority Class 0.61 0.61 1.00 0.76
Null Model 0.66 0.66 0.93 0.77
Customer 0.69 0.70 0.89 0.78
Many metrics 0.72 0.74 0.86 0.80
Interactions 0.73 0.76 0.85 0.80

38. Lead Generation Analytics – Back to Earth
Ansicht > Kopf und Fusszeile39 Datum
This is great stuff
guys, but...ehm.. how
exactly am I going to
make money with it?

39. Lead Generation Analytics – Costs and Profits
Ansicht > Kopf und Fusszeile40 Datum
• How much do I expect to make in Revenues if I win the contract?
• How much does it cost me to go from Lead to signed Contract?
• How much does it cost me to pay my employees to do the work once the contract is signed?

40. Lead Generation Analytics – Costs and Profits
Ansicht > Kopf und Fusszeile41 Datum
• How much do I expect to make in Revenues if I win the contract?
AVERAGE_CONTRACT_VALUE
MEDIAN_CONTRACT_VALUE (if the distribution is very skewed)
• How much does it cost me to go from Lead to signed Contract?
NUMBER_OF_DAYS_OF_WORK_AHED_OF_CONTRACT *
AVERAGE_DAILY_BRUTTO_SALARY_OF_EMPLOYESS
• How much does it cost me to pay my employees to do the work once the contract is signed?
AVERAGE_FTE * AVERAGE_DAILY_BRUTTO_SALARY_OF_EMPLOYEES

41. Lead Generation Analytics – Cost Benefit Matrix
Ansicht > Kopf und Fusszeile42 Datum
Win Lose
Win True Positive False Positive
Lose False Negative True Negative
Actual Ground Truth
Model
Predictions

42. Lead Generation Analytics – Cost Benefit Matrix
Ansicht > Kopf und Fusszeile43 Datum
Win Lose
Win BP+=Profit BC+=Cost
Lose BC-=Indirect Cost BP-=Indirect Profit
Actual Ground Truth
Model
Predictions

43. Lead Generation Analytics – Cost Benefit Matrix
Ansicht > Kopf und Fusszeile44 Datum
BP+ = Average Contract Value – (Average Lead to Offer Costs + Average Project Costs)
BP- = Average Lead to Offer Costs + Average Project Costs
BC+ = -(Average Lead to Offer Costs + Average Project Costs)
BC- = -Average Contract Value

44. Lead Generation Analytics – Cost Benefit Matrix
Ansicht > Kopf und Fusszeile45 Datum
Average Contract Value = 20000
Average Lead to Offer Costs = 6000
Average Project Costs = 0
Win Lose
Win 14000 -6000
Lose -20000 6000
Actual Ground Truth
Model
Predictions

45. Lead Generation Analytics – Profit Curve
Ansicht > Kopf und Fusszeile46 Datum
Optimal threshold: 0.39
Lower thresholds are pretty
much the same
No point in selecting leads
/ opportunities
Chase’em all kind-of
strategy
Does it make sense?

46. Lead Generation Analytics – Cost Benefit Matrix
Ansicht > Kopf und Fusszeile47 Datum
Average Contract Value = 20000
Average Lead to Offer Costs = 6000
Average Project Costs = 0 9800
Win Lose
Win 4200 -15800
Lose -20000 15800
Actual Ground Truth
Model
Predictions

47. Lead Generation Analytics – Profit Curve
Ansicht > Kopf und Fusszeile48 Datum
Optimal threshold: 0.62
Lower thresholds are very
different
Great benefit in selecting
leads / opportunities
Chase’em wisely kind-of
strategy
Allocate your resources
where you expect return

48. Is a Model better than Gut-Feeling?
Ansicht > Kopf und Fusszeile49 Datum
• The Model is the cold, unbiased projection of known data over
the future, finding patterns in data that we humans could never
identify.
• Gut-Feeling is the extordinary capabilities of us human beeings
of seeing nuances that a model cannot see.
• Why not a Model and our Gut-Feeling working together?
• Is it even possible?

49. Sir Reverend Thomas Bayes
Ansicht > Kopf und Fusszeile50 Datum

50. Sir Reverend Thomas Bayes
Ansicht > Kopf und Fusszeile51 Datum
Prior: my Model
Likelihood: how accurate are
our Sales when they estimate win
probability
Posterior: how
gut-feeling
changes model
estimate
«P(A) is what I believed about A until 2 minutes ago, based on my experience.
Than B happened.
P(A|B) is what I believe about A after having taken into account that B happened
Now I feel wiser»

51. Sir Reverend Thomas Bayes
Ansicht > Kopf und Fusszeile52 Datum
Prior: my Model
Likelihood: how accurate are
our Sales when they estimate win
probability
Posterior: how
gut-feeling
changes model
estimate
«P(A) is what I believed about lead A until 2 minutes ago, based on my model
forecast.
Than my head of Sales estimated his probability of victory.
P(A|B) is what I believe about lead A after having taken into account my head of
Sales experience and professionalism.
Now I feel wiser»

52. Bayes in Practice
Ansicht > Kopf und Fusszeile53 Datum
P(Win | Sales) =
P(Sales | Win) * P(Model)
P(Sales)
I can estimate this very easily from our CRM
I threshold on this probability to make
a business decision
This is the probability from our model

53. How do I Interact with the Model?
Ansicht > Kopf und Fusszeile54 Datum
For example with a BI Report that calls the model in the background and displays
the win probability for each currently open opportunity in the pipeline.
Otherwise (more typically):
• wrap the model as a Web-Service
• query it through a POST request passing all the necessary input parameters
• get a probability as a response and consume it with logic on the client side

54. Is this a Lead Generation analytical Data Product?
Ansicht > Kopf und Fusszeile55 Datum
Definition of
Business Context
Interpretation of
Results
Statistical
Modelling
Data
Munging
BusinessValue

55. Definition of Business Context
Ansicht > Kopf und Fusszeile56 Datum
 Did we do it right?
 What are we exactly modeling?
 Does it mean the model is useless?
 How should I use it correctly?

56. Definition of Business Context
Ansicht > Kopf und Fusszeile57 Datum
 Did we do it right?
Not even close
 What are we exactly modeling?
 Does it mean the model is useless?
 How should I use it correctly?

57. Definition of Business Context
Ansicht > Kopf und Fusszeile58 Datum
 Did we do it right?
Not even close
 What are we exactly modeling?
The estimate of win probability of a lead / opportunity once the sales person sits down and enters it in the CRM
 Does it mean the model is useless?
 How should I use it correctly?

58. Definition of Business Context
Ansicht > Kopf und Fusszeile59 Datum
 Did we do it right?
Not even close
 What are we exactly modeling?
The estimate of win probability of a lead / opportunity once the sales person sits down and enters it in the CRM
 Does it mean the model is useless?
Not at all. You simply have to change your point of view
 How should I use it correctly?

59. Definition of Business Context
Ansicht > Kopf und Fusszeile60 Datum
 Did we do it right?
Not even close
 What are we exactly modeling?
The estimate of win probability of a lead / opportunity once the sales person sits down and enters it in the CRM
 Does it mean the model is useless?
Not at all. You simply have to change your point of view
 How should I use it correctly?
Resource allocation. Only work leads / opportunities that are deemed promising and avoid wasting money and time
with the others

60. Ansicht > Kopf und Fusszeile61 Datum
Internal vs Market Skills Analytics

61. Internal vs Market Skills Analytics
Ansicht > Kopf und Fusszeile62 Datum
How can I have a good idea what the market is looking for?

62. Internal vs Market Skills Analytics
Ansicht > Kopf und Fusszeile63 Datum
How can I have a good idea what we are good at?

63. Internal vs Market Skills Analytics
Ansicht > Kopf und Fusszeile64 Datum
Or even better

64. Internal vs Market Skills Analytics
Ansicht > Kopf und Fusszeile65 Datum
If only we had a way to compare the two things...
How do you compare two documents in NLP to determine their similarity?
Text Normalization
Text Featurization
Similarity Score
Tokenization
Punctuation removal
Stopwords removal
Case convertion
Stemming / Lemmatization
Bag-of-words score
Tf-Idf score
Cosine similarity
Jaccard similarity

65. Internal vs Market Skills Analytics
Ansicht > Kopf und Fusszeile66 Datum
If only we had a way to compare the two things...
How do you compare two documents in NLP to determine their similarity?
Text Normalization
Text Featurization
Similarity Score
Tokenization
Punctuation removal
Stopwords removal
Case convertion
Stemming / Lemmatization
Bag-of-words score
Tf-Idf score
Cosine similarity
Jaccard similarity

66. A Job Add as a Matrix of Numbers
Ansicht > Kopf und Fusszeile67 Datum
Documents are single job posts
Di can be grouped according to geographic region («what are companies looking for in the Zürich region?»)
We can do the same with our internal skills:
• rows are single skills
• documents are employees
• group by location / solution / unit

67. We’re almost there
Ansicht > Kopf und Fusszeile68 Datum

68. Vector Space Model Representation
Ansicht > Kopf und Fusszeile69 Datum
Region aggregate job
posts are vectors in this n-
dimensional space
Employees aggregate HR
skill are also vectors in the
same n-dimensional space
PL/SQL
Java
SQL Server
D1 = Zürich Job Market
Q1= Zürich BI Divison
Q2= Zürich INFR Divison
Which division is more
aligned with the market?

69. Vector Space Model and Cosine Similarity
Ansicht > Kopf und Fusszeile70 Datum

70. Vector Space Model and Cosine Similarity
Ansicht > Kopf und Fusszeile71 Datum
It makes perfect sense to use it for our purpose
If Zürich Market is looking for PL/SQL skills lots of job posts will
contain it
If Trivadis Zürich is able to provide this skill via its employees lots of
them will mention it in their skills-could.
The two vectors will point in very similar directions (the angle will be
very small)

71. How do I Interact with the Model?
Ansicht > Kopf und Fusszeile72 Datum
With a simple BI Report
 HR as probably the most important user
 Can spot disallignments between current roster and market
 Can use the tool to guide new employee searches
 Can use the tool to determine reskill policies
 ....

72. Recap
Ansicht > Kopf und Fusszeile73 Datum
Using:
 CRM internal data
 HR internal data
 Job postings online data
We have developed 2 products with which the company could:
 Decide how to allocate its pre-sales efforts
 Assess and optimize its HR capital
It’s all about making data actionable and capitalizing on it.

73. Ansicht > Kopf und Fusszeile74 Datum
Data Science in the Silicon Valley

74. From text to structured data
Ansicht > Kopf und Fusszeile75 Datum

75. From text to structured data
Ansicht > Kopf und Fusszeile76 Datum

76. Politcs and NLP
Ansicht > Kopf und Fusszeile77 Datum

77. Politcs and NLP
Ansicht > Kopf und Fusszeile78 Datum

78. It’s not just for Netflix and Amazon
Ansicht > Kopf und Fusszeile79 Datum

79. It’s not just for Netflix and Amazon
Ansicht > Kopf und Fusszeile80 Datum

80. Analytics and Sports, a true love story
Ansicht > Kopf und Fusszeile81 Datum

81. Analytics and Sports, a true love story
Ansicht > Kopf und Fusszeile82 Datum

82. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Stefano Brunelli
Data Scientist (Business Intelligence Senior Consultant)
Tel.: +41 78 6302606
Stefano.Brunelli@trivadis.com
@Trivadis
Datum
Ansicht > Kopf und Fusszeile