This summer, during the third edition of Data Science Summit in Warsaw, Magdalena Wójcik (Senior Data Scientist at LogicAI) presented how we used Bayesian models in one of our projects.
Probability distribution is a way to shape the sample data to make predictions and draw conclusions about an entire population because most improvement projects and scientific research studies are conducted with sample data rather than with data from an entire population. Probability distribution helps finding all the possible values a random variable can take between the minimum and maximum possible values
Probability distribution is a way to shape the sample data to make predictions and draw conclusions about an entire population because most improvement projects and scientific research studies are conducted with sample data rather than with data from an entire population. Probability distribution helps finding all the possible values a random variable can take between the minimum and maximum possible values
Module Five Normal Distributions & Hypothesis TestingTop of F.docxroushhsiu
Module Five: Normal Distributions & Hypothesis Testing
Top of Form
Bottom of Form
·
Introduction & Goals
This week's investigations introduce and explore one of the most common distributions (one you may be familiar with): the Normal Distribution. In our explorations of the distribution and its associated curve, we will revisit the question of "What is typical?" and look at the likelihood (probability) that certain observations would occur in a given population with a variable that is normally distributed. We will apply our work with Normal Distributions to briefly explore some big concepts of inferential statistics, including the Central Limit Theorem and Hypothesis Testing. There are a lot of new ideas in this week’s work. This week is more exploratory in nature.
Goals:
· Explore the Empirical Rule
· Become familiar with the normal curve as a mathematical model, its applications and limitations
· Calculate z-scores & explain what they mean
· Use technology to calculate normal probabilities
· Determine the statistical significance of an observed difference in two means
· Use technology to perform a hypothesis test comparing means (z-test) and interpret its meaning
· Use technology to perform a hypothesis test comparing means (t-test) (optional)
· Gather data for Comparative Study Final Project.
·
DoW #5: The SAT & The ACT
Two Common Tests for college admission are the SAT (Scholastic Aptitude Test) and the ACT (American College Test). The scores for these tests are scaled so that they follow a normal distribution.
· The SAT reported that its scores were normally distributed with a mean μ=896 and a standard deviation σ=174
· The ACT reported that its scores were normally distributed with a mean μ=20.6 and a standard deviation σ=5.2.
We have two questions to consider for this week’s DoW:
2. A high school student Bobby takes both of these tests. On the SAT, he achieves a score of 1080. On the ACT, he achieves a score of 30. He cannot decide which score is the better one to send with his college applications.
. Question: Which test score is the stronger score to send to his colleges?
· A hypothetical group called SAT Prep claims that students who take their SAT Preparatory course score higher on the SAT than the general population. To support their claim, they site a study in which a random sample of 50 SAT Prep students had a mean SAT score of 1000. They claim that since this mean is higher than the known mean of 896 for all SAT scores, their program must improve SAT scores.
. Question: Is this difference in the mean scores statistically significant? Does SAT Prep truly improve SAT Scores?
.
Investigation 1: What is Normal?
One reason for gathering data is to see which observations are most likely. For instance, when we looked at the raisin data in DoW #3, we were looking to see what the most likely number of raisins was for each brand of raisins. We cannot ever be certain of the exact number of raisins in a box (because it varies) ...
250 words, no more than 500· Focus on what you learned that made.docxeugeniadean34240
250 words, no more than 500
· Focus on what you learned that made an impression, what may have surprised you, and what you found particularly beneficial and why. Specifically:
· What did you find that was really useful, or that challenged your thinking?
· What are you still mulling over?
· Was there anything that you may take back to your classroom?
· Is there anything you would like to have clarified?
Your Weekly Reflection will be graded on the following criteria for a total of 5 points:
· Reflection is written in a clear and concise manner, making meaningful connections to the investigations & objectives of the week.
· Reflection demonstrates the ability to push beyond the scope of the course, connecting to prior learning or experiences, questioning personal preconceptions or assumptions, and/or defining new modes of thinking.
BELOW ARE LESSON COVERED
· This week's investigations introduce and explore one of the most common distributions (one you may be familiar with): the Normal Distribution. In our explorations of the distribution and its associated curve, we will revisit the question of "What is typical?" and look at the likelihood (probability) that certain observations would occur in a given population with a variable that is normally distributed. We will apply our work with Normal Distributions to briefly explore some big concepts of inferential statistics, including the Central Limit Theorem and Hypothesis Testing. There are a lot of new ideas in this week’s work. This week is more exploratory in nature.
Goals:
· Explore the Empirical Rule
· Become familiar with the normal curve as a mathematical model, its applications and limitations
· Calculate z-scores & explain what they mean
· Use technology to calculate normal probabilities
· Determine the statistical significance of an observed difference in two means
· Use technology to perform a hypothesis test comparing means (z-test) and interpret its meaning
· Use technology to perform a hypothesis test comparing means (t-test) (optional)
· Gather data for Comparative Study Final Project.
·
DoW #5: The SAT & The ACT
Two Common Tests for college admission are the SAT (Scholastic Aptitude Test) and the ACT (American College Test). The scores for these tests are scaled so that they follow a normal distribution.
· The SAT reported that its scores were normally distributed with a mean μ=896 and a standard deviation σ=174
· The ACT reported that its scores were normally distributed with a mean μ=20.6 and a standard deviation σ=5.2.
We have two questions to consider for this week’s DoW:
2. A high school student Bobby takes both of these tests. On the SAT, he achieves a score of 1080. On the ACT, he achieves a score of 30. He cannot decide which score is the better one to send with his college applications.
. Question: Which test score is the stronger score to send to his colleges?
· A hypothetical group called SAT Prep claims that students who take their SAT Preparatory course score higher o.
Quantitative Methods for Management_MBA_Bharathiar University probability dis...Victor Seelan
unit 3 probability distribution
Probability – definitions – addition and multiplication Rules (only statements) – simple business application problems – probability distribution – expected value concept – theoretical probability distributions – Binomial, Poison and Normal – Simple problems applied to business.
Generalized Linear Models for Between-Subjects Designssmackinnon
Here is a tutorial on how to use generalized linear models in SPSS software. These are models that are frequently more appropriate than ANOVA or linear regression, especially when the distributions of outcome variables are non-normal and/or homogeneity of variance assumptions are violated.
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
More Related Content
Similar to Resolving e commerce challenges with probabilistic programming
Module Five Normal Distributions & Hypothesis TestingTop of F.docxroushhsiu
Module Five: Normal Distributions & Hypothesis Testing
Top of Form
Bottom of Form
·
Introduction & Goals
This week's investigations introduce and explore one of the most common distributions (one you may be familiar with): the Normal Distribution. In our explorations of the distribution and its associated curve, we will revisit the question of "What is typical?" and look at the likelihood (probability) that certain observations would occur in a given population with a variable that is normally distributed. We will apply our work with Normal Distributions to briefly explore some big concepts of inferential statistics, including the Central Limit Theorem and Hypothesis Testing. There are a lot of new ideas in this week’s work. This week is more exploratory in nature.
Goals:
· Explore the Empirical Rule
· Become familiar with the normal curve as a mathematical model, its applications and limitations
· Calculate z-scores & explain what they mean
· Use technology to calculate normal probabilities
· Determine the statistical significance of an observed difference in two means
· Use technology to perform a hypothesis test comparing means (z-test) and interpret its meaning
· Use technology to perform a hypothesis test comparing means (t-test) (optional)
· Gather data for Comparative Study Final Project.
·
DoW #5: The SAT & The ACT
Two Common Tests for college admission are the SAT (Scholastic Aptitude Test) and the ACT (American College Test). The scores for these tests are scaled so that they follow a normal distribution.
· The SAT reported that its scores were normally distributed with a mean μ=896 and a standard deviation σ=174
· The ACT reported that its scores were normally distributed with a mean μ=20.6 and a standard deviation σ=5.2.
We have two questions to consider for this week’s DoW:
2. A high school student Bobby takes both of these tests. On the SAT, he achieves a score of 1080. On the ACT, he achieves a score of 30. He cannot decide which score is the better one to send with his college applications.
. Question: Which test score is the stronger score to send to his colleges?
· A hypothetical group called SAT Prep claims that students who take their SAT Preparatory course score higher on the SAT than the general population. To support their claim, they site a study in which a random sample of 50 SAT Prep students had a mean SAT score of 1000. They claim that since this mean is higher than the known mean of 896 for all SAT scores, their program must improve SAT scores.
. Question: Is this difference in the mean scores statistically significant? Does SAT Prep truly improve SAT Scores?
.
Investigation 1: What is Normal?
One reason for gathering data is to see which observations are most likely. For instance, when we looked at the raisin data in DoW #3, we were looking to see what the most likely number of raisins was for each brand of raisins. We cannot ever be certain of the exact number of raisins in a box (because it varies) ...
250 words, no more than 500· Focus on what you learned that made.docxeugeniadean34240
250 words, no more than 500
· Focus on what you learned that made an impression, what may have surprised you, and what you found particularly beneficial and why. Specifically:
· What did you find that was really useful, or that challenged your thinking?
· What are you still mulling over?
· Was there anything that you may take back to your classroom?
· Is there anything you would like to have clarified?
Your Weekly Reflection will be graded on the following criteria for a total of 5 points:
· Reflection is written in a clear and concise manner, making meaningful connections to the investigations & objectives of the week.
· Reflection demonstrates the ability to push beyond the scope of the course, connecting to prior learning or experiences, questioning personal preconceptions or assumptions, and/or defining new modes of thinking.
BELOW ARE LESSON COVERED
· This week's investigations introduce and explore one of the most common distributions (one you may be familiar with): the Normal Distribution. In our explorations of the distribution and its associated curve, we will revisit the question of "What is typical?" and look at the likelihood (probability) that certain observations would occur in a given population with a variable that is normally distributed. We will apply our work with Normal Distributions to briefly explore some big concepts of inferential statistics, including the Central Limit Theorem and Hypothesis Testing. There are a lot of new ideas in this week’s work. This week is more exploratory in nature.
Goals:
· Explore the Empirical Rule
· Become familiar with the normal curve as a mathematical model, its applications and limitations
· Calculate z-scores & explain what they mean
· Use technology to calculate normal probabilities
· Determine the statistical significance of an observed difference in two means
· Use technology to perform a hypothesis test comparing means (z-test) and interpret its meaning
· Use technology to perform a hypothesis test comparing means (t-test) (optional)
· Gather data for Comparative Study Final Project.
·
DoW #5: The SAT & The ACT
Two Common Tests for college admission are the SAT (Scholastic Aptitude Test) and the ACT (American College Test). The scores for these tests are scaled so that they follow a normal distribution.
· The SAT reported that its scores were normally distributed with a mean μ=896 and a standard deviation σ=174
· The ACT reported that its scores were normally distributed with a mean μ=20.6 and a standard deviation σ=5.2.
We have two questions to consider for this week’s DoW:
2. A high school student Bobby takes both of these tests. On the SAT, he achieves a score of 1080. On the ACT, he achieves a score of 30. He cannot decide which score is the better one to send with his college applications.
. Question: Which test score is the stronger score to send to his colleges?
· A hypothetical group called SAT Prep claims that students who take their SAT Preparatory course score higher o.
Quantitative Methods for Management_MBA_Bharathiar University probability dis...Victor Seelan
unit 3 probability distribution
Probability – definitions – addition and multiplication Rules (only statements) – simple business application problems – probability distribution – expected value concept – theoretical probability distributions – Binomial, Poison and Normal – Simple problems applied to business.
Generalized Linear Models for Between-Subjects Designssmackinnon
Here is a tutorial on how to use generalized linear models in SPSS software. These are models that are frequently more appropriate than ANOVA or linear regression, especially when the distributions of outcome variables are non-normal and/or homogeneity of variance assumptions are violated.
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
Data Centers - Striving Within A Narrow Range - Research Report - MCG - May 2...pchutichetpong
M Capital Group (“MCG”) expects to see demand and the changing evolution of supply, facilitated through institutional investment rotation out of offices and into work from home (“WFH”), while the ever-expanding need for data storage as global internet usage expands, with experts predicting 5.3 billion users by 2023. These market factors will be underpinned by technological changes, such as progressing cloud services and edge sites, allowing the industry to see strong expected annual growth of 13% over the next 4 years.
Whilst competitive headwinds remain, represented through the recent second bankruptcy filing of Sungard, which blames “COVID-19 and other macroeconomic trends including delayed customer spending decisions, insourcing and reductions in IT spending, energy inflation and reduction in demand for certain services”, the industry has seen key adjustments, where MCG believes that engineering cost management and technological innovation will be paramount to success.
MCG reports that the more favorable market conditions expected over the next few years, helped by the winding down of pandemic restrictions and a hybrid working environment will be driving market momentum forward. The continuous injection of capital by alternative investment firms, as well as the growing infrastructural investment from cloud service providers and social media companies, whose revenues are expected to grow over 3.6x larger by value in 2026, will likely help propel center provision and innovation. These factors paint a promising picture for the industry players that offset rising input costs and adapt to new technologies.
According to M Capital Group: “Specifically, the long-term cost-saving opportunities available from the rise of remote managing will likely aid value growth for the industry. Through margin optimization and further availability of capital for reinvestment, strong players will maintain their competitive foothold, while weaker players exit the market to balance supply and demand.”
2. Content of the presentation
1) What is the Bayesian approach?
2) Profits of going Bayesian
3) Recap of distributions
4) Toolbox for a Bayesian Hacker
5) Case Study #1 - Price-demand change analysis
6) Case Study #2 - Hierarchical modeling
4. What is the Bayesian approach?
Thomas Bayes - XVIII century mathematician who
interpreted probability as the degree of belief, and
not the simple frequency of events.
5. Bayes’ Theorem
Posterior probability
of A given evidence B
Prior
probability
of A
Likelihood of
collecting evidence B
when A is true
Probability of
collecting B under all
circumstances
6. Mandatory naive example 🤒
What is the probability of Bob having a relatively rare disease,
given that he received positive result from the medical test?
Disease occurs in 1 / 250 people. Test has True Positive Rate of 0.99.
7. Mandatory naive example 🤒
What is the probability of Bob having a relatively rare disease,
given that he received positive result from the medical test?
Disease occurs in 1 / 250 people. Test has True Positive Rate of 0.99.
0.99
8. Mandatory naive example 🤒
What is the probability of Bob having a relatively rare disease,
given that he received positive result from the medical test?
Disease occurs in 1 / 250 people. Test has True Positive Rate of 0.99.
0.004
0.99
9. Mandatory naive example 🤒
What is the probability of Bob having a relatively rare disease,
given that he received positive result from the medical test?
Disease occurs in 1 / 250 people. Test has True Positive Rate of 0.99.
0.004
0.99
0.99×1 + 0.01×249
250
10. Mandatory naive example 🤒
What is the probability of Bob having a relatively rare disease,
given that he received positive result from the medical test?
Disease occurs in 1 / 250 people. Test has True Positive Rate of 0.99.
0.004
0.99
0.28
0.99×1 + 0.01×249
250
11. Bayes’ Theorem in 🔍 of Data Scientist
Posterior distribution
Our updated belief
Prior belief as a
distribution
Collected data
Constant normalization term
12. Profits of going Bayesian
● Instead of one value, we get a distribution of likely values.
● We get information on certainty of model output.
● More ways to compare outcomes.
● Spot for expert knowledge already incorporated in the model.
● Easy way to include external knowledge when data set is small.
13. It’s where we put our beliefs. So we choose the distribution wisely:
1. Empirically - we have already done some experiments and have actual data,
2. With expertise - we have expert domain knowledge on the subject,
3. With intent - we have reasons to prefer some values over the others,
4. YOLO - we have no idea and wouldn’t like to affect the outcome.
Choosing prior distribution
14. Uniform distribution
all possible values fall between minimum and
maximum bounds and have equal likelihood.
Conditions:
1. The minimum value is fixed.
2. The maximum value is fixed.
3. All values between the minimum and
maximum occur with equal likelihood.
Recap of distributions
15. Normal distribution
The most common distribution, which has
3 properties:
1. Some value (mean of the
distribution) is the most likely.
2. The uncertain variable could as
likely be above the mean as it could
be below the mean (symmetrical
about the mean).
3. The uncertain variable is more
likely to be in the vicinity of the
mean than further away.
Recap of distributions
16. Poisson distribution
describes the number of times an event occurs in a given
interval.
Conditions:
1. The number of possible occurrences in any
interval is unlimited.
2. The occurrences are independent. The number of
occurrences in one interval does not affect the
number of occurrences in other intervals.
3. The average number of occurrences must remain
the same from interval to interval
Recap of distributions
17. Gamma distribution
The gamma distribution is most often used as the
distribution of the amount of time until the rth
occurrence of an event in a Poisson process.
Conditions:
1. The number of possible occurrences in any unit
of measurement is not limited to a fixed
number.
2. The occurrences are independent.
3. The average number of occurrences must
remain the same from unit to unit.
Recap of distributions
19. Bayesian Hacker’s Toolbox - sampling!
When there is no closed-form solution, we can approximate with sampling, using
technique named MCMC (Markov Chain Monte Carlo).
PyMC gives us also a set of pre-defined distributions.
20. Markov Chain Monte Carlo - easy explanation!
1. Generate random guesses (Monte Carlo part)
2. Generate next generation guesses based only on the guesses before that.
(Markov Chain). Fun fact: this property defines that those chains are memoryless.
3. Accept new generation guesses if they “moving in the right direction”,
otherwise reject them.
22. Case Study #1: Price-Demand change analysis
We know when we changed the price for a particular product.
What we don’t know are:
● Was this price change noticeable (enough to change the demand)?
● When the price change affected the demand change?
● Can models detect this from the data?
23. Price change effect on demand
Price $99.98 Price $79.98
Real-world examples are often noisy and concern thousands of products at once. Below is an example of the
noisy plot.
24. Price change effect on demand
Price $99.98 Price $79.98
We’ll use a simplistic example for this demonstration.
25. First steps: sales distribution
1. Define the distribution of the data:
We want to predict sales - what can be the distribution?
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
26. First steps: sales distribution
1. Define the distribution of the data:
We want to predict sales - what can be the distribution?
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
Way of thinking
● Discrete
27. First steps: sales distribution
1. Define the distribution of the data:
We want to predict sales - what can be the distribution?
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
Way of thinking
● Discrete
● More than 3 unique values
28. First steps: sales distribution
1. Define the distribution of the data:
We want to predict sales - what can be the distribution?
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
Way of thinking
● Discrete
● More than 3 unique values
Shows number of trials
until the success is
achieved
Number of times event is
occurred during the time
interval
29. First steps: λ distribution
1. We choose the Poisson distribution: Poisson (λ)
2. Now we must define the distribution of the λ
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
30. First steps: λ distribution
1. We choose the Poisson distribution: Poisson (λ)
2. Now we must define the distribution of the λ
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
Way of thinking
● Continuous
31. First steps: λ distribution
1. We choose the Poisson distribution: Poisson (λ)
2. Now we must define the distribution of the λ
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
Way of thinking
● Continuous
● Positive
32. First steps: λ distribution
1. We choose the Poisson distribution: Poisson (λ)
2. Now we must define the distribution of the λ
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
Way of thinking
● Continuous
● Positive
● Time related
33. First steps: λ distribution
1. We choose the Poisson distribution: Poisson (λ)
2. Now we must define the distribution of the λ
Normal Exponential Cauchy Zero Inflated Binomial
Uniform Gamma Geometric Flat
Binomial Beta Minimum extreme Half Flat
Poisson Lognormal Negative binomial Logistic
Weibull Student’s t Discrete uniform Negative Normal
Way of thinking
● Continuous
● Positive
● Time related
Event can occur at
any time randomly
Event is more/less
likely occur over
time
Event can occur at
any time not
necessarily
randomly
35. Now we build the model
2. Check the results!
On 16th day
the demand
changed
36. Conclusions 🤔 ?
1. For some products the change may not be
noticeable.
2. For some products the change can be easily spotted.
Why?
How much the price
changed?
Dig deeper!
37. Case Study #2: Price elasticity of demand 💸
Calculate Price Elasticity of Demand
for all products in store. Except some
of them are very new and for some
price almost never changed.
38. Case Study #2: Hierarchical model 🏔
Hierarchical models allows you to define the parameters taking to the account
group means, using the informations about products similarity.
Group level
Product level
39. Shrinkage
Products are “pulled” towards the group
mean.
Why is it important?
If you have few data points or a new
product, you can leverage this facts by using
knowledge from the group mean.
source
40. The simple model ⛰
1. Distribution of the target - sales
(Your target is λ = 𝑤*x +𝑤)
2. Distribution on the group’s level:
a. 𝑤′1
b. 𝑤′0
3. Distributions on products level:
a. 𝑤1
b. 𝑤0
1. Poisson (λ)
2.
a. Normal ( 𝛍, 𝛔)
b. Normal ( 𝛍, 𝛔)
3.
a. Group’s 𝛍, 𝛔
b. Group’s 𝛍, 𝛔
Let’s build the model!
43. Performance - explained
Mean - average of distribution
SD - standard deviation of distribution
mc_error - standard error of posterior sample mean as estimate of theoretical expectation for given
parameter. Rule of thumb: want MC error < 1 − 5% of posterior SD
hpd_2.5 & hpd_97.5 - 2.5% and 97.5% percentiles of the posterior samples for each parameter give a 95%
posterior credible interval
N_eff - number of effective samples. Rule of thumb: want N_eff ~= number of samples
Rhat - Gelman-Rubin convergence diagnostic, Rule of thumb: want Rhat ~= 1
44. More complex model 🏔
Add more levels
(category, subcategory, etc)
Add more
parameters
(𝑤2
, 𝑤3
,... 𝑤n
)
45. If smth wrong 🌋
1. Check if distribution matches the target formula
2. Plot prior distribution, maybe your prior believes are absurd
3. Plot posterior distribution, maybe your evaluations are too vague
4. Try to redefine the model with offset
5. Check possible suggestions here
6. If nothing works, reparametrize!
46. Useful links 👊
Library: PyMC3
Books:
Bayesian Methods for Hackers (free)
Statistical Rethinking
Good guide to distribution description (free)
Introduction to Bayesian Monte Carlo
To follow:
Thomas Wiecki (and his great blog)
Richard McElreath (also, check out his awesome lectures on )
48. 1
2
About us
We are a Boutique Data Science consulting, specialising in leading digital transformations.
We successfully realized custom projects and won Data Science competitions for startups to
Fortune 500 companies, some of them listed below.
We organize biggest Data Science community events called Kaggle Days in
cooperation with Google-owned Kaggle. Demand for AI experts is extremely
high nowadays - LogicAI is a company which has direct access to the top 3M
data scientists in the world.
49. Our customers have access to 2.8mln world’s top talent
We build world’s biggest offline Data Science community
We build Data Science teams for our customers
+ =
”Great community with lots of diverse talent
and skill sets.”
“You rock. Being nice and generous is the most
important thing in community events and you
were!”
50. 1st place
Allstate
[2016]
1st place
Mercari
[2018]
2nd place
GE
[2014]
Problem solved:
Which potential
customers are at
risk of not repaying
their loans?
3rd place
Am Express
[2015]
3rd place
Deloitte
[2015]
Problem solved:
Which of our current
customers will stay
insured with us for
an entire policy
term?
Problem solved:
What is the best
price for the
product I want to
sell?
Problem solved:
How to predict
flight delays over
The US?
Problem solved:
What will future
rental prices for
properties across
Western Australia
be?
Our experience