SlideShare a Scribd company logo
PRIOR ON MODEL SPACE
W h a t m a k e s a m o d e l s i m p l e ?
M E I R M A O R
C h i e f A r c h i t e c t @ S p a r k B e y o n d
Outline
Why are simple models desirable
Traditional approaches for simplicity
Alternative approaches
Why Simple models?
PAC model
No Free Lunch
Better generalization
In Reality
Transfer learning and non stationary distributions
Robust against correlated samples, etc. Leslie Valiant, 1984
Why Simple Models? Cont.
Understandable
Trustworthy
Explainable (also for Regulatory reasons)
Understandable models are ultimately more accurate
Traditional complexity control
Bias / Variance tradeoff → We must limit our search space
Shrink the hypothesis space:
limit boosting iterations
tree size
min sample in leaf
number of hidden nodes
impose sparsity constraint
...
Traditional complexity control cont.
Penalize “less favorable” models:
Lasso / ridge regularization
Bagging / Boot strap sampling
Drop out
Which is more likely?
Coefficients from two feed forward ReLu NN
single-output single hidden layer
NETWORK A NETWORK B
Which feature is a more likely?
Both have ℝ2 = 0.1
Math.ulp(x) - The positive distance between this floating-point value and the
double value next larger in magnitude
vs.
Math.log(x) - Natural logarithm
Which is a more likely feature? #2
The distance to the nearest railway station?
vs.
arctan(latitude * longitude)
Everyone is a domain expert!
We are experts in the world we live in.
Currently, humans have a much better prior than machines.
Many ideas repeat themselves across domains.
For example, you don’t have to be a rocket scientist to be familiar with second
derivatives.
Transfer Learning to the rescue
We can and must learn from previous problems
How can a child learn to identify a Ring Tailed Lemur from a single photo?
Becoming common
Pre-Trained Neural networks
Pre-Trained embeddings
A lot of work on Images and Text
Much less research on other data:
TimeNet - RNN for embedding time series data
Most real-life problems tend to have a more complicated shape
A different approach
Use already codified human knowledge
Explicitly look for patterns similar to things you have seen before
Extraordinary claims require extraordinary evidence
At SparkBeyond
Find the best hypotheses, using simple compositions of tried and true building
blocks
The building block may require a lot of code to implement. Yet, will be useful
across domains
Incorporate pre-trained embeddings
Use external knowledge
Prioritize simple hypotheses
Always meta-learn how to learn
● Domain expert can review such a finding
● True phenomenon
● Insightful and actionable
Shops near recreational parks are more successful
Becomes intuitive when you see concrete examples
Colorful gadgets tend to be cheaper
EXTERNAL
DATA
Language
models
News
Social
Media
Wikipedia
Dictionaries
Maps
Simple = compressible = common
A simple model or feature is one which we can be expressed briefly.
MDL - minimum description length is optimal compression.
Better compression leads to better model performance.
But should we be using a vanilla Turing machine for MDL?
Benefits of a better prior
Learn with less data
More robust to change
More robust to data issues
Understandable and explainable
Actionability without a complete model
Open questions and challenges
What is simple?
What makes an insight insightful?
What makes a feature likely to generalize?
Efficient search over insightful hypothesis space
VISIT OUR BOOTH

More Related Content

Similar to Prior On Model Space

Knowledge representation in AI
Knowledge representation in AIKnowledge representation in AI
Knowledge representation in AI
Vishal Singh
 
[PR12] understanding deep learning requires rethinking generalization
[PR12] understanding deep learning requires rethinking generalization[PR12] understanding deep learning requires rethinking generalization
[PR12] understanding deep learning requires rethinking generalization
JaeJun Yoo
 
Semi supervised learning machine learning made simple
Semi supervised learning  machine learning made simpleSemi supervised learning  machine learning made simple
Semi supervised learning machine learning made simple
Devansh16
 
S.O.L.I.D. Principles for Software Architects
S.O.L.I.D. Principles for Software ArchitectsS.O.L.I.D. Principles for Software Architects
S.O.L.I.D. Principles for Software Architects
Ricardo Wilkins
 
Understanding Deep Learning Requires Rethinking Generalization
Understanding Deep Learning Requires Rethinking GeneralizationUnderstanding Deep Learning Requires Rethinking Generalization
Understanding Deep Learning Requires Rethinking Generalization
Ahmet Kuzubaşlı
 
Learning to Search Henry Kautz
Learning to Search Henry KautzLearning to Search Henry Kautz
Learning to Search Henry Kautz
butest
 
Learning to Search Henry Kautz
Learning to Search Henry KautzLearning to Search Henry Kautz
Learning to Search Henry Kautz
butest
 
Where Does It Break?
Where Does It Break?Where Does It Break?
Where Does It Break?
Frank van Harmelen
 
Brief Tour of Machine Learning
Brief Tour of Machine LearningBrief Tour of Machine Learning
Brief Tour of Machine Learning
butest
 
Machine Learning presentation.
Machine Learning presentation.Machine Learning presentation.
Machine Learning presentation.
butest
 
Deep learning for NLP
Deep learning for NLPDeep learning for NLP
Deep learning for NLP
Shishir Choudhary
 
kantorNSF-NIJ-ISI-03-06-04.ppt
kantorNSF-NIJ-ISI-03-06-04.pptkantorNSF-NIJ-ISI-03-06-04.ppt
kantorNSF-NIJ-ISI-03-06-04.ppt
butest
 
Limits of Machine Learning
Limits of Machine LearningLimits of Machine Learning
Limits of Machine Learning
Meir Maor
 
Gist od2-feb-2011
Gist od2-feb-2011Gist od2-feb-2011
Gist od2-feb-2011
ianibbo
 
Semi-Supervised.pptx
Semi-Supervised.pptxSemi-Supervised.pptx
Semi-Supervised.pptx
Tamer Nadeem
 
Presentation on Machine Learning and Data Mining
Presentation on Machine Learning and Data MiningPresentation on Machine Learning and Data Mining
Presentation on Machine Learning and Data Mining
butest
 
Data Mining: Practical Machine Learning Tools and Techniques ...
Data Mining: Practical Machine Learning Tools and Techniques ...Data Mining: Practical Machine Learning Tools and Techniques ...
Data Mining: Practical Machine Learning Tools and Techniques ...
butest
 
Kaggle presentation
Kaggle presentationKaggle presentation
Kaggle presentation
HJ van Veen
 
How data science works and how can customers help
How data science works and how can customers helpHow data science works and how can customers help
How data science works and how can customers help
Danko Nikolic
 
Multilayered paper prototyping for user concept modeling
Multilayered paper prototyping for user concept modelingMultilayered paper prototyping for user concept modeling
Multilayered paper prototyping for user concept modeling
UKOLN (dev), University of Bath
 

Similar to Prior On Model Space (20)

Knowledge representation in AI
Knowledge representation in AIKnowledge representation in AI
Knowledge representation in AI
 
[PR12] understanding deep learning requires rethinking generalization
[PR12] understanding deep learning requires rethinking generalization[PR12] understanding deep learning requires rethinking generalization
[PR12] understanding deep learning requires rethinking generalization
 
Semi supervised learning machine learning made simple
Semi supervised learning  machine learning made simpleSemi supervised learning  machine learning made simple
Semi supervised learning machine learning made simple
 
S.O.L.I.D. Principles for Software Architects
S.O.L.I.D. Principles for Software ArchitectsS.O.L.I.D. Principles for Software Architects
S.O.L.I.D. Principles for Software Architects
 
Understanding Deep Learning Requires Rethinking Generalization
Understanding Deep Learning Requires Rethinking GeneralizationUnderstanding Deep Learning Requires Rethinking Generalization
Understanding Deep Learning Requires Rethinking Generalization
 
Learning to Search Henry Kautz
Learning to Search Henry KautzLearning to Search Henry Kautz
Learning to Search Henry Kautz
 
Learning to Search Henry Kautz
Learning to Search Henry KautzLearning to Search Henry Kautz
Learning to Search Henry Kautz
 
Where Does It Break?
Where Does It Break?Where Does It Break?
Where Does It Break?
 
Brief Tour of Machine Learning
Brief Tour of Machine LearningBrief Tour of Machine Learning
Brief Tour of Machine Learning
 
Machine Learning presentation.
Machine Learning presentation.Machine Learning presentation.
Machine Learning presentation.
 
Deep learning for NLP
Deep learning for NLPDeep learning for NLP
Deep learning for NLP
 
kantorNSF-NIJ-ISI-03-06-04.ppt
kantorNSF-NIJ-ISI-03-06-04.pptkantorNSF-NIJ-ISI-03-06-04.ppt
kantorNSF-NIJ-ISI-03-06-04.ppt
 
Limits of Machine Learning
Limits of Machine LearningLimits of Machine Learning
Limits of Machine Learning
 
Gist od2-feb-2011
Gist od2-feb-2011Gist od2-feb-2011
Gist od2-feb-2011
 
Semi-Supervised.pptx
Semi-Supervised.pptxSemi-Supervised.pptx
Semi-Supervised.pptx
 
Presentation on Machine Learning and Data Mining
Presentation on Machine Learning and Data MiningPresentation on Machine Learning and Data Mining
Presentation on Machine Learning and Data Mining
 
Data Mining: Practical Machine Learning Tools and Techniques ...
Data Mining: Practical Machine Learning Tools and Techniques ...Data Mining: Practical Machine Learning Tools and Techniques ...
Data Mining: Practical Machine Learning Tools and Techniques ...
 
Kaggle presentation
Kaggle presentationKaggle presentation
Kaggle presentation
 
How data science works and how can customers help
How data science works and how can customers helpHow data science works and how can customers help
How data science works and how can customers help
 
Multilayered paper prototyping for user concept modeling
Multilayered paper prototyping for user concept modelingMultilayered paper prototyping for user concept modeling
Multilayered paper prototyping for user concept modeling
 

Recently uploaded

SAP BW4HANA Implementagtion Content Document
SAP BW4HANA Implementagtion Content DocumentSAP BW4HANA Implementagtion Content Document
SAP BW4HANA Implementagtion Content Document
newdirectionconsulta
 
A gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented GenerationA gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented Generation
dataschool1
 
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdfreading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
perranet1
 
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
Timothy Spann
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
9gr6pty
 
Sid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.pptSid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.ppt
ArshadAyub49
 
Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)
GeorgiiSteshenko
 
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
uevausa
 
06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus
Timothy Spann
 
一比一原版南昆士兰大学毕业证如何办理
一比一原版南昆士兰大学毕业证如何办理一比一原版南昆士兰大学毕业证如何办理
一比一原版南昆士兰大学毕业证如何办理
ugydym
 
How To Control IO Usage using Resource Manager
How To Control IO Usage using Resource ManagerHow To Control IO Usage using Resource Manager
How To Control IO Usage using Resource Manager
Alireza Kamrani
 
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
aguty
 
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service LucknowCall Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
hiju9823
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
hyfjgavov
 
一比一原版(UofT毕业证)多伦多大学毕业证如何办理
一比一原版(UofT毕业证)多伦多大学毕业证如何办理一比一原版(UofT毕业证)多伦多大学毕业证如何办理
一比一原版(UofT毕业证)多伦多大学毕业证如何办理
exukyp
 
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
ytypuem
 
社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .
NABLAS株式会社
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
bmucuha
 
Senior Software Profiles Backend Sample - Sheet1.pdf
Senior Software Profiles  Backend Sample - Sheet1.pdfSenior Software Profiles  Backend Sample - Sheet1.pdf
Senior Software Profiles Backend Sample - Sheet1.pdf
Vineet
 
Q4FY24 Investor-Presentation.pdf bank slide
Q4FY24 Investor-Presentation.pdf bank slideQ4FY24 Investor-Presentation.pdf bank slide
Q4FY24 Investor-Presentation.pdf bank slide
mukulupadhayay1
 

Recently uploaded (20)

SAP BW4HANA Implementagtion Content Document
SAP BW4HANA Implementagtion Content DocumentSAP BW4HANA Implementagtion Content Document
SAP BW4HANA Implementagtion Content Document
 
A gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented GenerationA gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented Generation
 
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdfreading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
 
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
 
Sid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.pptSid Sigma educational and problem solving power point- Six Sigma.ppt
Sid Sigma educational and problem solving power point- Six Sigma.ppt
 
Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)Telemetry Solution for Gaming (AWS Summit'24)
Telemetry Solution for Gaming (AWS Summit'24)
 
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
 
06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus
 
一比一原版南昆士兰大学毕业证如何办理
一比一原版南昆士兰大学毕业证如何办理一比一原版南昆士兰大学毕业证如何办理
一比一原版南昆士兰大学毕业证如何办理
 
How To Control IO Usage using Resource Manager
How To Control IO Usage using Resource ManagerHow To Control IO Usage using Resource Manager
How To Control IO Usage using Resource Manager
 
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
一比一原版澳洲西澳大学毕业证(uwa毕业证书)如何办理
 
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service LucknowCall Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
 
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
一比一原版兰加拉学院毕业证(Langara毕业证书)学历如何办理
 
一比一原版(UofT毕业证)多伦多大学毕业证如何办理
一比一原版(UofT毕业证)多伦多大学毕业证如何办理一比一原版(UofT毕业证)多伦多大学毕业证如何办理
一比一原版(UofT毕业证)多伦多大学毕业证如何办理
 
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
一比一原版(曼大毕业证书)曼尼托巴大学毕业证如何办理
 
社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
Senior Software Profiles Backend Sample - Sheet1.pdf
Senior Software Profiles  Backend Sample - Sheet1.pdfSenior Software Profiles  Backend Sample - Sheet1.pdf
Senior Software Profiles Backend Sample - Sheet1.pdf
 
Q4FY24 Investor-Presentation.pdf bank slide
Q4FY24 Investor-Presentation.pdf bank slideQ4FY24 Investor-Presentation.pdf bank slide
Q4FY24 Investor-Presentation.pdf bank slide
 

Prior On Model Space

  • 1. PRIOR ON MODEL SPACE W h a t m a k e s a m o d e l s i m p l e ? M E I R M A O R C h i e f A r c h i t e c t @ S p a r k B e y o n d
  • 2. Outline Why are simple models desirable Traditional approaches for simplicity Alternative approaches
  • 3. Why Simple models? PAC model No Free Lunch Better generalization In Reality Transfer learning and non stationary distributions Robust against correlated samples, etc. Leslie Valiant, 1984
  • 4. Why Simple Models? Cont. Understandable Trustworthy Explainable (also for Regulatory reasons) Understandable models are ultimately more accurate
  • 5. Traditional complexity control Bias / Variance tradeoff → We must limit our search space Shrink the hypothesis space: limit boosting iterations tree size min sample in leaf number of hidden nodes impose sparsity constraint ...
  • 6. Traditional complexity control cont. Penalize “less favorable” models: Lasso / ridge regularization Bagging / Boot strap sampling Drop out
  • 7. Which is more likely? Coefficients from two feed forward ReLu NN single-output single hidden layer NETWORK A NETWORK B
  • 8. Which feature is a more likely? Both have ℝ2 = 0.1 Math.ulp(x) - The positive distance between this floating-point value and the double value next larger in magnitude vs. Math.log(x) - Natural logarithm
  • 9. Which is a more likely feature? #2 The distance to the nearest railway station? vs. arctan(latitude * longitude)
  • 10. Everyone is a domain expert! We are experts in the world we live in. Currently, humans have a much better prior than machines. Many ideas repeat themselves across domains. For example, you don’t have to be a rocket scientist to be familiar with second derivatives.
  • 11. Transfer Learning to the rescue We can and must learn from previous problems How can a child learn to identify a Ring Tailed Lemur from a single photo?
  • 12. Becoming common Pre-Trained Neural networks Pre-Trained embeddings A lot of work on Images and Text Much less research on other data: TimeNet - RNN for embedding time series data Most real-life problems tend to have a more complicated shape
  • 13. A different approach Use already codified human knowledge Explicitly look for patterns similar to things you have seen before Extraordinary claims require extraordinary evidence
  • 14. At SparkBeyond Find the best hypotheses, using simple compositions of tried and true building blocks The building block may require a lot of code to implement. Yet, will be useful across domains Incorporate pre-trained embeddings Use external knowledge Prioritize simple hypotheses Always meta-learn how to learn
  • 15. ● Domain expert can review such a finding ● True phenomenon ● Insightful and actionable Shops near recreational parks are more successful
  • 16. Becomes intuitive when you see concrete examples Colorful gadgets tend to be cheaper
  • 18. Simple = compressible = common A simple model or feature is one which we can be expressed briefly. MDL - minimum description length is optimal compression. Better compression leads to better model performance. But should we be using a vanilla Turing machine for MDL?
  • 19. Benefits of a better prior Learn with less data More robust to change More robust to data issues Understandable and explainable Actionability without a complete model
  • 20. Open questions and challenges What is simple? What makes an insight insightful? What makes a feature likely to generalize? Efficient search over insightful hypothesis space