Sparse Regularization
•
1 like•747 views
That was my lecture in Machine Learning Practical Course: Introduction to Sparse Methods.
Recommended
More Related Content
Similar to Sparse Regularization
Similar to Sparse Regularization (20)
More from Shadi Nabil Albarqouni
Recently uploaded
Recently uploaded (20)
Sparse Regularization
- 1. Introduction to Sparse Methods Shadi Albarqouni, M.Sc. Research Assistant | PhD Candidate! shadi.albarqouni@tum.de ! Machine Learning in Medical Imaging! BioMedical Computing (BMC) Master Program Computer Aided Medical Procedures | Technische Universität München
- 3. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! Boss Christoph
- 4. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! Boss Christoph 1st day of his work
- 5. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! We want to build a Predictive Model for the wages! Boss Christoph 1st day of his work
- 6. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! We want to build a Predictive Model for the wages! What!! Boss Christoph 1st day of his work
- 7. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! We want to build a Predictive Model for the wages! What!! What kind of data you have? How Big? Boss Christoph 1st day of his work
- 8. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! We want to build a Predictive Model for the wages! What!! What kind of data you have? How Big? Data Cleanup, Pre- processing, overfitting! Boss Christoph 1st day of his work
- 9. Introduction Regularization Sparsity Extensions Model Monthly Wages January Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average Observations Boss
- 10. Introduction Regularization Sparsity Extensions Model Monthly Wages October Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average Observations Boss
- 11. Introduction Regularization Sparsity Extensions Model Monthly Wages October Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average Observations Boss Here the observations we have
- 12. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 13. Introduction Regularization Sparsity Extensions Model Curve fitting y=a0 Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 14. Introduction Regularization Sparsity Extensions Model Curve fitting y=a0 Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Under-fitting Christoph
- 15. Introduction Regularization Sparsity Extensions Model Curve fitting y=a0 Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 16. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 17. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a1x x y Christoph
- 18. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 19. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a0+a1x x y Christoph
- 20. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 21. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a0+a1x+a2x^2+…+aNx^N x y Christoph
- 22. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a0+a1x+a2x^2+…+aNx^N x y Over-fitting Christoph
- 23. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a0+a1x+a2x^2+…+aNx^N x y Over-fitting Christoph Which one will make the Boss happy!!
- 24. Introduction Regularization Sparsity Extensions Model Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average_Female Average_Male x y More Interesting and Complicated Boss
- 25. Introduction Regularization Sparsity Extensions Model Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average_Female Average_Male x y More Interesting and Complicated Boss I see! Based on your Model, we should hire in the first quarter, the question is Male or Female? Age? Experience?
- 26. Introduction Regularization Sparsity Extensions Model Building a Model / Formulation Christoph y x 7 9 20 35 1 2 1 7 3 1 12 26 Obser. Features Wages Month Age Gender Exp. …. = Linear Predictive Model few data points -> under-fitting more data points -> robust …. over-fitting? AModel 0.2 0.4 0.1 0.8 0.1 . y = A.x + e Where e is a white gaussian noise
- 27. Introduction Regularization Sparsity Extensions Model Building a Model / Formulation Christoph I decided for a Linear Predictive Model … Pretty Easy! y x 7 9 20 35 1 2 1 7 3 1 12 26 Obser. Features Wages Month Age Gender Exp. …. = Linear Predictive Model few data points -> under-fitting more data points -> robust …. over-fitting? AModel 0.2 0.4 0.1 0.8 0.1 . y = A.x + e Where e is a white gaussian noise
- 29. Introduction Regularization Sparsity Extensions Model Generalized Model Boss You did a good job Christoph! What about a generalized model, we would enter the image processing market, so we may need Image Restoration or Reconstruction!
- 31. Introduction Regularization Sparsity Extensions Model Challenges Christoph OMG, i haven’t heard about that …. how we can solve such a problem!
- 33. Introduction Regularization Sparsity Extensions Model Challenges Christoph Well, it seems not that easy! We need a regularization!
- 35. Introduction Regularization Sparsity Extensions Model Regularization Christoph Ah! Which one would be the best? … I can’t understand how this penalty function would solve the posedness?
- 36. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph
- 37. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features!
- 38. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 39. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 40. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 41. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 42. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 43. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 44. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 45. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 46. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 47. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 48. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 50. Introduction Regularization Sparsity Extensions Model Compressive Sensing Christoph Boss Interesting! We got few elements (features/weights) with L1 This would definitely help us in Memory saving, computations costs … less Money!
- 51. Introduction Regularization Sparsity Extensions Model Compressive Sensing Christoph Boss Interesting! We got few elements (features/weights) with L1 This would definitely help us in Memory saving, computations costs … less Money! That’s true! However, we don’t have prior info. about the latent variable (Sparse or not)? … Wavelet, Fourier, … Dictionary
- 54. Introduction Regularization Sparsity Extensions Model Dictionary Learning D xi =a5d5 +a8d8 +a35d35 …. +a120d120 xi Data points are represented sparely with few non-zero elements
- 58. Introduction Regularization Sparsity Extensions Model Boss Christoph Hey Christoph! We have a plan to work more in this sparse coding topic, please keep working on that.
- 59. Introduction Regularization Sparsity Extensions Model Boss Christoph Hey Christoph! We have a plan to work more in this sparse coding topic, please keep working on that. I have read some materials on Graph Sparse Coding … etc.
- 60. Introduction Regularization Sparsity Extensions Model Boss Christoph Hey Christoph! We have a plan to work more in this sparse coding topic, please keep working on that. I have read some materials on Graph Sparse Coding … etc. Great! Keep doing that! Please if you have any question, don’t hesitate to contact me: shadi.albarqouni@tum.de