Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Overton, Apple Flavored ML

35 views

Published on

In early September, Apple released a paper describing Overton, the framework they built to create, monitor, and improve production-based ML systems. After presenting the main lines that define this framework, we will take a closer look at the heart of Overton: slice-based learning.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Overton, Apple Flavored ML

  1. 1. OvertonOverton Apple- avored MLApple- avored ML 2019/10/07 Nantes Machine Learning Meetup2019/10/07 Nantes Machine Learning Meetup
  2. 2. OverviewOverview
  3. 3. OverviewOverview What / When / WhereWhat / When / Where Publishing date Early September Conference NeurIPS Goal ML software lifecycle management Maturity Production
  4. 4. OverviewOverview Challenges to overcomeChallenges to overcome Precise monitoring Complex pipelines Ef cient feedback loop
  5. 5. OverviewOverview Architectural choicesArchitectural choices Code-less deep learning Multi-task learning Weak supervision
  6. 6. Code-less deepCode-less deep learninglearning
  7. 7. Code-less deep learningCode-less deep learning PrinciplesPrinciples Models & training code = experts “black box” by engineers
  8. 8. Code-less deep learningCode-less deep learning ModularityModularity
  9. 9. Code-less deep learningCode-less deep learning Con gurationCon guration
  10. 10. Fine grained MLFine grained ML & multi-tasks& multi-tasks
  11. 11. Fine grained ML & multi-tasksFine grained ML & multi-tasks ProblemsProblems Lots of subtasks (implicit & explicit) Need to evaluate & monitor them Need to improve on them
  12. 12. Fine grained ML & multi-tasksFine grained ML & multi-tasks ApproachApproach Slice-based learning De nition of data subsets Augmentation of model capacity Dedicated metrics
  13. 13. Fine grained ML & multi-tasksFine grained ML & multi-tasks Slice-based learningSlice-based learning
  14. 14. Fine grained ML & multi-tasksFine grained ML & multi-tasks De nition of critical dataDe nition of critical data subsetssubsets With “slice functions”: def sf_bike(x): return "bike" in object_detector(x) def sf_night(x): return avg(X.pixels.intensity) < 0.3
  15. 15. Fine grained ML & multi-tasksFine grained ML & multi-tasks Slice expertsSlice experts For each slice, an expert: should add capacity to the model has to know when to trigger has to have dedicated metrics
  16. 16. Fine grained ML & multi-tasksFine grained ML & multi-tasks Hard partsHard parts Noise : slices are de ned with heuristics Scale 1 : when the number of slices goes up, does the model still run fast enough? Scale 2 : when the number of slices goes up, is the model still good enough?
  17. 17. Fine grained ML & multi-tasksFine grained ML & multi-tasks Proposed solutionProposed solution
  18. 18. ResourcesResources Snorkel blog “Slice-based Learning” NeurIPS paper “Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices”
  19. 19. WeakWeak supervisionsupervision
  20. 20. Weak supervisionWeak supervision Problem statementProblem statement We need data, but: It's expensive, not available, yadda yadda Especially for interesting corner cases It's noisy
  21. 21. Weak supervisionWeak supervision SolutionSolution Use heuristics @labeling_function() def lf_regex_check_out(x): """Spam comments say 'check out my video', 'check it out', et return SPAM if re.search(r"check.*out", x.text, flags=re.I) e
  22. 22. Weak supervisionWeak supervision SolutionSolution Then correct the loss accounting for their correlation
  23. 23. Weak supervisionWeak supervision ImplementationImplementation Overton uses a modi ed version of the “Label Model” from Snorkel.
  24. 24. ResourcesResources Snorkel blog « Introducing the New Snorkel » ICML paper “Learning Dependency Structures for Weak Supervision Models”
  25. 25. ConclusionConclusion Modularity (alike AllenNLP, tensor2tensor) Dedicated metrics & multi-tasks Improvements on critical data subsets Arti cal data with theoretical corrections
  26. 26. Questions /Questions / DiscussionDiscussion

×