Data Science Salon: Deep Learning as a Product @ Scribd
•Download as PPTX, PDF•
1 like•786 views
Presented by Kevin Perko, Head of Data Science at Scribd Next DSS NYC Event 👉 https://datascience.salon/newyork/ Next DSS LA Event 👉 https://datascience.salon/la/ Kevin will cover his experience using deep learning, going from scratch to deploying models in production to improve the product experience. He goes in-depth in terms of how we started deep learning from scratch, including navigating the maze of frameworks and hyper-parameters to optimize. Kevin will discuss pitfalls of using other people's algorithms and make a call for more rigor in publishing data science blog posts. Kevin closes with how his failure turned into an open source contribution and the work in moving from dev to production.
Recommended
More Related Content
Similar to Data Science Salon: Deep Learning as a Product @ Scribd
Similar to Data Science Salon: Deep Learning as a Product @ Scribd (20)
More from Formulatedby
More from Formulatedby (20)
Recently uploaded
Recently uploaded (20)
Data Science Salon: Deep Learning as a Product @ Scribd
- 3. Who is the talk for? People interested in our experience using deep learning Product side of deep learning Not a technical talk on how to best utilize LSTM cells/activation functions/networks architectures/GANS/etc
- 4. Motivation Spelling correction to improve the user experience
- 5. Deep Learning as Product Does the user even care?
- 6. What can Deep Learning even do? Image Editing Object Detection Boundary Detection Music Generation Voice Generation Translate Languages
- 7. This will be fun! Style Transfer - Product - Visual - Imperfect - Visualize the errors
- 8. Classic Van Gogh Style Transfer Example Source: http://ideate.xsead.cmu.edu/gallery/projects/art-transitions-style-transfers-by-neural-networks
- 10. STYLE TRANSFER FTW - PRISMA Base Foreground Transfer Heisenberg
- 11. Can it solve our problem? If they can translate styles onto images...
- 12. Motivation Spelling correction to improve the user experience
- 13. Spelling correction Spelling correction is solved right? If you have enough data Google is ok, we are not They can take a probabilistic approach, good for them :P We need to get fancier
- 14. Let’s get started 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch videos 5. Deep Learn
- 17. Which one to pick? We did a comprehensive framework bakeoff… AND THE WINNER IS ?!?! J/K We picked Keras - Abstraction layer (easier to get started) - Familiarity with Python
- 18. Life as a Deep Learner http://everythingbrilliant.co.uk/wp-content/uploads/2017/01/emotional-journey-of-creating-anything-new.jpeg
- 21. GANS eh? BAD DOG GOOD DOG
- 22. Which architecture to use? 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch Videos 5. Deep Learn
- 23. TL;DR Certain problems require certain architectures For example: CNN -> Image parsing/Object Detection
- 24. Where are we? What do we have? Keras (framework) What do we need? Architecture/algorithm Constraints? Avoid implementing our own from scratch
- 25. Can we do even do this? Let’s code some boilerplate for a basic neural network
- 27. Training How long does model development take TBD -> 1-2 weeks How long to let the model train 7 days Is more gpu moar better? Not yet...
- 28. So many parameters... 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch videos 5. Deep Learn
- 29. Details (Hyperparameters) Activation Units Sigmoid, relu, prelu (yes, not even joking), lolu (actually joking) Optimization Functions Adam, sgd, etc, etc Dropout Rate Batch Size Continues on forever...
- 31. Let’s find our algorithm 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch Videos 5. Deep Learn
- 32. Sequence to Sequence mapping TL;DR Given a sequence of things turn them into another sequence Spelling Correction ‘Cande’ -> ‘candy’ English -> Spanish language translation ‘Hello’ -> ‘hola’
- 33. Nice viz of LSTM cells
- 34. Where are we? 1. We have a framework 2. We have an algorithm 3. Let’s find an example
- 35. Deep Spelling Example Blog post on Medium Github code 90%+ Accuracy!
- 36. ALL PROBLEMS SOLVED !!!
- 37. WRONG :(
- 40. Data science blog posts Not reproducible Proprietary data sets that are not shared Incorrect accuracy metrics Someone please solve this (Academic + Industry partnership on peer reviewed + reproducible data science algorithms)
- 42. Let’s find a new approach 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch videos 5. Deep Learn
- 43. What about seq2seq? New approach - Existing libraries/frameworks - OpenNMT - Open Source contribution
- 44. SUCCESS!!!
- 45. What’s edit distance? Number of letters (operations) to replace to transform a given word into another kitten -> sitting = edit distance of 3 1. k -> s (substitution) 2. e -> i (substitution) 3. _ -> g (insertion)
- 46. SHOW ME THE DISTANCE
- 48. Production Developed on AWS, but need to deploy on internally We can’t use OpenNMT Ops :(, they have their reasons but FML What’s next? Tensorflow to the rescue!!!
- 49. Seq2Seq in Tensorflow Tensorflow is pain Tensorflow is a collection of a code not a “library/framework”
- 50. Let’s figure out Tensorflow 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch Videos 5. Deep Learn
- 51. Outcome Added a week of development time...BUT Simpler and Faster algorithm Basis for production deployment
- 54. SO CLOSE! We can properly correct spelling but… We also have to guess partial words correctly Query: ‘ub of the centry’ Corrected to: ‘club of the century’ ← Where we are Correct query: ‘pub of the century’ ← Where we need to go
- 55. Dictionary matching FTW Build a dictionary where the keys are phrases with 1 word dropped. { [joy luck]: “joy luck club”, [luck club]: “joy luck club”}
- 56. Does the user even care? YES
- 57. Seq2Seq + Dictionary = 90%
- 58. How much time does it add? ● Seq2Seq model adds 15ms ● Dictionary lookup adds .2ms
- 59. Why do we hard things Hard Problem (Spelling) -> Knowledge accumulation New problem -> Query parsing [‘john grisham pelican brief’] -> [author,author,title,title]
- 60. More Projects Query Parsing (Authors/Titles/Topics/Series/etc) Content Summarization (Books/Articles/Documents) Churn Prediction Document Classification (Study Guides/Court cases/etc)
- 61. Data Blog (coming in 2018) Posts on: Spelling Correction Query Tagging Multi-Armed Bandits A/B Test System Infrastructure
- 62. Work on hard data problems? Work at Scribd.
- 63. Questions Or we can all leave early!!!
Editor's Notes
- Path of least resistance
- MOAR PICTURES
- 66% + 24%