Deep Learning in the Real World
Jan. 29, 2018•0 likes•2,110 views
Download to read offline
Report
Engineering
The real opportunities and issues with actually using deep learning.
Lukas BiewaldFollow
Recommended
How technology shapes business innovationGiuseppe Costanza
Artificial Intelligence & Software Testing: Hype or Hysteria?Johan Steyn
AI and Machine Learning Demystified by Carol Smith at Midwest UX 2017Carol Smith
Artificial Intelligence (AI) & Machine Learning: Are You Ready?SilverTech
Demystifying AI | Mathias Vercauteren | Keynote at AI 4 Business Summit | Bru...Mathias Vercauteren
![[Datanest] AI startup in Indonesia - March 2018](https://cdn.slidesharecdn.com/ss_thumbnails/datanestaistartupinindonesia-march20181-180315041932-thumbnail.jpg?width=384&height=256&fit=bounds)
[Datanest] AI startup in Indonesia - March 2018Thibaud Plaquet
More Related Content
What's hot
Norman Sasono - Incorporating AI/ML into Your Application ArchitectureAgile Impact
Materials for getting started with data scienceihansel
Machine Learning for Non-Technical People - Turing Fest 2019Britney Muller
AI 4 Institution Leaders_Feb 2019Mahesh Kumar CV
Artificial intelligence & machine learning landscapeMOBConf
Demystifying AIAnurag Sahay
Similar to Deep Learning in the Real World
Présentation de Bruno Schroder au 20e #mforum (07/12/2016)Agence du Numérique (AdN)
Best practices in building machine learning models in Azure MLZeydy Ortiz, Ph. D.
The Sky’s the Limit – The Rise of Machine LearninInside Analysis
AI Happy Hour - Dr. Kai-Fu Lee - The Golden age of Artificial IntelligenceRicky Wong
IIPGH Webinar 1: Getting Started With Data Scienceds4good
MIT Deep Learning Basics: Introduction and Overview by Lex FridmanPeerasak C.
Recently uploaded
Instruction Set : Computer ArchitectureRitwik Mishra
VFD DRIVES TROUBLESHOOTING.pptxCONTROLS SYSTEMS
1st Ansys Technology Day in Athens, AgendaSIMTEC Software and Services
ML in Astronomy - Workshop 1.pptxAstronomyClubIITBHU
Chapter 2. Know Your Data.pptSubrata Kumer Paul
Finding Your Way in Container SecurityKsenia Peguero
Deep Learning in the Real World
- 1. Deep Learning in the Real World Lukas Biewald @L2K
- 2. CrowdFlower
- 3. O’Reilly
- 5. Excitement Around Deep Learning
- 6. Machine Learning vs Statistics Glossary (Robert Tibshirani) Machine Learning Statistics Learning Fitting Generalization Test Set Performance Supervised Learning Regression, Classification Unsupervised Learning Density estimation, clustering large grant = $1,000,000 large grant = $50,000 nice place to have a meeting: Snowbird, Utah, French Alps nice place to have a meeting: Las Vegas in August
- 8. Venture Capital Investment in Deep Learning https://medium.com/startup-grind
- 16. Medical
- 30. What are the Challenges?
- 31. Proprietary and Confidential - Do Not Distribute Human Generated Code vs Machine Generated Code
- 33. Machine Learning Projects are Really Hard to Manage
- 35. Proprietary & Confidential35 Kaggle Accuracy 0% 18% 35% 53% 70% Baseline 12-May 13-May 14-May 15-May Accuracy Accuracy of Best Performing Model
- 36. Proprietary & Confidential36 Kaggle accuracy over time 0% 20% 40% 60% 80% 13-May 14-May 15-May 16-May 17-May 18-May 19-May 31-May 16-Jun 1-Jul 7-Jul Accuracy Accuracy of the Best Performing Model
- 37. Proprietary & Confidential37 Kaggle Participation 0 350 700 1050 1400 13-May14-May15-May16-May17-May18-May19-May31-May 16-Jun 1-Jul Number of Participating Teams
- 38. Proprietary & Confidential38 Netflix Prize
- 39. Self Driving Cars - Close or Far?
- 40. Machine Learning Can Be Unpredictable and Opaque
- 49. Explainability of Neural Networks
- 50. Deep Learning Can Be Vulnerable to Hacking
- 53. Glasses Fooling Face Recognition
- 55. Machine Learning Requires Training Data
- 57. The Effect of Better Algorithms
- 58. The Effect of Better Features
- 59. The Effect of More Data
- 60. The Effect of Cleaner Data
- 61. Where Do Data Scientists Spend Their Time?
- 63. Proprietary and Confidential - Do Not Distribute CrowdFlower AI Platform: Training Data • Multiple use cases • Multiple data formats • Templatized workflow Training Data Human-in- the-loop Machine Learning
- 64. Proprietary and Confidential - Do Not Distribute CrowdFlower AI Platform: Training Data • Image labeling for self driving cars • Pixel-level categorization done by machine and humans Training Data Human-in- the-loop Machine Learning
- 65. ???
- 67. The Combination of Humans and Computers is Powerful
- 68. Advanced Chess
- 69. AI Classifier Output Human in the Loop Confident
- 72. United States Postal Service (1982)
- 76. Proprietary and Confidential - Do Not Distribute CrowdFlower AI Platform: Case Study Training Data Human-in-the- loop Machine Learning 400,000 structured support tickets create initial ML model 200,000 new support tickets per week fed into ML model 40% initial output by model; 60% handled by human review
- 77. Machine Learning Can Look at Far More Data than Humans
- 78. The Unreasonable Effectiveness of Data Revisited (Google Blog 2017)
- 79. Breakthroughs and Data Sets Alexander Wissner-Gross
- 81. Audio Set
- 83. Transfer Learning is the Future
- 84. New Data Sets
- 85. Freiburg Groceries Data Set
- 87. Inception
- 91. Visualizing Deep Learning Networks - Layer 1 https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
- 92. Visualizing Deep Learning Networks - Layer 2 https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
- 93. Visualizing Deep Learning Networks - Layer 3 https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
- 94. Visualizing Deep Learning Networks - Layer 4 https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
- 95. Visualizing Deep Learning Networks - Layer 5 https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
- 96. Using DNNs as feature extractors
- 97. Retraining Neural Networks (Fine Tuning)
- 98. Fine Tuning Accuracy Improvements 77% Accuracy With Fine Tuning 47% Accuracy Without Fine Tuning https://shuaiw.github.io/2017/03/09/smaller-faster-deep-learning-models.html
- 99. Dermatologist-level classification of skin cancer with deep neural networks
- 101. Multi Task Learning
- 107. Thank You Lukas Biewald (@L2K)
Editor's Notes
- I’m Lukas Biewald and I want to talk about Deep Learning in the Real World. What's actually working now and the problems we're likely to face in the next few years as it becomes more and more ubiquitous.
- The company I founded CrowdFlower, is a san francisco startup that’s helped companies like Bloomberg, Salesforce, Google, Coca-Cola, Home Depot and build and deploy deep learning systems so I’ve seen a lot of businesses succeed and fail.
- I also build robots in my basement, which keep me close to the applications of deep learning.
- I have to show you one of my projects- this a drone I built that recognizes chris and understands voice commands
- (5 minutes) Let’s talk about the excitement maybe hype around deep learning
- My Brother in law is a statistics professor and machine learning drives him crazy. He says machine learning takes statistical concepts, renames them and markets them. Deep learning drives him even crazier. There’s certainly truth to this table by Tibshirani.
- Gartner has a hype cycle curve and deep learning is right at the top of it.
- Venture investment s probably the best indicator of a hype bubble and it’s certainly growing exponentially over the past few years.
- Backlash articles have started to appear, maybe particularly with Watson, who has been making some especially bold claims.
- But I’m here to talk about deep learning in the real world so lets talk about what’s actually working. The set of real world applications is vast and strange. (10 minutes)
- One of the reasons deep learning is so exciting is this graph. In 2012 we saw massive improvement in image rcognition.
- I tried this in my garage labeling videos of cars outside in realtime.
- Speech recognition has had the same set of step function improvements which have led to products like the amazon echo and siri.
- So what do companies actually do with these algorithms? One thing they do is structure data. Over 80% of social media posts now contain images, this makes understanding who is talking about your brand much, much harder. You can’t just match keywords.
- A completely different application of the same technology is identifying skin cancer from images.
- Coca cola recently announced that they’ve been using deep learning to recognize rewards codes from bottle caps. This isn’t the hardest problem but there are many different cameras, lighting conditions, etc.
- Face recognition is now near human level performance as anyone that uses facebook will know. Computers can now recognize individual people along with mood and even if someone is lying.
- This is a doorbell I built that recognizes my friends.
- We can now recognize deforestation and measure climate impact in satellite photos at massive scale
- We track elephants to keep them safe from poachers.
- The TSA is starting to experiment with deep learning and has found that it can actually be more reliable than humans for detecting weapons.
- Deep learning is used to check handbags and decide if they are counterfit or now.
- Blue river builds tractors that can autamatically detect and kill weeds. They were a crowdflower customer recently bought by john deere for over $300M.
- Semantic segmentation can take images and just put bounding boxes around items but attach every pixel to a meaning such as tree or road or pedestrian.
- Real companies now use deep learning and video cameras or robots to check the placement of their products on shelves.
- 3dsignals build a system that recognizes mechanical failure by listening to engines.
- Researchers have started to build diagnostic systems for cancer just by looking at the shape and distribution of cells. Humayun did this.
- A whole ecosystem of companies has sprung up, including my company and Kaggle which was recently bought by google
- (15 minutes) But most companies really struggle to productize their Machine Learning – why does this happen?
- We have tensor flow, we have keras, why is it still hard to make deep learning projects work? I think it’s because machine generated code is fundamentally different than human generated code. On the left is some code I wrote and on the right is a model I built. If you’re a programmer you can understand the left but no one can understand the right. Other huge differences: 1) human code is at most 1mb. Machine generated code is 1gb and getting bigger. 2) human code has meaningful diffs. If I release a v2 I might edit 5% of this code. 3) We have 50 years experience debugging human code. Machine generated code is coming and it’s replacing human generated code everywhere, so how do we make it work?
- How do we know what’s easy and what’s hard?
- (20 minutes)
- I started a kaggle competiton a while back.
- In the first three days.
- After a week
- Netflix prize had the same phenomenon
- (25 minutes) We’re used to computers being predicatble and reliable and explainable. But deep learning is not
- Image net classification is spectacular on the kinds of images it was trained on.
- Here are some cases where it makes mistakes, why?
- Alpha Go crushed the best human go player, but it made on really bad move recognizable to amatures
- When machine learning gets into unfamiliar situations it can completely fall apart. My old lab made a helicopter fly upside down. But it took years. At first every time it got into trouble it would crash out of the sky. It turned out it needed to be trained on bad pilots that would consistently get into tricky situations.
- When the tesla autopilot resulted in deaths, our government asked for the code. With older style controllers its obvious where the fault lies, you can step through the code. But with ML it’s less clear – with just the code and not the data it was trained on, what does it tell you? Who is at fault – the team that wrote the code? Or the team that trained the model.
- Propublica research – risk of a criminal likely to be a repeat offender. Not a real “deep learning” model but this will be soon.
- One way to do explanations.
- (30 minutes) This might feel like science fiction but it’s a real issue. When you have a brain and you can run experiments on it, it can be very vulnerable to hacking.
- On the left the model is predicting panda. A tiny bit of noise is added. On the right the model is predicting gibbon. This might only happen on one in a trillion images, but because our code is essentially a formula, we can systematically find the images that will mess it up.
- On the left an 8. On the right it thinks the image is 5.
- Researchers have applied this to the real world. Adding custom glasses to the subject on the tops face, the facial recognition system thinks it’s the people on the bottom.
- Street signs and self driving cars may become a huge issue. By adding a little noise to a street sign, the deep learning can be tricked. Does this mean deep learning is bad? Would our brains be just as vulnerable if we could run millions of experiments on them?
- (35 minutes) Training data is essential to deep learning.
- Peter Norvig, head of research at Google, observed this in 2004 in his famous paper The Unreasonable Effectiveness of Data. I’ll quote him – circa 10 years ago: The biggest successes in natural-language-related machine learning have been speech recognition And machine translation. The reason for these successes is not that these tasks are easier than other tasks; they are in fact much harder … The reason is that a large training set is available to us in the wild. In other words the use cases that work, work because there’s lots of data available.
- Norvig wasn’t the first to notice this phenomenon. Banko and Brill at Microsoft research actually observed the same thing as Norvig a few years before by measuring the accuracy of several different machine learning algorithms what seemed like at the time a wide range of training data sets 100,00 words to a billion words. They saw an effect that we see all the time, where the algorithms are very similar in accuracy at every training set size, but consistently increase with more data. It says something amazing about our industry that a paper 10 years old feels almost like an anachronism, but a billion words feels like a tiny data set today
- A crucial piece of what my company crowdflower does is help companies build training data sets. We give our customers templates.
- Collecting training data can be tough, like with this interface labelin every pixel in an image.
- If you look at these particular algorithms they seem like they might be flattening out. Did the accuracy curve flatten out at bigger data sets? What happened? A lot of you probably know. My old professor Andrew Ng (another one of the CrowdFlower) inspirations has an answer in one of his famous slides
- Deep learning has allowed the trend to continue. Deep learning models are able to ingest and use even more data than models in the past.
- (40 minutes) So why are companies using deep learning with all these flaws? At the highest level I think it’s because deep learning is a different kind of intelligence and the combination of human and computers is really powerful.
- 20 years ago deep blue beat gary kasparov. There’s a game called advanced chess.
- There’s a simple, powerful design pattern behind a huge fraction of the successful deployments of deep learning. Human in the loop is simple and almost all of our customers that really use deep learning use some form of it. In the real world, deep learning algorithms make mistakes and we have to be able to deal with those mistakes. At its most basic you take the result of a classifier and use the output where its confident. Say we have a document classification algorithm. If it’s more than 99% sure that there is a pedestrian it decides there is a pedestiran.
- But if the algorithm is not confident it sends the output to a human operator to label. The business process can then work at a very high degree of confidence even if the algorithm is less than 100% accurate.
- Very importantly, the human annotation can be sent back to the algorithm to make it better. This process is called active learning. “Active Learning” is sending the human labels back to the classifier for retraining. The human labels can be reused used to improve the machine learning classifier over time. So less and less results are sent to a human and your business process becomes more and more efficient.
- The us post office has been doing this since 1982. They now have 99.5% accuracy but there’s still 0.5% of letters that go to a human. Without the human in the loop design patter
- Coca cola recently announced that they’ve been using deep learning to recognize rewards codes from bottle caps. This isn’t the hardest problem but there are many different cameras, lighting conditions, etc.
- So where there’s a mitake, they have the human check that the code was entered correctly.
- In the cases where the
- To give you a real world example of a crowdflower customer
- (45 minutes) Humans get bored and machines don’t. Computers can ingest more training data than a human can in millions of lifetimes. This lets deep learning do things that humans never could.
- I’ve worked on human-in-the-loop and training data for over 10 years. Why do I care so much about this? Obviously correlation doesn’t imply causation but major breakthroughs seem to to come just after the data set is made available and long after the algorithm is invented. This is a table showing just that. Speech recognition came long after hidden markov models but just after a wall street journal corpus. Google’s object classification came long after neural networks but just after the collection of imagenet, the first large image corpus. We want to help you make breakthroughs by giving you crucial data sets. And that ‘s why the think our work is so important. People might not talk as much about training data but it’s the engine that drives innovation.
- (50 minutes) Where are things going? I believe the most important development that everyone should know about is transfer learning.
- Your data set probably isnt in imagenet
- This is a little robot I built that recognized objects
- Deep learning models are typicall modeled after the human visual cortex and build in layers. The pixels come in the left and the predictions leave out the right. Each layer recognized progressively more complicated features.
- For example here’s what the first layers is seeing.
- 129,450 clinical images—two orders of magnitude larger than previous datasets12—consisting of 2,032 different diseases.
- This is a robot trained on simulations. Simulations can generate infinite amounts of training data. But computers are incredibly good and finding and exploiting flaws in the simulator. We need the simulations to be as varied and messy as the world we live in.
- But we’re working on itIn the future machine learning may be deeply tied to effectively simulating the real world. It sounds like science fiction but it’s really 3-5 years away. Unfortunately we don’t really know how to simulate things as simple as towel folding. Simulation could actually become the most important field of ML.