Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Neural Networks and Deep Learning for Physicists


Published on

Introduction to neural networks and deep learning. Seminar given by Héloïse Nonne on February 19th, 2015 at CINaM (Centre Interdisciplinaire de Nanosciences de Marseille) at Aix-Marseille University

Published in: Data & Analytics
  • Be the first to comment

Neural Networks and Deep Learning for Physicists

  1. 1. February 19th, 2015 Data Science Consulting Héloïse Nonne, Data Scientist Big Data & deep learning CINaM, Aix-Marseille University
  2. 2. Big Data?
  3. 3. Big Data? Explosion of data size Falling cost of data storage Increase of computing power “Information is the oil of the 21st century, and analytics is the combustion engine.” Peter Sondergaard, Senior Vice President, Gartner Research
  4. 4. The falling cost of data storage 1980 1990 2000 2014 300 000 $ 1 000 $ 100$ 0,1$ 1956 IBM 350 RAMAC Capacity: 3.75 MB Storage cost for 1 Go
  5. 5. Data growing exponentially • Over 90% of all the data in the world was created in the past 2 years. • Now, every year, 2 ZB are generated 1 ZB (zettabyte) = 1 trillion GB • IDC (International Data Corporation) predicts a generation of 40 ZB in 2020 • Around 100 hours of video are uploaded to YouTube every minute • Today’s datacenters occupy an area of land equal in size to almost 6,000 football fields
  6. 6. Where data comes from?
  7. 7. Two approaches to large databases Total failure rate = product of local failure rates Design for failure at software level Source; High-Tech hardware • Roughly double the cost of commodity • Roughly 5% failure rate Commodity (≠ low end) hardware • Roughly half the cost • Roughly 10-15% failure rate
  8. 8. Distribution algorithm: MapReduce Key principles of a DFS • Duplication of data • Distribution of data • Colocalization of treatments • Parallel treatments • Horizontal and vertical elasticity Hadoop Distributed File System (HDFS) / Computing Distribution of data over multiple servers
  9. 9. Yes but, what for? Big Data is about having an understanding of what your relationship is with the people who are the most important to you and an awareness of the potential in that relationship Joe Rospars, Chief Digital Strategist, Obama for America
  10. 10. Les tendances de fond du Big Data 10 La digitalisation massive des sphères économique, industrielle et sociale ouvre le champ à de nouvelles approches dans les domaines du marketing, de la finance et de l’industrie. L’enjeu pour les Directions Générales et les Directions Opérationnelles est de maîtriser cette opportunité pour faire face aux changements profonds des marchés et anticiper les évolutions des attentes des clients, des usages, des processus et des infrastructures. La Data Science ou l’art de maîtriser le Big Data tend à supplanter son aspect technologique, de part son importante stratégique. Le Big Data et la Data Science redéfinissent profondément les relations entre les métiers, la statistique et la technologie. Digitalisation des relations sociales Marketing Entreprise digitale Finance Usine digitaleIndustrie Monétisation des datas TMT/Banque
  11. 11. • Création et développement de produits spécifiques autour des technologies Big Data • Veille technologique et scientifique • Recherche et développement en Data Science • Quantmetry est un cabinet de conseil « pure player » de la Data Science et du Big Data • Nous aidons les entreprises à créer de la valeur grâce à l’analyse de leurs données • Nous sommes une équipe pluridisciplinaire de consultants, data scientists, experts Big Data • Nous appuyons nos recommandations sur des modèles mathématiques et statistiques Quantmetry : Big Data & Data Science 11
  12. 12. Quantmetry 12
  13. 13. Exemples de Projets data 13 • Marketing, ciblage • Compteurs intelligents: prédiction de consommation d’électricité ou d’eau • Identification des molécules les plus efficaces dans la chimiothérapie contre le cancer du sein • Prédiction d’occupation de station Vélib • Optimisation des routes aériennes en fonction du trafic • Prédiction de pannes sur des flottes automobiles • Prédiction de sécheresse en utilisant les photos satellites • Détection de fraude (sécurité sociale, assurance, impôts)
  14. 14. DataminingInterpretation Actions Modeling Collection Preparation • Reporting • Visualization • Analysis • Predictions Data Science Process
  15. 15. Artificial intelligence and neurons
  16. 16. Artificial intelligence (1956) 16 How to mimic the brain? Build artificial intelligences able to think and act like humans • Information travels as electric signals (spikes) along the dendrites and axon • Neuron gets activated if electric signal is higher than a threshold at the synapse • Activation is more intense if the frequency of the signal is high
  17. 17. McCulloch & Pitts, Rosenblatt (1950s) The perceptron 17 a 𝑥 = 𝑤1 𝑥1 + 𝑤2 𝑥2 + 𝑏 ℎ 𝑥 = 𝑔(𝑎 𝑥 ) Artifical neuron = a computational unit that makes a computation based on the information it gets from other neurons • 𝑥 = input vector (real valued) electric signal • 𝑤 = connection weights excitation or inhibition of the neuron • 𝑏 = neuron bias simulates a threshold (in combination with the weights) • 𝑔 = activation function Activation of the neuron
  18. 18. Activation functions 18 • Heaviside (perceptron): 𝑔 𝑎 = 1 if 𝑎 > 0 0 otherwise • Linear function 𝑔 𝑎 = 𝑎 • Sigmoid 𝑔 𝑎 = 1 1 + exp −𝑎 • Tanh 𝑔 𝑎 = 𝑒 𝑎 − 𝑒−𝑎 𝑒 𝑎 + 𝑒−𝑎 Linear function: • Does not introduce non linearity • Does not bound the output -> Not very interesting Heaviside function: • A little too harsh -> smoother activation is preferable to extract valuable information Sigmoid and tanh are commonly used (with softmax)
  19. 19. Capacity of a neuron: how much can it do? 19 Sigmoid function 𝑔 𝑎 = 1 1 + exp −𝑎 Output ∈ [𝟎, 𝟏] h x = p(y = 1|x) Interpretation: the output is the probability to belong to a given class (y = 0 or 1) x1 x2 A neuron can solve linearly separable problems
  20. 20. Boolean functions 20 0 1 1 0 0 0 1 0 x1 x2 0 1 1 0 0 1 0 0 x1 x2 0 1 1 0 0 1 1 1 x1 x2 0 1 1 0 0 0 0 1 x1 x2 OR (𝑥1, 𝑥2) AND (𝑥1, 𝑥2) AND (𝑥1, 𝑥2) AND (𝑥1, 𝑥2)
  21. 21. The XOR affair (1969) 21 Minsky and Papert (1969), Perceptrons: an introduction to computational geometry XOR (𝑥1, 𝑥2) impossible with only two layers 0 1 1 0 0 1 1 0 x1 x2 OK with three layers An intermediate layer builds a better representation (with AND functions)
  22. 22. Multilayer neural networks Can they recognize objects? Can they build their own representations like humans?
  23. 23. Towards a multiply distributed representation 23 Multiple layers neural networks Each layer is a distributed representation. The units are not mutually exclusive (neurons can all be activated simultaneously). Different from a partition of the input (the input belong to a specific cluster)
  24. 24. The treachery of images 24 The CAR concept • An infinity of possible images! • A high-level abstraction represented by pixels • Many problems: – Orientation – Perspective – Reflection – Irrelevant background
  25. 25. A CAR detector Built a CAR detector: decompose the problem • What are the different shapes? • How are they combined? • Orientation? • Perspective Pixels Low level abstraction Intermediate level abstraction … High level abstraction Car
  26. 26. Spectrum of machine learning tasks (Hinton’s view) Statistics • Low-dimensional data (<100 dimensions) • Lots of noise in the data • Little structure that can be captured by a rather simple model Main problematic: Separate true structure from noise Artificial Intelligence • High-dimensional data (>100 dimensions) • Noise should not be a problem • Huge amount of structure, very complicated Main problematic: Represent the complicated structure so that it can be learned
  27. 27. Training a NN / Learning 27 Training / learning is an optimization problem M examples with n features 𝑥1, 𝑥2, … , 𝑥 𝑛 Two class 𝟎, 𝟏 classification Prediction 1 if f x = p y = 1 x > 0.5 0 otherwise • Classification error is not a smooth function • Better optimize a smooth upper bound substitute: the loss function
  28. 28. Learning algorithm 28 Backpropagation algorithm • Invented in 1969 (Bryson and Ho) • Independently re-discovered in the mid-1980s by several groups • 1989: First successful application to deep neural network (LeCun) – Recognition of hand-written digits 1. Initialize the parameters 𝜃 = (𝑤, 𝑏) 2. For i = 1…M iterations (examples) • Each training example 𝑥 𝑡 , 𝑦 𝑡 ∆= −𝛻𝜃l f 𝑥 𝑡 ; 𝜃 , 𝑦 𝑡 − 𝜆𝛻𝜃 𝛺 𝜃 𝜃= 𝜃+𝛼∆ • The gradient tells in what direction the biggest decrease in the loss function is, i.e. how can we change the parameters to reduce the loss. • 𝛼: hyperparameter = learning rate Important things: a good loss function, an initialization method, an efficient way of computing the gradient many times (for each example!)
  29. 29. Training a NN / Learning 29 Then backpropagate -> modify (w,b) for each layer For each training example, do forward propagation -> get f(x)
  30. 30. Many tricks for training a NN 30 • Mini-batch learning • Regularization: the bias and variance • How much variance in the correct model: 𝜆 ≫ 0 • Bias: how far away from the true model are we? 𝜆 ∼ 0 • Tuning hyperparameter for a better generalization: do not optimize too much Early stopping
  31. 31. Deep learning
  32. 32. Why is it so difficult? Usually better to use only 1 layer! Why? • Underfitting situation: a very difficult optimization problem We would do better with a better optimization procedure. • Saturated units -> vanishing gradient -> updates are difficult (close to 0) • But saturation corresponds to the nonlinearity of NN, their interesting part • Overfitting situation: too many layers -> too fancy model • Not enough data!!!! -> But with big data, things tend to improve Better optimization Better initialization and better regularization
  33. 33. 2006: The Breakthrough Before 2006: training deep neural networks was unsuccessful! (except for CNN) 2006: 3 seminal papers • Hinton, Osindero, and Teh, A Fast Learning Algorithm for Deep Belief Nets Neural Computation, 2006 • Bengio, Lamblin, Popovici, Larochelle, Greedy Layer-Wise Training of Deep Networks Advances in neural information processing systems, 2007 • Ranzato, Poultney, Chopra, LeCun, Efficient Learning of Sparse Representations with an Energy-Based Model Advances in neural information processing systems, 2006
  34. 34. The main point: greedy learning Find the good representation: do it using unsupervised training -> let the neural networt learn by itself!! • Recognize the difference between a character and a random image -> try to understand instead of copying -> less overfitting and improved generalization • Unsupervised pretraining: Train layer by layer (greedy learning) -> local extraction of information -> the previous layer is seen as raw input representing features • Each layer is able to find the most common features in the training inputs (more common than random). Once a good representation has been found at each level: it can be used to initialize and successfully train a deep neural network with usual supervised gradient-base optimization (backpropagation)
  35. 35. MNIST 35
  36. 36. Result of pretraining 36 Larochelle, Bengio, Louradour, Lamblin JMLR (2009)
  37. 37. Many unsupervised learning techniques • Restricted Boltzmann machines • Stack denoising autoencoders • Semi-supervised embeddings • Stacked kernel PCA • Stacked independent subspace analysis • … Partially solves the problem of unlabelled data • Pre-train on unlabelled data • Fine-tuning using labelled data (supervised learning)
  38. 38. Pretraining does help deep learning 38 Why does unsupervised pre- training help deep learning? Erhan, Courville, Manzagol, Bengio, 2011
  39. 39. Google Brain 39 2012: Google’s Large Scale Deep Learning Experiments • an artificial neural network • computation spread across 16,000 CPUs • models with more than 1 billion connections
  40. 40. The next steps 40 Deep learning is good for: • Automatic speech recognition • Image recognition • Natural language processing • How well can deep learning be adapted to distributed systems (Big Data)? • Learning Online? • Application to other problems? • Time series (consumption prediction) • Scoring (churn prediction, marketing) • Application to clustering • How much more data?
  41. 41. Questions? @heloisenonne