The document summarizes a presentation on building artificial neural networks. It discusses an overview of machine learning algorithms that will be covered in upcoming sessions, including supervised and unsupervised learning methods as well as deep learning. It then provides details on feedforward neural networks, including their structure, how data is fed through the network, and how weights are learned through backpropagation and gradient descent. Applications discussed include voice recognition, object recognition, conversation bots, auto-driving cars, and gaming.

1. Yan Xu
Houston Machine Learning Meetup
April 8, 2017
Building an artificial neural network

2. Roadmap: Method
• Tour of machine learning algorithms (1 session)
• Feature engineering (1 session)
– Feature selection - Yan
• Supervised learning (4 sessions)
– Regression models -Yan
– SVM and kernel SVM - Yan
– Tree-based models - Dario
– Bayesian method - Xiaoyang
– Ensemble models - Yan
• Unsupervised learning (3 sessions)
– K-means clustering
– DBSCAN - Cheng
– Mean shift
– Agglomerative clustering – Kunal
– Spectral clustering – Yan
– Dimension reduction for data visualization - Yan
• Deep learning (4 sessions)
_ Neural network - Yan
– Convolutional neural network – Hengyang Lu
– Recurrent neural networks
– Train deep nets with open-source tools
Slides posted on:
http://www.slideshare.net/xuyangela

3. Outline
• Introduction
• Feed forward neural network
• Learning process - by Geoffrey Hinton
• Overfitting issue
• Voice recognition
• Bonus – Machine learning in Rock Facies
Classification by Cheng Zhan

4. Application: Object recognition

5. Application: Conversation Bot

6. Application: Auto-driving Car

7. Application: Gaming
lee sedol

8. What are Neural Networks?
• Models of the brain and nervous system
• Highly parallel
– Process information much more like the brain than a serial
computer
• Simple principles
Complex behaviours

9. Biological Neural Nets
• Pigeons as art experts (Watanabe et al. 1995)
– Experiment:
• Pigeon in Skinner box
• Present paintings of two different artists (e.g. Chagall / Van
Gogh)
• Reward for pecking when presented a particular artist (e.g. Van
Gogh)

11. • Pigeons were able to discriminate between Van Gogh
and Chagall with 95% accuracy (when presented with
pictures they had been trained on)
• Discrimination still 85% successful for previously
unseen paintings of the artists
• Pigeons do not simply memorise the pictures
• They can extract and recognise patterns (the ‘style’)
• They generalise from the already seen to make
predictions
• This is what neural networks (biological and artificial)
are good at (unlike conventional computer)

12. • ANNs incorporate the two fundamental components
of biological neural nets:
1. Neurones (nodes)
2. Synapses (weights)
Artificial Vs. Biological Neural Nets

13. • Neurone vs. Node

14. • Structure of a node:
• Squashing function limits node output:

15. • Synapse vs. weight

16. Feed-forward nets
• Information flow is unidirectional
• Data is presented to Input layer
• Passed on to Hidden Layer
• Passed on to Output layer
• Information is distributed
• Information processing is parallel
Internal representation (interpretation) of data

17. Feeding data through the net:
(1  0.25) + (0.5  (-1.5)) = 0.25 + (-0.75) = - 0.5
0.3775
1
1
5.0

 e
Squashing:

18. • Data is presented to the network in the form of
activations in the input layer
• Examples
– Pixel intensity (for pictures)
– Molecule concentrations (for artificial nose)
– Share prices (for stock market prediction)
• How to represent more abstract data, e.g. a name?
– Choose a pattern, e.g.
• 0-0-1 for “Chris”
• 0-1-0 for “Becky”
Data Input

19. •Structure
•Exclusive-OR
•problem
•Classes with
•meshed regions
•Most general
•region shapes
•Single-Layer
•Two-Layer
•Three-Layer
•A
•A•B
•B
•A
•A•B
•B
•A
•A•B
•B
•B
•A
•B
•A
•B
•A
Hidden Layers: Separability

20. •Gradient:
E[w]=[E/w0,… E/wn]
•(w1,w2)
•(w1+w1,w2 +w2)
•w=- E[w]
•wi=- E/wi
Learning: Gradient Descent
•Chain rule:

21. • How can we find the right weights?
By Geoffrey Hinton, Godfather of deep learning
https://www.youtube.com/watch?v=qStsskXgZq8&index
=3&list=PLnnr1O8OWc6arChulXyEhNvEuZcSHv2aV
https://www.youtube.com/watch?v=xfPz92B0rv8&index
=4&list=PLnnr1O8OWc6arChulXyEhNvEuZcSHv2aV

22. Recap:

23. Recap:

24. Overfitting: The downside of using
powerful models

25. Ways to reduce overfitting

26. Example: Voice Recognition
• Task: Learn to discriminate between two different
voices saying “Hello”
• Data
– Sources
• Steve
• David
– Format
• Frequency distribution (60 bins)
• Analogy: cochlea

27. • Network architecture
– Feed forward network
• 60 input (one for each frequency bin)
• 6 hidden
• 2 output (0-1 for “Steve”, 1-0 for “David”)

28. • Presenting the data
Steve
David

29. • Presenting the data (untrained network)
Steve
David
0.43
0.26
0.73
0.55

30. • Calculate error
Steve
David
0.43 – 0 = 0.43
0.26 –1 = 0.74
0.73 – 1 = 0.27
0.55 – 0 = 0.55

31. • Repeat process (sweep) for all training pairs
– Present data
– Calculate error
– Backpropagate error
– Adjust weights
• Repeat process multiple times

32. • Presenting the data (trained network)
Steve
David
0.01
0.99
0.99
0.01

33. Conclusion
• Learn how artificial neural network is related to
biological neural network
• Feedforward neural network
• How weights are learned through gradient descent
• Applications

34. From neural network to deep learning
• Gradient descent will not work for deep neural network (vanishing gradients).
• More effective ways to train deep networks:
• Breakthrough: Unsupervised pre-training by Geoffrey Hinton, 2006.
• Deep learning via Hessian-free optimization by Martens, 2010.
• On the importance of initialization and momentum in deep learning, 2013.
• Massive increase in computing power and massive increase in the amount of
available training data.

35. Roadmap: Method
• Tour of machine learning algorithms (1 session)
• Feature engineering (1 session)
– Feature selection - Yan
• Supervised learning (4 sessions)
– Regression models -Yan
– SVM and kernel SVM - Yan
– Tree-based models - Dario
– Bayesian method - Xiaoyang
– Ensemble models - Yan
• Unsupervised learning (3 sessions)
– K-means clustering
– DBSCAN - Cheng
– Mean shift
– Agglomerative clustering – Kunal
– Spectral clustering – Yan
– Dimension reduction for data visualization - Yan
• Deep learning (4 sessions)
_ Neural network - Yan
– Convolutional neural network – Hengyang Lu
– Recurrent neural networks
– Train deep nets with open-source tools
Slides at:
http://www.slideshare.net/xuyangela

36. Machine Learning in
Rock Facies Classification
An Application of XGBoost
Licheng Zhang
Cheng Zhan

37. Thank you
Machine learning in Oil and Gas Conference @ Houston, April 19-20:
https://energyconferencenetwork.com/machine-learning-oil-gas-2017/
20% off, PROMO code: HML
Data Disruptors Conference, ddc (energy) @ Houston, June 14
PROMO code: PROMO: HEDS49
http://www.slideshare.net/xuyangela