Neural networks are modeled after the human brain and consist of interconnected nodes that process information using activation functions. They can be trained to recognize patterns in data and make predictions. The network is initialized with random weights and biases then trained via backpropagation to minimize an error function by adjusting the weights. Issues that can arise include overfitting, choosing the number of hidden layers and units, and multiple local minima. Bayesian neural networks place prior distributions over the weights to better model uncertainty. Ensemble methods like bagging and boosting can improve performance.

1. (Artificial) Neural Network
Putri Wikie Novianti
Reading Group
July 11, 2012

2. Analogy with human brain

3. Input Output
X1
W1
W2 Y_in Y_out
X2 F Y
. Activation functions:
. 1. Binary step function
WN
.
2. Bipolar step function
3. Sigmoid function
XN 4. Linear function
b

6. Perceptron
X1 W1
Y1
Y_in Y_out
F
W2
X2
Y2
Initialization: Stopping criterions:
- Weight (wi) and bias - Weight changes
- Learning rate (α) - Error ≠ 0
b
- Maximum epoch - Reach maximum epoch
Update weight and bias
W ij = W ij + α * tj * Xki

7. Backpropagation

8. Backpropagation

9. Backpropagation

10. Backpropagation

11. Backpropagation

12. Backpropagation

13. Backpropagation

14. Backpropagation

15. Backpropagation

16. Backpropagation

17. Backpropagation

18. Some Issues in Training NN
1. Starting values
Usually starting values for weights are chosen to be random values near zero.
Hence the model starts out nearly linear, and becomes nonlinear as the weights
increase.
Use of exact zero weights leads to zero derivatives and perfect symmetry, and the
algorithm never moves
2. Overfitting
Typically we don’t want the global minimizer of R(θ), as this is likely to be an overfit
solution. Instead some regularization is needed: this is achieved directly through a
penalty term, or indirectly by early stopping.
R(θ) : Error as a function of the complete ser of weight θ

19. Some Issues in Training NN

20. Some Issues in Training NN

21. Some Issues in Training NN
3. Scaling the input
- Since scaling the input determines the effective scaling of the weight in the
bottom layer, It can have the large effect of the final model
- It is better to standardize all inputs (mean = 0 and std.dev = 1)
- Ensure all inputs are treated equally in regulation process and allows one to
choose a meaningfull range for random starting weights.
- Typical weight for standardized inputs: random uniform weight over the range
[-0.7, 0.7]

22. Some Issues in Training NN
4. Number of hidden units and layers
- Better to have too many hidden units than too few
Too few hidden units: Less flexibility of the model (hard to capture nonlinearity)
Too many hidden units: extra weight can be shrunk towards zero with
appropriate regularization used
- Typical # of hidden units : [5, 100]
5. Multiple Minima
- The error function R(θ) is non-convex, possessing many local minima.
As result, the final solution quite depends on the choice of starting weight.
- Solution:
* averaging the predictions over the collection of networks as the final prediction
* averaging the weight
* bagging: averaging the prediction of the networks training from randomly
perturbed version of the training data.

23. Example: Simulated Data

24. Example: Simulated Data

25. Example: Simulated Data

26. Example: ZIP Code Data

27. Example: ZIP Code Data

28. Example: ZIP Code Data

29. Bayesian Neural Nets
A classification was held in 2003, by Neural Information Processing
System (NIPS) workshop
The winner: Neal and Zhang (2006) used a series of preprocessing
feature selection steps, followed by Bayesian NN, Dirichelet
diffusion trees, and combination of these methods.

30. Bayesian Neural Nets
Bayesian approach review:

31. Bayesian Neural Nets

32. Bayesian Neural Nets

33. Bayesian Neural Nets

34. Bagging and Boosting Neural Nets

36. Computational consideration

37. References
[1] Zhang, X. Support Vector Machine. Lecture slides on Data Mining course. Fall 2010, KSA:
KAUST
[2] Hastie, T., Tibshirani, R., Friedman, J. The elements of statistical learning, second edition.
2009. New York: Springer