the presentation at america training program

1. A Feature-Based Design Method of Learning
Model for Deep Learning and its Evaluation
1
Ryu(Ryunosuke) Ota
Graduate Program of Software Engineering,
Graduate School of Science and Engineering
Nanzan University

2. Scenario
1. Background and research questions
2. Related work
3. Approach
4. Proposed method
5. Prototype for proposed method
6. Application to actual data
7. Evaluations
8. Conclusions
2

3. 3
RQ1: Is it possible to improve efficiency by incremental process based on features?
RQ2: Is the incremental process effective for actual data?
Background
Data collection Data generation Modeling Model evaluation
Training data Model
Development process of deep learning model is unestablished
A ad hoc development process of deep learning models [1]
Difficult to feedback
Difficult to analyze the effect of the data on models
Research questions
[1] S. Amershi, et al., Software Engineering for Machine Learning: A Case Study Proc. of ICSE
2019 Software Engineering in Practice, ACM, May 2019, pp. 291-300.
1. Background and research questions

4. 4
Improving the accuracy and speed of learning
A Technology for designing features that express the attributes of data
Purpose
Feature design [8,9]
Pipeline of machine learning model
[8] J. Li, et al., Feature Selection: A Data Perspective,
ACM Computing Surveys, Vol. 50, No. 6, Dec. 2017, 45 pages.
[9] S. Ozdemir, et al., Feature Engineering Made Easy, Packt, 2018.
2. Related work : Feature design
Target data
Data
collection
Feature design
Feature analysis
Data cleansing
Feature generation,
transformation, selection
and scaling
Optimization
algorithm
Model evaluation Learned model
Model learning
…
…
…

5. 5
Test loop
[1] Purpose setting [2] Hypothesis
setting
Train loop
[3] Data collection
[4] Feature
extraction
[7] Modeling
[9] Hypothesis
verification
[5] Feature-based
data selection
[6] Training data
generation
[11] Evaluation
[8] Evaluation
[10] Test data
generation
Evaluation with
generalization error
Evaluation with
training error
Feature-based training
data generation
Feedback
Easy analysis
Add small
amounts of data
3. Approach
Proposal of double incremental process
+Apply the idea of
incremental process
Staged development
Double process according to the
differences training and generalization error
Feature-based design

6. 6
Use VGG16 as learned model for feature extraction
Feature extraction method
[3] Data collection
Image data collection
[4] Feature extraction Feature pool
Image
data
VGG16 [11]
VGG16
output
Disconnect
output layer
Image
feature
Convolutional layer
Fully connected
layer
Store pairs of image
and its feature
4. Proposed method: (1) Feature extraction method
[11] K. Simonyan, et al., Very Deep Convolutional Networks for Large-Scale Image Recognition,
arXiv:1409.1556, Apr. 2015, 14 pages.
Feature
extraction

7. 7
Separate evaluation of training and
generalization error
Determine the learning feature set
based on V and S
Convergence is faster
if the feature set is small
Et Feature # Data #
V ! < # ≤ % -1< # ≤ !
S ! < # ≤ %
Error differences between training and generalization
& = (
)*%,,)-,).%
/
,) − ,)1%
2 = 3
!
/
{5 ) − 6 ) }
8 9 : Approximatefuncofpredictionerror
J 9 : Expectedvaluefuncofprediction error
NO: Error value at 9 epoch
n: Epoch number
RS: Reference value of R
US: Reference value of U
Evaluation index of prediction error
Convergence velocity:
Convergence Stability:
Convergence is faster
if the feature set is large
Eg Feature # Data #
V -1< # ≤ ! -1< # ≤ !
S
4. Proposed method:
(2) Feature-based data selection
r : Correlation coefficient
Et : Training Error
Eg : Generalization Error
Lower prediction error by feature-based selection
Purpose of feature-based data selection
Effect of feature-based data selection

8. 8
(1) Classify # of features
(3) Add data from selected #
of features to training data
(2) Select # of features
based on evaluation
Training error evaluation
Generalization error evaluation
[" ≤ "$ ∩ & ≤ &$]
[" ≤ "$ ∩ ( ≤ ($]
[" > "$ ∪ & > &$]
[" > "$ ∪ ( > ($]
(4) Prediction
error evaluation
Data selection process
& = ,
-./,1-21-3/
4
1- − 1-6/
" = 7
8
4
{: - − ; - }
4. Proposed method:
(3) Data selection process
Convergence velocity:
Convergence Stability:
Feature pool
800-899
700-799
# of features
600-699
Data
Training data
Select additional
data from here
Feature selection
process
Feature selection

9. Prototype for double incremental process by three-process
Number in () is LOC
Original
image data
Feature
pool
Feature extraction
Python(39)
Feature-based data selection
Python(138)
Selected dataUnselected data
Training data generation
Python(137)
Training data
Model learning
Python(Chainer [10])(110)
Display training error
(ChainerUI)
Training error Training error graph
Learned model
Test data generation
Python(33)
Model prediction
Python(Chainer)
Display generalization error
(ChainerUI)
Test data
Generalization
error
Generalization error graph
(1) Data analysis process
(2) Training process
(3) Testing process
[10] Preferred Networks, Chainer, https://chainer.org/. 9
5. Prototype for proposed method

10. 10
Apply to image classification problem
Images of plastic bottles (3 classes 4,000 images)
Target data
6. Application to actual data

11. 11
Transition of V in training error
and generalization error
Et (Training error) : V
Improved controllability of training error and generalization error
Data #
750,900
Data #
150,300,450,600
Data #
150,300,450,600
Data #
750,900
! = #
$
%
{' ( − * ( }
, - : Approximatefuncofpredictionerror
> - : Expectedvaluefuncofprediction error
n: Epoch number
Conventional methodProposed method
Eg(Generalizationerror):V
Et (Training error) : V
Convergence velocity:7. Evaluations (1/2)

12. 12
Improved accuracy
convergence
Conventional methodProposed method
C > "/$
1st
2nd
3rd
4th
Accuracy
Data # Data #
Data # Data #
7. Evaluations (2/2) Convergence coefficient: C =
& '$ ()('$)
& '" ()('")
M(n): Approximate curve connecting max values
m(n): Approximate curve connecting min values
n2: Max data #n1: Min data #
Improved V of Eg
convergence Conventional methodProposed method
Accuracy
C > "/,C > "/-
C > "/.
M(900)
m(150)
m(900)
M(150)
Eg(Generalizationerror):V
Eg(Generalizationerror):V

13. RQ1: Is it possible to improve efficiency by incremental process based on features?
RQ2: Is the incremental process effective for actual data?
13
Research questions
Double incremental process for staged development and feature-based design
Feature-based data selection process for identification of valid feature set
Proposed methods
8. Conclusions
RQ1: yes.
Feedback of model evaluation enables identification of valid feature set
Improved controllability of learning speed and stability
RQ2: yes.
Feature-based design enables control the risk of over-fitting
Evaluation

14. A Feature-Based Design Method of Learning
Model for Deep Learning and its Evaluation
14
Ryu(Ryunosuke) Ota
Graduate Program of Software Engineering,
Graduate School of Science and Engineering
Nanzan University
Thank You for Your Attention