The document summarizes a discussion from a workshop on machine learning systems engineering. It discusses the need for interdisciplinary research between software engineering and machine learning, as well as the need to establish rational evaluation criteria for machine learning systems that can be understood by both software engineers and machine learning researchers. The authors hope to contribute to the development of a "body of evaluation criteria" for machine learning systems from an industrial engineering perspective.

1. © Hitachi, Ltd. 2018. All rights reserved.
1st International Workshop on Machine Learning Systems Engineering
Discussion and Closing
ML verification in industry
Expectations to the research community
HITACHI, Ltd., Research & Development Group
Center for Technology Innovation -Systems Engineering
Dec 4, 2018
Hideto Ogawa, Yuichiro Nakagawa,
Naoto Sato and Hironobu Kuruma

2. © Hitachi, Ltd. 2018. All rights reserved.
Software engineering researchers in industry.
• Software testing, formal method and formal verification
• We are working for verification technology for machine-learning-based systems
1 Our position
1
Research & Development Group
ML Verification Tech.
Business Units
ML System Dev. Quality Assurance
Customers
Research Community
HITACHISoftware 2.0
Software 1.0
Deductive
development
Inductive
development
Traditional
testing methods
May be useless
Motivation

3. © Hitachi, Ltd. 2018. All rights reserved.
2 Worries and Expectations
2
Worries Expectations
1 Research field Interdisciplinary engineering research
community between SE and ML
2 Rational evaluation Evaluation criteria
3

4. © Hitachi, Ltd. 2018. All rights reserved.
2-1(1) In which research field are we ?
3
I thought this paper is a little out of scope.
However, the technical contributions and innovations are still sufficient.
This paper is very interesting and it focuses on the current topic in general.
However, one of the evaluation criteria for APSEC 2018 ERA papers is the
relevance to software engineering concepts and technologies, however,
the reviewer believes that this paper does not match this criterion.
Reviewer 1
Reviewer 3
Though we believes that we are software engineering guys,
a software engineering research community did not consider our works
in the focus of software engineering …
Reviewers’ comments to our paper:

5. © Hitachi, Ltd. 2018. All rights reserved.
2-1(2) We are in the field of …
4
Our expectation
Software engineering might includes ML = software 2.0 engineering
in its focus or establish new Interdisciplinary engineering research
community (MLSE !)
ML SEWe are HERE
Our expectation to SE community, but …
Though we believes that we are software engineering guys,
a software engineering research community did not consider our works
in the focus of software engineering …

6. © Hitachi, Ltd. 2018. All rights reserved.
There are many kinds of evaluation criteria.
2-2(1) Rational evaluation for new technology
5
Example
A technology based on DNN coverage
Software Engineers
It’s like source code
coverage technique.
It may be useful as a
testing criteria,
but does it mean for
system quality?
Some ML researchers
YES !
I got adversarial examples.
Yaaaay !!
Some ML Engineers
I do not understand
what it means for
accuracy.
Our sad experience
We failed to get the understanding of its usefulness …

7. © Hitachi, Ltd. 2018. All rights reserved.
Since ML technology is changing swiftly, I’m not sure that Body of Knowledge is useful.
Evaluation criteria can be constructed independent from technology.
We’ll be able to contribute to it from view of industrial engineering
We need a kind of body of ML evaluation criteria
2-2(2) Body of evaluation criteria (?)
6
Our expectation
• Body of ML evaluation engineering criteria
• Evaluation of novel researches based on the body