Managing the Machine Learning Lifecycle with MLflow
bx16_debreceni
1. Budapest University of Technology and Economics
Department of Measurement and Information Systems
Budapest University of Technology and Economics
Fault Tolerant Systems Research Group
MTA-BME Lendület Research Group on Cyber-Physical Systems
Change Propagation of View Models
by Logic Synthesis using SAT solvers
Oszkár Semeráth, Csaba Debreceni,
Ákos Horváth, Dániel Varró
BX 2016, Eindhoven
1
2. Artemis Concerto Project
Project Goal:
o Wearable sensors and smart home solutions
o Telecare systems for remote medical
examinations and diagnostics
o Important aspects: Reliability and Safety
Our Tasks:
o Component modeling
o Telecare specific code generators
o Query-based visualization with Sirius (Telecare specific views)
Forward and backward transformation of view models
2
3. Running Example: Blood Pressure Measurement
Architecture model:
Blood pressure + Pulse measure
Model queries: EMF-IncQuery
o Graph patterns
o Incremental evaluation
Well-formedness constraint:
o Error pattern: no match Valid
o Consistency of the source model
3
pt:PeriodicTrigger
measures
triggers
phone: Sensor measures
pressure: Type pulse: Type
when m2 :Measurem1: Measure
type type
pulseDone:
EventTrigger
triggeredBy
pressureDone:
EventTrigger
triggeredBy
triggers
triggers
Architecture Model
unreported(m)
m:Measurement
:Report
what NEG
WF constraint
emerg:Hostgp:Host
where where
r2 :Reportr1 :Report
what what
4. Running Example: View Models
View Model: match traceability model element in view model
Automatically created + incrementally maintained
4
dataflow(type, host)
:Measurement
:EventTrigger
:Report
host:Host
triggeredBy
type: Type
type where
what
Pressure
Pulse
GP
Emerg
when
phone: Sensor
m2 :Measure
pt:PeriodicTrigger
pulseDone:
EventTrigger
r2 :Report
emerg:Host
measures
triggeredBy
m1: Measure
pressureDone:
EventTrigger
r1 :Report
gp:Host
measures
triggeredBy
triggers
triggers
triggerspressure: Type pulse: Type
type type
where where
what what
1. Initial Model
3. View Model
2. Transfor-
mation
Architecture Model
Data-flow model
C. Debreceni, Á. Horváth, Á. Hegedüs, Z. Ujhelyi, I. Ráth, and D. Varró. Query-driven incremental synchronization of view
models. In Proceedings of the 2nd Workshop on View-Based, Aspect-Oriented and Orthographic Software Modelling, 2014.
5. Properties of the Forward Transformation
S. Hidaka, M. Tisi, J. Cabot, and Z. Hu. Feature-based classification of bidirectional
transformation approaches. Software & Systems Modeling, pages 1–22, 2015.
Forward functional transformation
Consistency definition: Unidirectional, Total target
Expressiveness: Turing incomplete
Forward propagation: Automatic, Operation-based, Live
View models: regular models
Incomplete change support: only the source can be edited
5
Contribution: Backward change propagation of View models
6. Running Example: Change in the View Model
6
dataflow(type, host)
:Measurement
:EventTrigger
:Report
host:Host
triggeredBy
type: Type
type where
what
Pressure
Pulse
GP
Emerg
when
phone: Sensor
m2 :Measure
pt:PeriodicTrigger
pulseDone:
EventTrigger
r2 :Report
emerg:Host
measures
triggeredBy
m1: Measure
pressureDone:
EventTrigger
r1 :Report
gp:Host
measures
triggeredBy
triggers
triggers
triggerspressure: Type pulse: Type
type type
where where
what what
1. Initial Model
Pressure
Pulse
GP
Emerg
3. View Model 5. Changed Model
«new»
«del»
2. Transfor-
mation
4. Change
Architecture Model
Data-flow modelData-flow models
Abstract view models can be edited (Conceptual changes)
Pulse measurements are redirected to the General Practicioner
7. Running Example: Backward Change Propagation
7
unreported(m)
m:Measurement
:Report
what NEG
dataflow(type, host)
:Measurement
:EventTrigger
:Report
host:Host
triggeredBy
type: Type
type where
what
Pressure
Pulse
GP
Emerg
when
phone: Sensor
m2 :Measure
pt:PeriodicTrigger
pulseDone:
EventTrigger
r2 :Report
emerg:Host
measures
triggeredBy
m1: Measure
pressureDone:
EventTrigger
r1 :Report
gp:Host
measures
triggeredBy
triggers
triggers
triggerspressure: Type pulse: Type
type type
where where
what what
1. Initial Model
Pressure
Pulse
GP
Emerg
3. View Model 5. Changed Model
«new»
«del»
2. Transfor-
mation
6. Change
propagation
Architecture Model
Data-flow models
WF
?
4. ChangeCreate solution candidates:
Consistent with the Views
Satisfies the well-formedness constraints
8. Running Example: Backward Change Propagation
8
unreported(m)
m:Measurement
:Report
what NEG
dataflow(type, host)
:Measurement
:EventTrigger
:Report
host:Host
triggeredBy
type: Type
type where
what
Pressure
Pulse
GP
Emerg
when
phone: Sensor
m2 :Measure
pt:PeriodicTrigger
pulseDone:
EventTrigger
r2 :Report
emerg:Host
measures
triggeredBy
m1: Measure
pressureDone:
EventTrigger
r1 :Report
gp:Host
measures
triggeredBy
triggers
triggers
triggerspressure: Type pulse: Type
type type
where where
what what
1. Initial Model
Pressure
Pulse
GP
Emerg
3. View Model 5. Changed Model
«new»
«del»
7. A.
pulseDone:
EventTrigger
«del» r2
pressureDone:
EventTrigger
r1 :Report
GP:Host
where
what «new» what
2. Transfor-
mation
Architecture Model Architecture Model Variants
Data-flow models
pulseDone:
EventTrigger
«new» :Report
pressureDone:
EventTrigger
r1: Report
GP:Host
where «new»
where
what «new» what
7. B.
«del» r2
WF
4. Change
Multiple valid solutions
Sol 1: pulse and pressure in the same message
Sol 2: two separate reports
6. Change
propagation
9. Properties of the New Backward Transformation
S. Hidaka, M. Tisi, J. Cabot, and Z. Hu. Feature-based classification of bidirectional
transformation approaches. Software & Systems Modeling, pages 1–22, 2015.
Implicit backward consistency: View models derived from
a changed source model are isomorphic to the changed view
Well-formedness: Changed source models are well-formed
State based and offline back-propagation:
changes on view stable view source changes
Interactive execution: Developer may select the most suitable one
source change
9
11. Overview of the Approach
11
𝑇 = 𝑇𝐹 + 𝑇𝑂
Target (View) Models Source ModelTrace
Change
on target
𝑇′
= 𝑇𝐹 + 𝑇 𝑁𝑀 𝑉
′
= 𝑀 𝑉
𝐹
+ 𝑀 𝑉
𝑁
difference
𝑀 𝑉 = 𝑀 𝑉
𝐹
+ 𝑀 𝑉
𝑂 𝑙𝑜𝑜𝑘𝑢𝑝
𝑙𝑜𝑜𝑘𝑢𝑝
difference
The change is traced back to changeing Trace model 𝑇 𝑇′
Fix: 𝑇𝐹
Old: 𝑇𝑂
New: 𝑇 𝑁
12. Overview of the Approach
12
𝑀𝑆 = 𝑀𝑆
𝐹
+ 𝑀𝑆
𝐶
+ 𝑀𝑆
𝑂𝑇 = 𝑇𝐹 + 𝑇𝑂
Target (View) Models Source ModelTrace
Change
on target
𝑇′
= 𝑇𝐹 + 𝑇 𝑁𝑀 𝑉
′
= 𝑀 𝑉
𝐹
+ 𝑀 𝑉
𝑁
difference
𝑙𝑜𝑜𝑘𝑢𝑝𝑀 𝑉 = 𝑀 𝑉
𝐹
+ 𝑀 𝑉
𝑂 𝑙𝑜𝑜𝑘𝑢𝑝
𝑙𝑜𝑜𝑘𝑢𝑝
difference
Impact analysis Source model is separated to three partitions
Fix: 𝑀𝑆
𝐹
(unaffected, don’t hurt)
Changing: 𝑀𝑆
𝐶
(affected,
but can not be removed)
Old: 𝑀𝑆
𝑂
(affected, can be removed)
when
phone: Sensor
m2 :Measure
pt:PeriodicTrigger
pulseDone:
EventTrigger
r2 :Report
emerg:Host
measures
triggeredBy
m1: Measure
pressureDone:
EventTrigger
r1 :Report
gp:Host
measures
triggeredBy
triggers
triggers
triggerspressure: Type pulse: Type
type type
where where
what what
13. Overview of the Approach
13
𝑀𝑆 = 𝑀𝑆
𝐹
+ 𝑀𝑆
𝐶
+ 𝑀𝑆
𝑂𝑇 = 𝑇𝐹 + 𝑇𝑂
Target (View) Models Source ModelTrace
Change
on target
𝑇′
= 𝑇𝐹 + 𝑇 𝑁𝑀 𝑉
′
= 𝑀 𝑉
𝐹
+ 𝑀 𝑉
𝑁
Logic
Solver+WF
difference
𝑙𝑜𝑜𝑘𝑢𝑝𝑀 𝑉 = 𝑀 𝑉
𝐹
+ 𝑀 𝑉
𝑂 𝑙𝑜𝑜𝑘𝑢𝑝
𝑙𝑜𝑜𝑘𝑢𝑝
difference
?
Model synthesis by logic solvers:
Language Specification
Well-formedness constraints
Changing + Fix part as Partial Model
Definition of the patterns
Matches of the changed view model
14. Overview of the Approach
14
𝑀𝑆 = 𝑀𝑆
𝐹
+ 𝑀𝑆
𝐶
+ 𝑀𝑆
𝑂𝑇 = 𝑇𝐹 + 𝑇𝑂
Target (View) Models Source ModelTrace
Change
on target
𝑇′
= 𝑇𝐹 + 𝑇 𝑁𝑀 𝑉
′
= 𝑀 𝑉
𝐹
+ 𝑀 𝑉
𝑁 𝑀𝑆
𝑁1
WF WF
Logic
Solver+WF
difference
WF
«del»
𝑀𝑆
𝑁2
𝑀𝑆
𝑁3
𝑙𝑜𝑜𝑘𝑢𝑝
𝑀𝑆
′
= 𝑀𝑆
𝐹
+ 𝑀𝑆
𝐶
+
«new»
𝑀 𝑉 = 𝑀 𝑉
𝐹
+ 𝑀 𝑉
𝑂 𝑙𝑜𝑜𝑘𝑢𝑝
𝑙𝑜𝑜𝑘𝑢𝑝
difference
Model synthesis by logic solvers:
Language Specification
Well-formedness constraints
Changing + Fix part as Partial Model
Definition of the patterns
Matches of the changed view model
Valid instance model
15. Measurement Setup
Prototype implementation using Alloy with Sat4j
Benchmark based on the healthcare domain:
o Size of the source model
o Number of changes
o Incremental vs Full generation
Average runtime in 3 runs
15
16. Measurement Results: Scalability
16
0
20
40
60
80
100
120
140
160
180
200
15 35 55 75 95 115 135 155 175
Runtime(s)
Number of objects in the original source model
|N|=0
|N|=1
|N|=2
|N|=3
|N|=4
|N|=5
|N|=6
|N|=7
|N|=8
|N|=9
|N|=10
O(|S|^3)
O(|S|^3)
O(|S|^3)
proportional to the
cube of the size
proportional to the
cube of the size
proportional to the
cube of the size
17. Measurement Results: Full vs Incremental
17
0
50
100
150
200
250
300
350
400
0 2 4 6 8 10 12 14 16 18 20
Runtime(s)
Number of intrduced new objects to the changes source model
Incr,|S|=15
Incr,|S|=27
Full,|S|=15
Full,|S|=27
proportional to the
cube of the size
Full generation
Incremental
18. Conclusion
18
Target (View) Models Source ModelTrace
Change
on target WF WF
Logic
Solver+WF
difference
WF
«del»
difference
dataflow(type, host)
:Measurement
:EventTrigger
:Report
host:Host
triggeredBy
type: Type
type where
what
Pressure
Pulse
GP
Emerg
when
phone: Sensor
m2 :Measure
pt:PeriodicTrigger
pulseDone:
EventTrigger
r2 :Report
emerg:Host
measures
triggeredBy
m1: Measure
pressureDone:
EventTrigger
r1 :Report
gp:Host
measures
triggeredBy
triggers
triggers
triggerspressure: Type pulse: Type
type type
where where
what what
1. Initial Model
Pressure
Pulse
GP
Emerg
3. View Model 5. Changed Model
«new»
«del»
7. A.
pulseDone:
EventTrigger
«del» r2
pressureDone:
EventTrigger
r1 :Report
GP:Host
where
what «new» what
2. Transfor-
mation
Architecture Model Architecture Model Variants
Data-flow models
pulseDone:
EventTrigger
«new» :Report
pressureDone:
EventTrigger
r1: Report
GP:Host
where «new»
where
what «new» what
7. B.
«del» r2
WF
4. Change
6. Change
propagation
Editor's Notes
1. Welcome everyone, my name is Csaba Debreceni from Budapest University of Technology. I would like to present our work with the title of Change Propagation of View Models with logical solvers.
2. This work is partially supported by the Artemis Concerto project which aims to support of the development of wearable sensors and smart home solutions, also the integration of telecare systems for remote examinations and diagnostics. The main aspect of project is the reliability and safety.
The task of our research group is to create the component modeling environment and to add telecare specific code generators to this environment. For the built system, we also want to provide query-based visualization with Sirius.
For this purpose, we have to provide a forward and backward transformation of view models. The forward tranformation of our solution is already presented, the main contribution of this paper is backward transformation part.
3. Our approach is illustrated on a simplified telecare system. Our smart phone is able to measure blood pressure and pulse. A periodic trigger triggers the phone periodic to measure these values. The event finished triggers trigger the job for reporting the results to a host. In our case the pulse will be reported to an emergency room in the hospital, while the blood pressure is going to be reported to a general practitioner.
We are also interested in looking for specific properties of the model. For example, we are looking for the measurements that are not reported to anywhere. For this purpose, we are using model queries defined by graph patterns. Several technology, such as EMF-IncQuery provides incremental evaluation of the queries.
These patterns can be used as well-formedness constraints. If these error patterns have no matches, the current state of the model is valid. These constraints enforce the consistency of the models.
4. As we mentioned, we already published the forward transformation, which works as follows: the operations of consists of graph patterns, where each match defines an element of the view model. The correspondence between a match and a view model element is stored in a traceability model. These view models are automatically created and maintained upon a change introduced into the source model.
5. The survey paper of Hidaka et al. defines several properties that our previous solution satisfies.
Forward functional
Unidirectional
Total Target
Turing Incomplete
Automatic
Operation-based
Live
Regular view models
INCOMPLETE CHANGE SUPPORT -> the main contribution of this report to tackle this issue
6. Lets continue with our paper. Image that a user modifies the view model as follows: redirect the pulse to the be reported to GP's office.
7. At that point, we are looking for solutions that consistent with the view model and satisfies all the well-formedness constraints.
8. As you can see in our example, there are at least two possible solutions for this changes. Both of them valid and well-formed.
9. From now, our solution satisfies the following additional properties:
Implicit backward consistency
Well-formedness
State-based and offline back-propagation
10. The overview of our approach is going to be presented on the following slides. Upon a change introduced into the view model, we are able to select all the objects that removed or modified, all the objects that are newly created, and related to these objects, we can specify the fix part of the view model, that didn't change.
11. Using the traceability model, we are able to classify all the pattern matches that are related to the fixed, old and new part of the view model.
12. From the traces, we are executing an affected part calculation, which separates the objects in the source model that are fixed, changeable, and removable. In our example, the objects are marked as follows.
13. To generate possible source solutions, we mapped this problem to first order logic constraints and passed it to SAT solver.
14. The output of the solver is a set of valid and well-formedness models.
18. To sum, we extended our previous work the backward propagation of the changes from the view model. For this purpose, we calculated the affected part of the source model. We mapped the whole problem to first order logic constraints and passed it to a logic solver component. Instead of generating a full new source model that is consistent with the view model, we only modify the affected part of the source model. because we expect that this solution is more efficient.