Firma convenzione
Politecnico di Milano e Veneranda Fabbrica del
Duomo di Milano
• Aula Magna – Rettorato
• Mercoledì 27 maggio 2015
Model-driven Development of
User Interfaces for IoT via
Domain-specific Components & Patterns
Marco Brambilla, Eric Umuhoza
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Context and Motivation
 More than 30 billion IoT objects connected by 2020
 More than 2.6 billion of smart-phone users by 2020
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Context and Motivation
(*) IN 2010:
Predictions said:
from 50 billions to 1 trillion (IBM) in 2020!!!
(*) NOW:
At the moment:
from 7 billions (Gartner) to 9 billions (IDC) to 18 billions (HIS)
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Motivation
The role of user interfaces towards IoT
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Application Modeling
Mobile Applications
Contributions
IoT-based Applications
Platform Independent
Extension
(Mobile IFML)
Platform
Specific
Extension
Design
Patterns
Modeling
Tool
Platform
Specific
Extension
IoT-
Extension
Design
Patterns
Domain
Model for IoT
Implementation
Declarative rules for
Code Generation
Monitoring
Model Interpretation
Execution
Logs
Application
Model
Data Analysis and
Visualization
Database
Firma convenzione
Politecnico di Milano e Veneranda Fabbrica del
Duomo di Milano
• Aula Magna – Rettorato
• Mercoledì 27 maggio 2015
UI Modeling
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Interaction Flow Modeling Language (IFML)
Album
Search
«Window» AlbumSearch
Album
List
«Window» Albums
Album
Details
«Window» Album
«ParameterBindingGroup»
Title  AlbumTitle
Year  AlbumYear
«ParameterBindingGroup»
SelectedAlbum  AnAlbum
«Form» «List» «Details»
EventViewContainer
View Component
Album
Deletion
«ParameterBindingGroup»
SelectedAlbum  AnAlbum
Action
Navigation Flow
ParameterBinding
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Models in software development
Models as programs
• Applications are
generated (semi)
automatically from
models
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Models in software development
Models as programs
• Applications are
generated (semi)
automatically from
models
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Mobile–specific extension of IFML
 Four main sets of mobile-specific
concepts
• Mobile Containers and Components
• Events
• Mobile Actions
• Mobile Context
Bottom-up approach
Investigation of
Mobile
Platforms
Extraction of
Common
Features
Abstraction
and
Conceptualization
 Meta-model excerpts:
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Mobile IFML - Events
 Events generated by the
interaction of the user
(gestures, …)
 Mobile container
MobileContainer
MobileEvent
«Screen» List
«List» Lists
Options
«LongPress»
«Screen» Options
«Details» List
Edit list Delete list
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Mobile IFML – Access to native features
MobileComponent
MobileAction MobileActionEvent
 Access to system features
 Native functions
 Phone sensors
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Mobile IFML – the Context object
 It assumes particular relevance in mobile apps
 The context must gather all the dimensions that characterize:
• The user's intent
• The capacity of access device
• The communication network
Firma convenzione
Politecnico di Milano e Veneranda Fabbrica del
Duomo di Milano
• Aula Magna – Rettorato
• Mercoledì 27 maggio 2015
IoT Modeling
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
IoT Extensions - Components
Get Action Set Action Plan Action
Action SystemEvent
Single Information
Event
Approaching Event
 IoT Actions
 IoT Events (approaching, repeat, etc.)
 No new ViewComponent and ViewContainer
Meta-model excerpts:
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
IoT user interaction patterns – Set
Make
Cappuccino
Turn on
100°C
 Set patterns
• One device – One operation
• One device – More operations
• More devices – One operation
• More devices – More operations
• One device – One Program
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
IoT user interaction patterns – Get
• State of a device
• Details of a device
• Information from device
• Search device
• Nearby devices
• …
 Get patterns
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Patterns- based UI modeling
 Get Information from the Device
 Get State of the Device
 Get Details of a Device
 Get Information from whole Category
 One Device One Operation
 Store Information
 Push Information
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
The full example…
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Firma convenzione
Politecnico di Milano e Veneranda Fabbrica del
Duomo di Milano
• Aula Magna – Rettorato
• Mercoledì 27 maggio 2015
Code Generation
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Code generation strategies
 CIM to several PIMs
 PIM to several PSMs
• Many other combinations
• Skipping one of the levels
Model-driven architecture (MDA)
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Code generation strategies
Model-to-Model
Transformation (M2M)
Computation
Independent Model (CIM)
Platform Specific
Model (PSM)
Platform Specific
Model (PSM)
Platform Specific
Model (PSM)
M2M
Application Code
Model-to-Text
Transformation (M2T)
FSM :Cross-platform
Framework Specific Model
PIM
Native
Code
M2TM2M
PSM
PIM
Native
Code
M2T
Native
Code
M2T
PSM
PIM
Cross-platform
Code
M2T
M2T
FSM
Cross-platform
CodePIM
M2M
(1)
(2)
(3)
(4)
(5)
MDA
Platform Independent
Model (PIM)
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
PIM to Cross-platform Code
 From PIM it generates the code required by the cross platform framework
• PhoneGap, AppCelerator Titanium, and Xamarin
 Requires only one model and one generator
• Plus tricks and tweaks!!
 USE: Multiple platform availability (and time to market) is more important than high performance
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
User Behavior Analytics Approach
Example of WebRatio Runtime log line.RTXLog,
22 Jun 2016 11:10:51,761 DEBUG [http-bio-8080-exec-5] com.webratio.rtx.core.ServiceProvider:45) -
[119354A67C7C0177D4A7F411E75BCDE7][page21][pwu6Block] Creating service: WEB-INF/descr/pwu6Block.descr
Example of enriched Log line
[119354A67C7C0177D4A7F411E75BCDE7][ViewContainer[Type:page,id:page21,..]][List[id=pwu6Block][dataBinding
=Book]][[title:"...",author:"...",...]]
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Approach - Analysis and Visualization
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Experience: the Sem-IoT-y platform
www.semioty.com
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
A Predefined
Content Model
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Experience: the Sem-IoT-y platform
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
Conclusions
 Model-driven development of mobile and IoT
applications
 Very much Domain Specific
 Content model is predefined
 Basic interaction is back to content-model driven (as
from early 2000s)
Thanks!
Questions?
Marco Brambilla
@marcobrambi
marco.brambilla@polimi.it
www.marco-brambilla.com
M Brambilla, E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns

Model-driven Development of User Interfaces for IoT via Domain-specific Components & Patterns. ICEIS 2017

  • 1.
    Firma convenzione Politecnico diMilano e Veneranda Fabbrica del Duomo di Milano • Aula Magna – Rettorato • Mercoledì 27 maggio 2015 Model-driven Development of User Interfaces for IoT via Domain-specific Components & Patterns Marco Brambilla, Eric Umuhoza
  • 2.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Context and Motivation  More than 30 billion IoT objects connected by 2020  More than 2.6 billion of smart-phone users by 2020
  • 3.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Context and Motivation (*) IN 2010: Predictions said: from 50 billions to 1 trillion (IBM) in 2020!!! (*) NOW: At the moment: from 7 billions (Gartner) to 9 billions (IDC) to 18 billions (HIS)
  • 4.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Motivation The role of user interfaces towards IoT
  • 5.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Application Modeling Mobile Applications Contributions IoT-based Applications Platform Independent Extension (Mobile IFML) Platform Specific Extension Design Patterns Modeling Tool Platform Specific Extension IoT- Extension Design Patterns Domain Model for IoT Implementation Declarative rules for Code Generation Monitoring Model Interpretation Execution Logs Application Model Data Analysis and Visualization Database
  • 6.
    Firma convenzione Politecnico diMilano e Veneranda Fabbrica del Duomo di Milano • Aula Magna – Rettorato • Mercoledì 27 maggio 2015 UI Modeling
  • 7.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Interaction Flow Modeling Language (IFML) Album Search «Window» AlbumSearch Album List «Window» Albums Album Details «Window» Album «ParameterBindingGroup» Title  AlbumTitle Year  AlbumYear «ParameterBindingGroup» SelectedAlbum  AnAlbum «Form» «List» «Details» EventViewContainer View Component Album Deletion «ParameterBindingGroup» SelectedAlbum  AnAlbum Action Navigation Flow ParameterBinding
  • 8.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Models in software development Models as programs • Applications are generated (semi) automatically from models
  • 9.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Models in software development Models as programs • Applications are generated (semi) automatically from models
  • 10.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Mobile–specific extension of IFML  Four main sets of mobile-specific concepts • Mobile Containers and Components • Events • Mobile Actions • Mobile Context Bottom-up approach Investigation of Mobile Platforms Extraction of Common Features Abstraction and Conceptualization  Meta-model excerpts:
  • 11.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Mobile IFML - Events  Events generated by the interaction of the user (gestures, …)  Mobile container MobileContainer MobileEvent «Screen» List «List» Lists Options «LongPress» «Screen» Options «Details» List Edit list Delete list
  • 12.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Mobile IFML – Access to native features MobileComponent MobileAction MobileActionEvent  Access to system features  Native functions  Phone sensors
  • 13.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Mobile IFML – the Context object  It assumes particular relevance in mobile apps  The context must gather all the dimensions that characterize: • The user's intent • The capacity of access device • The communication network
  • 14.
    Firma convenzione Politecnico diMilano e Veneranda Fabbrica del Duomo di Milano • Aula Magna – Rettorato • Mercoledì 27 maggio 2015 IoT Modeling
  • 15.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns IoT Extensions - Components Get Action Set Action Plan Action Action SystemEvent Single Information Event Approaching Event  IoT Actions  IoT Events (approaching, repeat, etc.)  No new ViewComponent and ViewContainer Meta-model excerpts:
  • 16.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns IoT user interaction patterns – Set Make Cappuccino Turn on 100°C  Set patterns • One device – One operation • One device – More operations • More devices – One operation • More devices – More operations • One device – One Program
  • 17.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns IoT user interaction patterns – Get • State of a device • Details of a device • Information from device • Search device • Nearby devices • …  Get patterns
  • 18.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Patterns- based UI modeling  Get Information from the Device  Get State of the Device  Get Details of a Device  Get Information from whole Category  One Device One Operation  Store Information  Push Information
  • 19.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns The full example…
  • 20.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns
  • 21.
    Firma convenzione Politecnico diMilano e Veneranda Fabbrica del Duomo di Milano • Aula Magna – Rettorato • Mercoledì 27 maggio 2015 Code Generation
  • 22.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Code generation strategies  CIM to several PIMs  PIM to several PSMs • Many other combinations • Skipping one of the levels Model-driven architecture (MDA)
  • 23.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Code generation strategies Model-to-Model Transformation (M2M) Computation Independent Model (CIM) Platform Specific Model (PSM) Platform Specific Model (PSM) Platform Specific Model (PSM) M2M Application Code Model-to-Text Transformation (M2T) FSM :Cross-platform Framework Specific Model PIM Native Code M2TM2M PSM PIM Native Code M2T Native Code M2T PSM PIM Cross-platform Code M2T M2T FSM Cross-platform CodePIM M2M (1) (2) (3) (4) (5) MDA Platform Independent Model (PIM)
  • 24.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns PIM to Cross-platform Code  From PIM it generates the code required by the cross platform framework • PhoneGap, AppCelerator Titanium, and Xamarin  Requires only one model and one generator • Plus tricks and tweaks!!  USE: Multiple platform availability (and time to market) is more important than high performance
  • 25.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns User Behavior Analytics Approach Example of WebRatio Runtime log line.RTXLog, 22 Jun 2016 11:10:51,761 DEBUG [http-bio-8080-exec-5] com.webratio.rtx.core.ServiceProvider:45) - [119354A67C7C0177D4A7F411E75BCDE7][page21][pwu6Block] Creating service: WEB-INF/descr/pwu6Block.descr Example of enriched Log line [119354A67C7C0177D4A7F411E75BCDE7][ViewContainer[Type:page,id:page21,..]][List[id=pwu6Block][dataBinding =Book]][[title:"...",author:"...",...]]
  • 26.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Approach - Analysis and Visualization
  • 27.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Experience: the Sem-IoT-y platform www.semioty.com
  • 28.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns A Predefined Content Model
  • 29.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Experience: the Sem-IoT-y platform
  • 30.
    M Brambilla, E.Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns Conclusions  Model-driven development of mobile and IoT applications  Very much Domain Specific  Content model is predefined  Basic interaction is back to content-model driven (as from early 2000s)
  • 31.
    Thanks! Questions? Marco Brambilla @marcobrambi marco.brambilla@polimi.it www.marco-brambilla.com M Brambilla,E. Umuhoza. Model-driven Development of UIs for IoT via Domain-specific Components and Patterns

Editor's Notes

  • #3 Mobile devices as most common computing devices Fragmented market of mobile OSs Cross-platform development Cost for cross-platform and multi-device apps development A barrier for IT solution providers especially for SMEs
  • #4 Mobile devices as most common computing devices Fragmented market of mobile OSs Cross-platform development Cost for cross-platform and multi-device apps development A barrier for IT solution providers especially for SMEs
  • #11 Two main motivation: Complexity of interaction patterns to better exploit the limited space Interaction with other hardware and software components(sensors, apps,…)
  • #12 Three categories of mobile events: Events generated by the interaction of the user such as LongPress, swipe, etc. Events triggered by the mobile device features such as sensors, battery, etc. Events triggered by user actions related to the device components such as taking a photo, recording a video or using the microphone
  • #26 The process starts by integrating the runtime data (Web Server access log and WebRatio runtime log) with the structural information gathered from the conceptual models of the application (content and interaction flow models); then it enriches the integrated logs with the database objects used to populate the requested pages. The third phase denormalizes the rich logs; and finally we perform the user interaction data analysis and visualize the results.
  • #31 The process starts by integrating the runtime data (Web Server access log and WebRatio runtime log) with the structural information gathered from the conceptual models of the application (content and interaction flow models); then it enriches the integrated logs with the database objects used to populate the requested pages. The third phase denormalizes the rich logs; and finally we perform the user interaction data analysis and visualize the results.