An Open Source Workbench for Prototyping
Multimodal Interactions Based on Off-The-Shelf
Heterogeneous Components
www.openi...
2
Outline
• Motivations
• Goals, Motivations & Scenario
• How
• Requirements
• Design overview
• Progress
• Questions
Goals & Motivations
• Multimodal Interactions Exploration Tool
• Effective prototyping (iterative, designer-centered)
• En...
5
• Fast high-fidelity prototyping
• Available Materials:
• Several interaction devices
and miscellaneous libraries
as het...
6
Workbench Scope
• Runtime:
• Facilitate efficient integration of heterogeneous
software
Lightweight middleware
• Allow ...
7
Workbench Scope (contd.)
• Design time:
Support designer  programmer collaboration
Allow dynamic exploration
• Runtim...
8
Outline
• Motivations
• Goals, Motivations & Scenario
• How
• Requirements
• Design overview
• Platform Progress
• Quest...
9
Runtime Platform Requirements,
Overview
• Useful high level programming language
support:
• C/C++, Java, Matlab, .NET …
...
10
Runtime Platform Design, Overview
• Component Oriented.
11
Runtime Platform Design, Overview
• Component Oriented
Software
• With API
• With installation/configuration doc
• Wit...
12
Runtime Platform Design, Overview
• Component Oriented
• Semi-Automatic Heterogeneous Components Integration
• Componen...
13
Heterogeneous Components
Integration
OpenInterface
Kernel
Finger Tracker
Component
C++
Speech
Component
Java
Image View...
14
OpenInterface Runtime Kernel:
Design Overview
• Component Oriented.
• Semi-Automatic Heterogeneous Components Integrati...
OpenInterface Design Platform:
Requirements ?
• Participatory design exercise
Support for:
Sketching and Dataflow progra...
OpenInterface Design Platform:
SKEMMI
• Interaction Design Exploration
• Dataflow programming
Wiimote
CDShelf
Volume Contr...
OpenInterface Design Platform:
SKEMMI
• Interaction Design Exploration
• Dataflow programming
• Design-by-example
• Visual...
OpenInterface Design Platform:
SKEMMI
• Interaction Design Exploration
• Dataflow programming
• Design-by-example
• Unifie...
SKEMMI: Sketch design level
SKEMMI: Dataflow design level
SKEMMI: Component design level
• Re-design of existing components
• Design of new components
OpenInterface Design Platform:
SKEMMI
• Interaction Design Exploration
• Dataflow programming
• Design-by-example
• Unifie...
Public Components Database
(incomplete)
https://forge.openinterface.org
Performance of OpenInterface
Workbench
• OpenInterface Runtime: OIKernel
• ~5% Memory overhead
• ~3% CPU overhead
• Manual...
Future Work
• Interaction Task design-level (on-going)
• Reusing existing task-model plug-in (from DAI-Labor)
• Advanced d...
www.openinterface.org
2
7
Questions?
28
Runtime, Pipeline
• Pipeline description language to setup a running
application.
• Rewiring through API and connectors
29
Runtime, Pipeline
30
Pipeline configuration
• Components
• Remote or local
• Connections
• Simple direct link(cross-language method call)
• ...
31
OIRuntime Current Features
• Supported Platform
• Linux and Windows and Mac OS/X
• Online component repository: https:/...
32
OIDesign-Time Current Features
• Supported Platform
• Eclipse (Linux and Windows and Mac OS/X)
• Component Design
• Dyn...
3
3
Public Connectors Database
(incomplete)
Pipeline
Wiimote
CDShelf
Volume Control
MP3 Player
C#
Java
C++ Javaacc
Navigate
setLevel
SetVolume
Event
Select
B
+-A
35
Component: OI point of view
Facets can be seen
as services provided
by a component
A Component is a
set of facets
Th...
36
OpenInterface CIDL
• Designed to describe Independent and Reusable
component.
• (Intentionally)Not as expressive as WSD...
37
Exported/Imported IO
• Components only declare communication interface
• Enforce the requirement for « independence »
•...
3
8
Dynamic Pipeline
• Connect and disconnect components at runtime
• Cannot force a device/component to implement disconn...
3
9
Runtime API
• Simple XML oriented API to control a pipeline from a third-party software
• C++ API
• Bindings to a wide...
4
0
Current Results
• Flexible Adapter/Connector model
• For extending the pipeline semantic
• Captures the generic aspect...
An Open Source Workbench for Prototyping Multimodal Interactions Based on Off-The-Shelf Heterogeneous Components
Upcoming SlideShare
Loading in …5
×

An Open Source Workbench for Prototyping Multimodal Interactions Based on Off-The-Shelf Heterogeneous Components

1,809 views
1,734 views

Published on

In this paper we present an extensible software workbench for supporting the effective and dynamic prototyping of multimodal interactive systems. We hypothesize the construction of such applications to be based on the assembly of several components, namely various and sometimes interchangeable modalities at the input, fusion-fission components, and also several modalities at the output. Successful realization of advanced interactions can benefit from early prototyping and the iterative implementation of design requires the easy integration, combination, replacement, or upgrade of components. We have designed and implemented a thin integration platform able to manage these key elements, and thus provide the research community a tool to bridge the gap of the current support for multimodal applications implementation. The platform is included within a workbench offering visual editors, non-intrusive tools, components and techniques to assemble various modalities provided in different implementation technologies, while keeping a high level of performance of the integrated system.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,809
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

An Open Source Workbench for Prototyping Multimodal Interactions Based on Off-The-Shelf Heterogeneous Components

  1. 1. An Open Source Workbench for Prototyping Multimodal Interactions Based on Off-The-Shelf Heterogeneous Components www.openinterface.org/platform Jean-Yves Lionel Lawson, Université catholique de Louvain; Ahmad-Amr Al-Akkad, Fraunhofer FIT; Jean Vanderdonckt and Benoit Macq, Université catholique de Louvain
  2. 2. 2 Outline • Motivations • Goals, Motivations & Scenario • How • Requirements • Design overview • Progress • Questions
  3. 3. Goals & Motivations • Multimodal Interactions Exploration Tool • Effective prototyping (iterative, designer-centered) • Enhanced developers-designers collaboration • Laziness • Benefit from numerous existing open source code, APIs, tools, etc: “Off-The-Shelf components” • Reuse and share
  4. 4. 5 • Fast high-fidelity prototyping • Available Materials: • Several interaction devices and miscellaneous libraries as heterogeneous software code (Matlab, C/C++, Java,.NET…) • Don’t want to: • Spend too much time on theory • Build glue code yourself • Reinvent the wheel Usage Scenario
  5. 5. 6 Workbench Scope • Runtime: • Facilitate efficient integration of heterogeneous software Lightweight middleware • Allow rapid composition of multimodal interactions Bundled generic connectors: “fusion”, multicast, smoothing, conditions, OSC, precision, conversion, etc. Seamless software connection
  6. 6. 7 Workbench Scope (contd.) • Design time: Support designer  programmer collaboration Allow dynamic exploration • Runtime evaluation • Runtime reconfiguration
  7. 7. 8 Outline • Motivations • Goals, Motivations & Scenario • How • Requirements • Design overview • Platform Progress • Questions
  8. 8. 9 Runtime Platform Requirements, Overview • Useful high level programming language support: • C/C++, Java, Matlab, .NET … •“Plug n Play” behavior •Lightweight skeleton platform.
  9. 9. 10 Runtime Platform Design, Overview • Component Oriented.
  10. 10. 11 Runtime Platform Design, Overview • Component Oriented Software • With API • With installation/configuration doc • Without explicit dependencies with other components • Dependency Injection Pattern
  11. 11. 12 Runtime Platform Design, Overview • Component Oriented • Semi-Automatic Heterogeneous Components Integration • Component Interface Description Language • Generated from source code • Proxies and Stubs • Generated from CIDL
  12. 12. 13 Heterogeneous Components Integration OpenInterface Kernel Finger Tracker Component C++ Speech Component Java Image Viewer Component Java Image Server Component Matlab Component CIDL XML CIDL XML CIDL XML CIDL XML CIDL XML CIDL XML C++ C++ C++ C++ • Standard description of interfaces and properties. • Automatic conversion of data towards a common language.
  13. 13. 14 OpenInterface Runtime Kernel: Design Overview • Component Oriented. • Semi-Automatic Heterogeneous Components Integration • Component Interface Description Language • Generated from source code • Proxies and Stubs • Generated from CIDL • Multimodal interaction and data oriented features • Extensible set of fusion and data transformation component (plugins) • Software Component Composition • Pipeline Description and Configuration Language. • Generated by front-ends or by applications
  14. 14. OpenInterface Design Platform: Requirements ? • Participatory design exercise Support for: Sketching and Dataflow programming Components development and tailoring Documentation, annotation Debugging (also as documentation) Dynamic reconfiguration Syntactic sugar
  15. 15. OpenInterface Design Platform: SKEMMI • Interaction Design Exploration • Dataflow programming Wiimote CDShelf Volume Control MP3 Player C# Java C++ Javaacc Navigate setLevel SetVolume Event Select B +-A
  16. 16. OpenInterface Design Platform: SKEMMI • Interaction Design Exploration • Dataflow programming • Design-by-example • Visualization tools • Signal processing components • Dynamic-Time-Warping (time --) • Hidden Markov Model, Bayesian Network (model --) • Model-free learning algorithms (experimental)
  17. 17. OpenInterface Design Platform: SKEMMI • Interaction Design Exploration • Dataflow programming • Design-by-example • Unifies Sketch and Implementation design levels • Integrated support for component implementation
  18. 18. SKEMMI: Sketch design level
  19. 19. SKEMMI: Dataflow design level
  20. 20. SKEMMI: Component design level • Re-design of existing components • Design of new components
  21. 21. OpenInterface Design Platform: SKEMMI • Interaction Design Exploration • Dataflow programming • Design-by-example • Unifies sketch and implementation design levels • Integrated support for component implementation • Audio and text annotation • Runtime manipulation of pipelines • Integration with component repository
  22. 22. Public Components Database (incomplete) https://forge.openinterface.org
  23. 23. Performance of OpenInterface Workbench • OpenInterface Runtime: OIKernel • ~5% Memory overhead • ~3% CPU overhead • Manual component integration • ~one hour • OpenInterface Design Platform: SKEMMI • Initial Experts Evaluation (6 Designer and Programmers) Audio Annotation Multi-level design Component tailoring o Debug -- o Advanced pipeline visual complexity --
  24. 24. Future Work • Interaction Task design-level (on-going) • Reusing existing task-model plug-in (from DAI-Labor) • Advanced debugging and performance analysis tools • High and low level logging • Step-by-step • Process monitoring • Device and interaction evaluation tools • [Schedlbauer et al.] • Formal user evaluation
  25. 25. www.openinterface.org
  26. 26. 2 7 Questions?
  27. 27. 28 Runtime, Pipeline • Pipeline description language to setup a running application. • Rewiring through API and connectors
  28. 28. 29 Runtime, Pipeline
  29. 29. 30 Pipeline configuration • Components • Remote or local • Connections • Simple direct link(cross-language method call) • Multicast, Publisher/Subscriber • (Temporal) Fusion • Rewiring • Custom (plugin) • Execution • Initialization • Threaded execution
  30. 30. 31 OIRuntime Current Features • Supported Platform • Linux and Windows and Mac OS/X • Online component repository: https://forge.openinterface.org • Supported Languages • Java,C/C++,Matlab, .NET • Description languages&API • CIDL, interfaces description • PDCL, pipeline description • Runtime API • Temporal Fusion components • Simple components distribution • Users and developers Documentation.
  31. 31. 32 OIDesign-Time Current Features • Supported Platform • Eclipse (Linux and Windows and Mac OS/X) • Component Design • Dynamic composition • Runtime debugging and evaluation
  32. 32. 3 3 Public Connectors Database (incomplete)
  33. 33. Pipeline Wiimote CDShelf Volume Control MP3 Player C# Java C++ Javaacc Navigate setLevel SetVolume Event Select B +-A
  34. 34. 35 Component: OI point of view Facets can be seen as services provided by a component A Component is a set of facets This is how any external software is seen by the Kernel
  35. 35. 36 OpenInterface CIDL • Designed to describe Independent and Reusable component. • (Intentionally)Not as expressive as WSDL (etc.) • Restricted on purpose to cover common attributes of various programming languages • Describes component interface using a programming language independent description(XML). • Not intended to be written by human (but must be readable more debug or customization) • Generation tools are available though
  36. 36. 37 Exported/Imported IO • Components only declare communication interface • Enforce the requirement for « independence » • Export IO (functions, etc…) when providing a functionality Display an image, … • Import IO for requesting a functionality An event handler
  37. 37. 3 8 Dynamic Pipeline • Connect and disconnect components at runtime • Cannot force a device/component to implement disconnection logic • Logic will be provided within platform • Adapter vs. embedded in proxies, shared access issue Switch Callback Mouse Callback Disconnectable mouse
  38. 38. 3 9 Runtime API • Simple XML oriented API to control a pipeline from a third-party software • C++ API • Bindings to a wide range of languages using SWIG • Java, Matlab, JavaScript, Perl, Tk, Python, etc. • Java bindings used by SKEMMI • Networked pipeline control • API Overview • Proxies Generator API • Generate & compile proxies • Kernel Registry API • Install, Register Components • Pipeline API • Pipeline runtime • Debug API • Debug and error information retrieval
  39. 39. 4 0 Current Results • Flexible Adapter/Connector model • For extending the pipeline semantic • Captures the generic aspect of highly reusable component • Operation defined on a set of data, rather than fixed parameters • Example: Switch, Bus, Stream, Data filter, Synchronization, Fusion • Dynamic Pipeline • Use the above model to implement a highly reusable switch adapter • API provided • Component Generator and Packager • Standard storage of OI Component • Automatic Component installation

×