How to Survive Multi-Device User
Interface Development with UIML
Jo Vermeulen and Jan Meskens
{jo.vermeulen,jan.meskens}@uhasselt.be
Hasselt University – tUL – IBBT
Expertise Centre for Digital Media
Belgium

A little bit about us …

A little bit about us …
About 80 researchers …
Specializing in …

A little bit about us …
About 80 researchers …
Specializing in …

The UIML Crew
Kris Luyten Jo Vermeulen Jan Meskens
Professor PhD student PhD student
Since 2002 Since 2004 Since 2006
• PhD about XML UIDLs • extended Uiml.net • extended Uiml.net
• developed Uiml.net • MSc thesis • developed Gummy
• several research projects • 2 applied research proj. • BSc & MSc thesis
• > 25 related publications • 6 related publications • basic research project
• numerous students • 6 students • 5 related publications
• 1 student

What is UIML?

A description of a user interface

UIDLs in practice

UIML is different
UIML is different

UIML is different
• High level
• Generic
– No specific tags (e.g. <button>)
• Meta: build your own User Interface Language
– cfr. XML

What does UIML look like?

What does UIML look like?
<uiml>
<interface>
<structure>
<part>...</part>
</structure>
<style>
<property>...</property>
</style>
<behavior>
<rule>...</rule>
</behavior>
</interface>
<peers>...</peers>
</uiml>

What does UIML look like?

A bit of history

UIML: a pioneering UIDL
1997: UIML is born.
Professor Marc Abrams (Virginia Tech, USA)
• 20 years experience with building GUIs
• Frustrated with then-current methods
• UIML 1.0 spec and Java renderer
• Founded Harmonia to provide tools & services for UIML

UIML: a pioneering UIDL
1998-2000: updated specification & tool support.
• 1998
– PhD dissertation by Constantinos Phanouriou.
– Corresponding UIML 2.0 spec
• 1999:
– Windows CE renderer
– Demo at WWW6 conference / W3C DI workshop
• 2000:
– HTML, WML, VoiceXML renderers

UIML: a pioneering UIDL
2001-2004: UIML comes of age
• 2001
– UIML 3.0 spec
– LiquidUI tool (Harmonia)
– UIML Europe 2001 conference (Paris, France)
• 2002:
– UIML Technical Committee starts
• 2003:
– Kris Luyten creates open source Uiml.net renderer
• 2004:
– UIML 3.1 specification

The company behind UIML
Harmonia, Inc.
• contracts with
US DoD
• e.g. UIs for
missile control

UIML as abstraction

UIML as abstraction
UIML
UI toolkits Logic
OS
Hardware

Same device, different toolkit
UIML UIML
Toolkit 1 … Toolkit 2 …
… …
… …
Device A Device A

Different device/OS/toolkit
UIML UIML
… … … …
… …
… …
Device A Device B

Multi-device/toolkit examples

Extra component necessary
UIML
UIML renderer
UI toolkits Logic
OS
Hardware

UIML Renderer
• Runtime environment
• Allows for:
– rapid prototyping
– dynamic changes at runtime

Our UIML Renderers
• .NET
– Works on most .NET-enabled platforms:
• Mono (*NIX)
• Microsoft .NET
• Compact .NET
• Java
– Runs on various Java VMs and APIs, e.g.
• Java ME
• IBM J9
• MHP Xlet

UIML research/teaching

Dynamic mobile guide UIs
Context-aware UIs
Adapt according to the
context-of-use
(e.g. location, user profile)
Dynamic changes at
runtime

OSGi + UIML
UIML Activator Bundle
controls rendering core and
connects with other OSGI
components

Distributed Applications
• MSc thesis Neal Robben
• Distributed user interfaces with Uiml.net
• Synchronization between different UI
instances

UIML and distributed
Applications

UIML and X + V
• MSc thesis Rob Van Roey
• X+V backend for UIML
• multimodal user interfaces
– GUI interaction (XHTML)
– speech (VoiceXML)

UIML and X + V

UIML on CarMelody platform
• Thesis Pieter Gevers (industrial engineering)
• Prototyping UI with Uiml.net that invokes
hardware APIs of Philips car radio
• Philips PNX0106 running
Montavista Linux

UIML on CarMelody platform

Disadvantages of UIML
• Lacks mature/available tools
• Lacks community
• Lacks documentation
• Lacks proof that it “just works” in
industrial/commercial settings

What about performance?

Smart optimization
• Rendering certainly slower
• But on the other hand, …
• The UI description provides more data,
allowing for smarter optimizations

Case study
• iDTV application, several linked UIML “pages”
• Painfully slow (7 seconds to render a page)
• Serious performance boost after optimization using
pre-caching and MPEG-2 frames

Performance comparison
8
7
7
6
5
4 Unoptimized
3 Pre-caching
2
1
0,35
0
Rendering time per page (seconds)

Intermediate UIML

Motivation
• UIML language is platform independent
• Renderers have to be created
– For different computing platforms
– Different programming languages

UIML for Java
UIML
Swing
AWT UIML Java
Backend renderer
Java Swing
Java AWT Logic
• Containment
• UI type conversion
OS
Hardware

UIML for .NET
UIML
GTK#
SWF UIML .NET
Backend renderer
GTK#
SWF Logic
OS
Hardware

High maintenance and
development cost
UIML UIML
Toolkit UIML .NET UIML Java Toolkit
Backend renderer renderer Backend
Toolkit Logic Logic Toolkit
OS OS
Hardware Hardware

Flexible Intermediate
rendering
UIML UIML
Toolkit UIML .NET Toolkit UIML Java
iRenderer iRenderer
Backend renderer Backend renderer
Toolkit Logic Toolkit Logic
OS OS
Hardware Hardware

Intermediate Renderer
UIML iUIML
Concrete Widget
Structure Properties
Callbacks
Style
Concrete Widget Concrete Widget
Intermediate Properties Properties
Behavior Renderer Callbacks Callbacks
Concrete Widget Concrete Widget
Properties Properties
Vocabulary Callbacks Callbacks

Example UIML
<d-class id=« frame »
maps-to=« System.Windows.Forms.GroupBox »>
…
</d-class>

Corresponding iUIML

Intermediate UIML
• JSON language (http://json.org)
• All UIML abstractions are resolved
– Concrete widget names
– Concrete property names
– Concrete event names
• iUIML is easy to translate into a running UI
• We built an Adobe Flex renderer in 3 days…

Adobe Flex Renderer

Design Tools

Motivation
• Learning UIML takes time
– Especially for non-technical designers
• Polishing a UI in UIML is difficult
– Requires switching between edit and run mode

Motivation
• Creating one UI for a range of devices is
challenging
• Automatic transformations often not
aesthetically pleasing
• Manual adjustment is needed

Design Tool: Gummy

Gummy
• Platform independent GUI builder
• Features
– Design GUIs graphically
– Export GUIs as UIML
– Transform GUIs from one platform to another
(based on UIML)
– Load a platform-specific design workspace
dynamically

Dynamic Design Workspace
Loading

Gummy Architecture
Desktop renderer
<uiml>
Desktop Proxy
</uiml>
PDA proxy
PDA renderer
W2P
DVB-MHP proxy
Desktop
DVB-MHP renderer

Problem in Mobile UI Design
Target Device
Designer Device

Spacing and Positioning
• Foto

Touch Screen Characteristics
Touch Screen Sensitivity?
Occluded Areas?

Interaction Techniques and
Emulators

Our Solution: Gummy-Live
“Mirror design
changes to
the mobile
target device
in real time”

Gummy-Live

Conclusion
Renderer Tool Support Designer Support