Talk given at the DSP Valley seminar "Exploring GUI design for embedded systems" together with Jan Meskens on June 9, 2009. See also: http://www.dspvalley.com/upload/event/site/5270/index.html Abstract: The User Interface Markup Language has been around for 10 years now and it is still evolving. The language did not reach a large audience yet and there are few publicly known cases where the language is used. Nevertheless, UIML is probably one of the most generic and flexible user interface markup languages available, mainly because it is in fact a meta-language that is not limited to a particular domain, device, platform or widget set. The design of the language makes it an ideal candidate to support future evolutions in user interface techniques and guarantees the survival of user interface designs over time. At the dawn of the 4.0 version of the UIML specification—which will be released soon—this talk provides an overview of the language, the novelties of the 4.0 specification and some tools that we have built and are available today. We also provide an overview of our work to integrate UIML in a multi-device user interface design process that allows designers to create user interfaces for multiple devices without having to deal with an abstract user interface specification. Finally, we will highlight our experiences with designing tailored user interfaces for embedded platforms with UIML, and discuss the pros and cons of using a high-level user interface description language on devices with limited resources.

1. How to Survive Multi-Device User
Interface Development with UIML
Jo Vermeulen and Jan Meskens
{jo.vermeulen,jan.meskens}@uhasselt.be
Hasselt University – tUL – IBBT
Expertise Centre for Digital Media
Belgium

2. A little bit about us …

3. A little bit about us …
About 80 researchers …
Specializing in …

4. A little bit about us …
About 80 researchers …
Specializing in …

5. The UIML Crew
Kris Luyten Jo Vermeulen Jan Meskens
Professor PhD student PhD student
Since 2002 Since 2004 Since 2006
• PhD about XML UIDLs • extended Uiml.net • extended Uiml.net
• developed Uiml.net • MSc thesis • developed Gummy
• several research projects • 2 applied research proj. • BSc & MSc thesis
• > 25 related publications • 6 related publications • basic research project
• numerous students • 6 students • 5 related publications
• 1 student

6. What is UIML?

7. A description of a user interface

8. UIDLs in practice

9. UIML is different
UIML is different

10. UIML is different
• High level
• Generic
– No specific tags (e.g. <button>)
• Meta: build your own User Interface Language
– cfr. XML

11. What does UIML look like?

12. What does UIML look like?
<uiml>
<interface>
<structure>
<part>...</part>
</structure>
<style>
<property>...</property>
</style>
<behavior>
<rule>...</rule>
</behavior>
</interface>
<peers>...</peers>
</uiml>

13. What does UIML look like?

14. A bit of history

15. UIML: a pioneering UIDL
1997: UIML is born.
Professor Marc Abrams (Virginia Tech, USA)
• 20 years experience with building GUIs
• Frustrated with then-current methods
• UIML 1.0 spec and Java renderer
• Founded Harmonia to provide tools & services for UIML

16. UIML: a pioneering UIDL
1998-2000: updated specification & tool support.
• 1998
– PhD dissertation by Constantinos Phanouriou.
– Corresponding UIML 2.0 spec
• 1999:
– Windows CE renderer
– Demo at WWW6 conference / W3C DI workshop
• 2000:
– HTML, WML, VoiceXML renderers

17. UIML: a pioneering UIDL
2001-2004: UIML comes of age
• 2001
– UIML 3.0 spec
– LiquidUI tool (Harmonia)
– UIML Europe 2001 conference (Paris, France)
• 2002:
– UIML Technical Committee starts
• 2003:
– Kris Luyten creates open source Uiml.net renderer
• 2004:
– UIML 3.1 specification

18. The company behind UIML
Harmonia, Inc.
• contracts with
US DoD
• e.g. UIs for
missile control

19. UIML as abstraction

20. UIML as abstraction
UIML
UI toolkits Logic
OS
Hardware

21. Same device, different toolkit
UIML UIML
Toolkit 1 … Toolkit 2 …
… …
… …
Device A Device A

22. Different device/OS/toolkit
UIML UIML
… … … …
… …
… …
Device A Device B

23. Multi-device/toolkit examples

24. Extra component necessary
UIML
UIML renderer
UI toolkits Logic
OS
Hardware

25. UIML Renderer
• Runtime environment
• Allows for:
– rapid prototyping
– dynamic changes at runtime

26. Our UIML Renderers
• .NET
– Works on most .NET-enabled platforms:
• Mono (*NIX)
• Microsoft .NET
• Compact .NET
• Java
– Runs on various Java VMs and APIs, e.g.
• Java ME
• IBM J9
• MHP Xlet

27. UIML research/teaching

28. Dynamic mobile guide UIs
Context-aware UIs
Adapt according to the
context-of-use
(e.g. location, user profile)
Dynamic changes at
runtime

29. OSGi + UIML
UIML Activator Bundle
controls rendering core and
connects with other OSGI
components

30. Distributed Applications
• MSc thesis Neal Robben
• Distributed user interfaces with Uiml.net
• Synchronization between different UI
instances

31. UIML and distributed
Applications

32. UIML and X + V
• MSc thesis Rob Van Roey
• X+V backend for UIML
• multimodal user interfaces
– GUI interaction (XHTML)
– speech (VoiceXML)

33. UIML and X + V

34. UIML on CarMelody platform
• Thesis Pieter Gevers (industrial engineering)
• Prototyping UI with Uiml.net that invokes
hardware APIs of Philips car radio
• Philips PNX0106 running
Montavista Linux

35. UIML on CarMelody platform

37. Disadvantages of UIML
• Lacks mature/available tools
• Lacks community
• Lacks documentation
• Lacks proof that it “just works” in
industrial/commercial settings

38. What about performance?

39. Smart optimization
• Rendering certainly slower
• But on the other hand, …
• The UI description provides more data,
allowing for smarter optimizations

40. Case study
• iDTV application, several linked UIML “pages”
• Painfully slow (7 seconds to render a page)
• Serious performance boost after optimization using
pre-caching and MPEG-2 frames

41. Performance comparison
8
7
7
6
5
4 Unoptimized
3 Pre-caching
2
1
0,35
0
Rendering time per page (seconds)

42. Intermediate UIML

43. Motivation
• UIML language is platform independent
• Renderers have to be created
– For different computing platforms
– Different programming languages

44. UIML for Java
UIML
Swing
AWT UIML Java
Backend renderer
Java Swing
Java AWT Logic
• Containment
• UI type conversion
OS
Hardware

45. UIML for .NET
UIML
GTK#
SWF UIML .NET
Backend renderer
GTK#
SWF Logic
OS
Hardware

46. High maintenance and
development cost
UIML UIML
Toolkit UIML .NET UIML Java Toolkit
Backend renderer renderer Backend
Toolkit Logic Logic Toolkit
OS OS
Hardware Hardware

47. Flexible Intermediate
rendering
UIML UIML
Toolkit UIML .NET Toolkit UIML Java
iRenderer iRenderer
Backend renderer Backend renderer
Toolkit Logic Toolkit Logic
OS OS
Hardware Hardware

48. Intermediate Renderer
UIML iUIML
Concrete Widget
Structure Properties
Callbacks
Style
Concrete Widget Concrete Widget
Intermediate Properties Properties
Behavior Renderer Callbacks Callbacks
Concrete Widget Concrete Widget
Properties Properties
Vocabulary Callbacks Callbacks

49. Example UIML
<d-class id=« frame »
maps-to=« System.Windows.Forms.GroupBox »>
…
</d-class>

50. Corresponding iUIML

51. Intermediate UIML
• JSON language (http://json.org)
• All UIML abstractions are resolved
– Concrete widget names
– Concrete property names
– Concrete event names
• iUIML is easy to translate into a running UI
• We built an Adobe Flex renderer in 3 days…

52. Adobe Flex Renderer

53. Design Tools

54. Motivation
• Learning UIML takes time
– Especially for non-technical designers
• Polishing a UI in UIML is difficult
– Requires switching between edit and run mode

55. Motivation
• Creating one UI for a range of devices is
challenging
• Automatic transformations often not
aesthetically pleasing
• Manual adjustment is needed

56. Design Tool: Gummy

57. Gummy
• Platform independent GUI builder
• Features
– Design GUIs graphically
– Export GUIs as UIML
– Transform GUIs from one platform to another
(based on UIML)
– Load a platform-specific design workspace
dynamically

58. Dynamic Design Workspace
Loading

59. Gummy Architecture
Desktop renderer
<uiml>
Desktop Proxy
</uiml>
PDA proxy
PDA renderer
W2P
DVB-MHP proxy
Desktop
DVB-MHP renderer

60. Problem in Mobile UI Design
Target Device
Designer Device

61. Spacing and Positioning
• Foto

62. Touch Screen Characteristics
Touch Screen Sensitivity?
Occluded Areas?

63. Interaction Techniques and
Emulators

64. Our Solution: Gummy-Live
“Mirror design
changes to
the mobile
target device
in real time”

65. Gummy-Live

66. Conclusion
Renderer Tool Support Designer Support