Pen-based Gestures and Sketching User Interfaces

Jean Vanderdonckt
Louvain School of Management (LSM)
Université catholique de Louvain (UCL)
Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium
jean.vanderdonckt@uclouvain.be – http://www.uclouvain.be/jean.vanderdonckt
Pen-based gestures and Sketching

Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium
http://www.lilab.be, http://www.lilab.eu, http://www.lilab.info
Louvain Interaction Laboratory
(LILab)

Who is talking?
• Jean VANDERDONCKT
– Lic. & Agreg. Mathematics, Univ. Namur, 1987
– Lic. & Master in Computer Science, Univ. Namur, 1989
– PhD in Computer Science, Univ. Namur, 1997
– Post-doc in eLearning, Univ. Lille 1, 1998
– Post-doc in HCI, Stanford Univ., 2000
• Researcher in Human-Computer Interaction (HCI) since 1988
• Full professor at UCL, Past President of LSM Research Institute
• Involvement in various R&D projects
– local level, Walloon Region
– Federal level, Belgium
– In Europe and outside (e.g., with USA)
• Scientific coordinator of the UsiXML Consortium
• See profile at
– DBLP: http://www.informatik.uni-trier.de/~ley/pers/hd/v/Vanderdonckt:Jean
– Google Scholar: http://scholar.google.com/citations?user=U-FgGrkAAAAJ&hl=fr
– Microsoft Academic: http://academic.research.microsoft.com/Author/495619.aspx
– ArnetMiner: http://arnetminer.org/person/j-vanderdonckt-1306161.html
3

• LinkedIn Profile: http://www.linkedin.com/in/jeanvdd
• Academia Profile:
http://uclouvain.academia.edu/JeanVanderdonckt
• YouTube:
http://www.youtube.com/results?search_query=usixml
• Slides: http://www.slideshare.net/jeanvdd
• Books
4
Who is talking?

• Introduction and motivations
• What Sketching as a Service is made for?
• What can we do with a Skaas tool?
• Perspectives
• Conclusion
Overview
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018

• Ink-based interaction, pen-based interaction
Introduction and motivations

• Gesture interaction (2D, 2D½ , 3D): EdgeWrite

• Gesture interaction (2D, 2D½ , 3D): DocToon

• Gesture interaction (2D, 2D½ , 3D): DocToon
– In an other room, a person is operating remotely an
avatar with gestures on WACOM Cintiq
Backgrounds
Room cameras
Animation

• Drawing
Source: Luca Galuzzi [CC-BY-SA-2.5] via Wikimedia Commons

• Sketching
Source: http://min-linesonpaper.blogspot.be/2011_12_01_archive.html

• Sketching: workspace activity of design teams
Source: [Tang88]
Action/function List Draw Gesture Total
Store
information
40 19 1 60 (27%)
Convey ideas 0 22 24 46 (20%)
Represent
ideas
2 41 9 52 (23%)
Engage
attention
0 21 46 67 (30%)
Total 42 (19%) 103 (46%) 80 (35%) 225 (100%)

• Sketching for User Interface Prototyping

• Functions of Sketching
– Externalize ideas
– Interpret each other’s ideas
– Stimulate use of early ideas

• UI design is a wicked problem
– There is no definitive statement of the problem
– The search for solutions is open ended
– Constraints are constantly changing
• UI development life cycle is
– Open
– Ill-defined
– Incomplete
– Iterative (e.g., Spiral model)
• Three strategies
– Authoritative: impose a solution (maybe sub-optimal)
– Competitive: race for a solution (maybe not the best one)
– Collaborative: gather for a solution (perhaps the most accepted)

• Observations
• Comments
– To register design sessions was a problem
– Designers and users worked together
– Both sketches on paper and pixel-perfect drawings
– “Test on the device”
Source: [Sangiorgi2014]

• How previous work addressed the four activities

• Requirements
– R1 Support iterative design of interfaces
• Wicked Nature of UI design
– R2 Support collaboration between designers and
users
• UCD, brainstorming, collaborative solution to wicked problems

• Sketching evolves from informal to more formal
– Different levels of fidelity: low, medium, high
Source: [Coyette2007]
Low
Check box
Identification label
Medium Final
SketchiXML

• Sketching evolves from informal to more formal
– Different levels of fidelity: low, medium, high
Source: [Coyette2007]

• Requirements
users
– R3 Support different levels of fidelity
• Designers used sketches and pixel-perfect UIs

• Multiple devices
Source: [Google 2012. The New Multi-screen World: Understanding Cross-platform Consumer Behavior]

– By the end of 2015, the number of connected mobile devices will
exceed the Earth’s population.
– By 2018 there will be 1.4 devices per person on the planet
Source: [CISCO Visual Networking Index
http://www.cisco.com/c/en/us/solutions/collateral/service-provider/
visual-networking-index-vni/white_paper_c11-520862.html]

– Design with/for multiple devices
– Responsive design = graceful degradation + progressive
enhancement
Source: http://smartwebsiteguide.com/2013/11/responsive-web-design/

• Requirements
users
– R3 Support different levels of fidelity
• Designers used sketches and pixel-perfect UI’s
– R4 Support the design of interfaces on multiple
platforms
• Contemporary context, Designers used multiple devices
– R5 Support the prototyping of interfaces on multiple
platforms
• Contemporary context, Test on the device

• Sketching as a Service (Skaas) is a delivery model where
the sketching activity is offered as a service for
– Multiple users
– Multiple platforms
– Multiple environments
– Multiple purposes
• A significant example:
GAMBIT
What Skaas is made for?

• GAMBIT configurations

– Single User, Single Device (SUSD)

– Single User, Multiple Devices (SUMD)

– Multiple Users, Single Device (MUSD)

– Multiple Users, Multiple Devices (MUMD)

• GAMBIT: infinite workspace

• GAMBIT: architecture

• GAMBIT: Step 1= define structure

• GAMBIT: Step 2= define behavior

• GAMBIT: Step 3= test

• GAMBIT: Step 4= reflect: video
Gambit

• Experiment #1
– 9 UI designers
What can we do with a Skaas tool?

• Experiment #1: configuration

• Experiment #1: results
– Preferred devices for Sketching were: Other (Pen and paper),
Tabletop, Wall Screen, Tablet, Smartphone
– The tool did not have all the functions designers expected (CSUQ)

• Experiment #2:
– 6 UI designers

• Experiment #2: setup

• Experiment #2: results = 11.529 words

– Qualitative impact – designers talked about different aspects while
using different devices
– Quantitative impact – designers talked more or less according to
which device they were using
Smartphone Tablet Tabletop
0
500
1000
1500
2000
Words
0
500
1000
1500
2000
Words
TabletSmartphone Tabletop
r2= 0.43

• Experiment #3: real-world case study
– Factory for steel galvanisation

• Experiment #3: real-world case study
– Mobile application

• Experiment #3: brainstorming

• Experiment #3: GAMBIT prototype

• Experiment #3: setup

– Prototyping with GAMBIT represented only 8% of the project
lifecycle and the satisfaction with the system (CSUQ) increased by
30%.
– Prototyping with multiple devices is possible on realistic contexts

• Prototyping
different UI types:
Zoomable UIs
Perspectives

• Sketching as a Service (Skaas) is appropriate for
– Sketching any diagram
• An informal sketch: e.g., a graphical user interface (at any CRF level)
• A formal sketch: e.g., a UML model
– From any input material
• E.g., for UI design: screenshot, picture, mockup, sketch, wireframe,…
– With any level of fidelity
• Low, medium, high
• With/out recognition for transition
– On/for multiple devices
• Any interaction surface in principle
– By multiple users
• Different categories, profiles
– In various environments
• Stationary vs mobile, centralized vs distributed
• Same time vs asynchronous – Same space vs remote
Conclusion

About ACM
 ACM, the Association for Computing Machinery (www.acm.org), is the
premier global community of computing professionals and students
with nearly 100,000 members in more than 170 countries interacting
with more than 2 million computing professionals worldwide.
 OUR MISSION: We help computing professionals to be their best and
most creative. We connect them to their peers, to what the latest
developments, and inspire them to advance the profession and make a
positive impact on society.
 OUR VISION: We see a world where computing helps solve tomorrow’s
problems – where we use our knowledge and skills to advance the
computing profession and make a positive social impact throughout the
world.

Questions?

25H@USV
Students’ contest for
development of new technology
5th edition, Suceava
15-16 November 2018
6225h@USV | www.eed.usv.ro/25h

11/15/2018 6325h@USV | www.eed.usv.ro/25h
Contest theme
In a world of smart devices, homes, cars, and
clothes, design smart interactive jewelry.

Contest theme
Stage #1: capture the movement of the body, or body parts,
using the acceleration sensors that were provided to you.
Stage #2: design, engineer, and demonstrate an interactive
application that makes use of this information for a practical,
useful purpose.
Stage #3 (not mandatory): include in your design other sensors,
devices, or software libraries to support your goal. Unleash your
creativity to create beautiful apps for smart jewelry!

Contest theme
application for smart jewelry that makes use of the information
collected during stage #1 for a practical, useful purpose.

Contest theme
devices, or online APIs to support your goal. Unleash your

Contest theme
This is a minimum requirement for you to qualify
for further evaluation and the awards!

Contest theme
• Total freedom to envision smart jewelry (e.g.,
ring, earrings, bracelet, necklace, brooch, etc.)
• Be creative! Keep in mind that one criteria is how
creative you were in your idea, design,
implementation, and demonstration

Contest theme
• Total freedom to make your own electronics
• Be creative! You will be evaluated by how
innovative your extended design is.

During your evaluation:
 You will demonstrate to the jury how well your project works
 The jury will try out your prototype
 Communicate your solution clearly to the jury, provide
strong arguments, and answer carefully jury questions

 Evaluation criteria:
 The first stage (capture motion) is mandatory to qualify
for the awards
 Criterion #1 - Functionality: your prototype works (50%
of your evaluation)
 Criterion #2 - Originality and creativity: your prototype
brings a new perspective on smart jewelry
 Criterion #3 - Presentation: you manage to deliver a
great presentation to our jury (20%)

 Evaluation criteria:
 The first stage (capture motion) is mandatory to qualify
for the awards
 Criterion #1 - Functionality: your prototype works (50%
of your evaluation)
 Criterion #2 - Originality and creativity: your prototype
brings a new perspective on smart jewelry
 Criterion #3 - Presentation: you manage to deliver a
great presentation to our jury (20%)
1st prize = 900 RON
2nd prize = 450 RON
3rd prize = 300 RON

During the contest:
 Free access to your own documentation, development
equipment (laptops, tablets, phones, video projectors, etc.),
and electronic devices and prototypes that you made
 Free access to internet
 You are free to use any development environment
 The solution you propose must use the Phidgets HUB and
sensors that your received

Pen-based Gestures and Sketching User Interfaces

Recommended

Recommended

More Related Content

Similar to Pen-based Gestures and Sketching User Interfaces

Similar to Pen-based Gestures and Sketching User Interfaces (20)

More from Jean Vanderdonckt

More from Jean Vanderdonckt (20)

Recently uploaded

Recently uploaded (20)

Pen-based Gestures and Sketching User Interfaces