This presentation contains the slides used in the ACM Distinguished Program lecture entitled "Pen based Gestures and Sketching" presented at University of Suceava, November 15th, 2018
1. Jean Vanderdonckt
Louvain School of Management (LSM)
Université catholique de Louvain (UCL)
Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium
jean.vanderdonckt@uclouvain.be – http://www.uclouvain.be/jean.vanderdonckt
Pen-based gestures and Sketching
2. Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium
http://www.lilab.be, http://www.lilab.eu, http://www.lilab.info
Louvain Interaction Laboratory
(LILab)
3. Who is talking?
• Jean VANDERDONCKT
– Lic. & Agreg. Mathematics, Univ. Namur, 1987
– Lic. & Master in Computer Science, Univ. Namur, 1989
– PhD in Computer Science, Univ. Namur, 1997
– Post-doc in eLearning, Univ. Lille 1, 1998
– Post-doc in HCI, Stanford Univ., 2000
• Researcher in Human-Computer Interaction (HCI) since 1988
• Full professor at UCL, Past President of LSM Research Institute
• Involvement in various R&D projects
– local level, Walloon Region
– Federal level, Belgium
– In Europe and outside (e.g., with USA)
• Scientific coordinator of the UsiXML Consortium
• See profile at
– DBLP: http://www.informatik.uni-trier.de/~ley/pers/hd/v/Vanderdonckt:Jean
– Google Scholar: http://scholar.google.com/citations?user=U-FgGrkAAAAJ&hl=fr
– Microsoft Academic: http://academic.research.microsoft.com/Author/495619.aspx
– ArnetMiner: http://arnetminer.org/person/j-vanderdonckt-1306161.html
3
4. • LinkedIn Profile: http://www.linkedin.com/in/jeanvdd
• Academia Profile:
http://uclouvain.academia.edu/JeanVanderdonckt
• YouTube:
http://www.youtube.com/results?search_query=usixml
• Slides: http://www.slideshare.net/jeanvdd
• Books
4
Who is talking?
5. • Introduction and motivations
• What Sketching as a Service is made for?
• What can we do with a Skaas tool?
• Perspectives
• Conclusion
Overview
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
6. • Ink-based interaction, pen-based interaction
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
7. • Gesture interaction (2D, 2D½ , 3D): EdgeWrite
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
8. • Gesture interaction (2D, 2D½ , 3D): DocToon
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
9. • Gesture interaction (2D, 2D½ , 3D): DocToon
– In an other room, a person is operating remotely an
avatar with gestures on WACOM Cintiq
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Backgrounds
Room cameras
Animation
10. • Drawing
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: Luca Galuzzi [CC-BY-SA-2.5] via Wikimedia Commons
11. • Sketching
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: http://min-linesonpaper.blogspot.be/2011_12_01_archive.html
12. • Sketching: workspace activity of design teams
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Tang88]
Action/function List Draw Gesture Total
Store
information
40 19 1 60 (27%)
Convey ideas 0 22 24 46 (20%)
Represent
ideas
2 41 9 52 (23%)
Engage
attention
0 21 46 67 (30%)
Total 42 (19%) 103 (46%) 80 (35%) 225 (100%)
13. • Sketching for User Interface Prototyping
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
14. • Functions of Sketching
– Externalize ideas
– Interpret each other’s ideas
– Stimulate use of early ideas
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
15. • UI design is a wicked problem
– There is no definitive statement of the problem
– The search for solutions is open ended
– Constraints are constantly changing
• UI development life cycle is
– Open
– Ill-defined
– Incomplete
– Iterative (e.g., Spiral model)
• Three strategies
– Authoritative: impose a solution (maybe sub-optimal)
– Competitive: race for a solution (maybe not the best one)
– Collaborative: gather for a solution (perhaps the most accepted)
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
16. • Observations
• Comments
– To register design sessions was a problem
– Designers and users worked together
– Both sketches on paper and pixel-perfect drawings
– “Test on the device”
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
17. • How previous work addressed the four activities
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
18. • Requirements
– R1 Support iterative design of interfaces
• Wicked Nature of UI design
– R2 Support collaboration between designers and
users
• UCD, brainstorming, collaborative solution to wicked problems
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
19. • Sketching evolves from informal to more formal
– Different levels of fidelity: low, medium, high
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Coyette2007]
Low
Check box
Identification label
Medium Final
SketchiXML
20. • Sketching evolves from informal to more formal
– Different levels of fidelity: low, medium, high
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Coyette2007]
21. • Requirements
– R1 Support iterative design of interfaces
• Wicked Nature of UI design
– R2 Support collaboration between designers and
users
• UCD, brainstorming, collaborative solution to wicked problems
– R3 Support different levels of fidelity
• Designers used sketches and pixel-perfect UIs
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
22. • Multiple devices
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Google 2012. The New Multi-screen World: Understanding Cross-platform Consumer Behavior]
23. • Multiple devices
– By the end of 2015, the number of connected mobile devices will
exceed the Earth’s population.
– By 2018 there will be 1.4 devices per person on the planet
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [CISCO Visual Networking Index
http://www.cisco.com/c/en/us/solutions/collateral/service-provider/
visual-networking-index-vni/white_paper_c11-520862.html]
24. • Multiple devices
– Design with/for multiple devices
– Responsive design = graceful degradation + progressive
enhancement
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: http://smartwebsiteguide.com/2013/11/responsive-web-design/
25. • Requirements
– R1 Support iterative design of interfaces
• Wicked Nature of UI design
– R2 Support collaboration between designers and
users
• UCD, brainstorming, collaborative solution to wicked problems
– R3 Support different levels of fidelity
• Designers used sketches and pixel-perfect UI’s
– R4 Support the design of interfaces on multiple
platforms
• Contemporary context, Designers used multiple devices
– R5 Support the prototyping of interfaces on multiple
platforms
• Contemporary context, Test on the device
Introduction and motivations
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
26. • Sketching as a Service (Skaas) is a delivery model where
the sketching activity is offered as a service for
– Multiple users
– Multiple platforms
– Multiple environments
– Multiple purposes
• A significant example:
GAMBIT
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
27. • GAMBIT configurations
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
28. • GAMBIT configurations
– Single User, Single Device (SUSD)
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
29. • GAMBIT configurations
– Single User, Multiple Devices (SUMD)
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
30. • GAMBIT configurations
– Multiple Users, Single Device (MUSD)
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
31. • GAMBIT configurations
– Multiple Users, Multiple Devices (MUMD)
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
32. • GAMBIT: infinite workspace
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
33. • GAMBIT: architecture
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
34. • GAMBIT: Step 1= define structure
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
35. • GAMBIT: Step 2= define behavior
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
36. • GAMBIT: Step 3= test
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
37. • GAMBIT: Step 3= test
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
38. • GAMBIT: Step 4= reflect: video
What Skaas is made for?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
Gambit
39. • Experiment #1
– 9 UI designers
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
40. • Experiment #1: configuration
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
41. • Experiment #1: results
– Preferred devices for Sketching were: Other (Pen and paper),
Tabletop, Wall Screen, Tablet, Smartphone
– The tool did not have all the functions designers expected (CSUQ)
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
42. • Experiment #2:
– 6 UI designers
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
43. • Experiment #2: configuration
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
44. • Experiment #2: setup
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
45. • Experiment #2: results = 11.529 words
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
46. • Experiment #2: results
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
47. • Experiment #2: results
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
48. • Experiment #2: results
– Qualitative impact – designers talked about different aspects while
using different devices
– Quantitative impact – designers talked more or less according to
which device they were using
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
Smartphone Tablet Tabletop
0
500
1000
1500
2000
Words
0
500
1000
1500
2000
Words
TabletSmartphone Tabletop
r2= 0.43
49. • Experiment #3: real-world case study
– Factory for steel galvanisation
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
50. • Experiment #3: real-world case study
– Mobile application
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
51. • Experiment #3: real-world case study
– Mobile application
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
52. • Experiment #3: configuration
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
53. • Experiment #3: brainstorming
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
54. • Experiment #3: GAMBIT prototype
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
55. • Experiment #3: setup
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
56. • Experiment #3: setup
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
57. • Experiment #3: results
– Prototyping with GAMBIT represented only 8% of the project
lifecycle and the satisfaction with the system (CSUQ) increased by
30%.
– Prototyping with multiple devices is possible on realistic contexts
What can we do with a Skaas tool?
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
Source: [Sangiorgi2014]
58. • Prototyping
different UI types:
Zoomable UIs
Perspectives
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
59. • Sketching as a Service (Skaas) is appropriate for
– Sketching any diagram
• An informal sketch: e.g., a graphical user interface (at any CRF level)
• A formal sketch: e.g., a UML model
– From any input material
• E.g., for UI design: screenshot, picture, mockup, sketch, wireframe,…
– With any level of fidelity
• Low, medium, high
• With/out recognition for transition
– On/for multiple devices
• Any interaction surface in principle
– By multiple users
• Different categories, profiles
– In various environments
• Stationary vs mobile, centralized vs distributed
• Same time vs asynchronous – Same space vs remote
Conclusion
Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
60. Ștefan cel Mare University of Suceava - Suceava, November 15th, 2018
About ACM
ACM, the Association for Computing Machinery (www.acm.org), is the
premier global community of computing professionals and students
with nearly 100,000 members in more than 170 countries interacting
with more than 2 million computing professionals worldwide.
OUR MISSION: We help computing professionals to be their best and
most creative. We connect them to their peers, to what the latest
developments, and inspire them to advance the profession and make a
positive impact on society.
OUR VISION: We see a world where computing helps solve tomorrow’s
problems – where we use our knowledge and skills to advance the
computing profession and make a positive social impact throughout the
world.
63. 11/15/2018 6325h@USV | www.eed.usv.ro/25h
Contest theme
In a world of smart devices, homes, cars, and
clothes, design smart interactive jewelry.
64. 11/15/2018 6425h@USV | www.eed.usv.ro/25h
Contest theme
In a world of smart devices, homes, cars, and
clothes, design smart interactive jewelry.
Stage #1: capture the movement of the body, or body parts,
using the acceleration sensors that were provided to you.
Stage #2: design, engineer, and demonstrate an interactive
application that makes use of this information for a practical,
useful purpose.
Stage #3 (not mandatory): include in your design other sensors,
devices, or software libraries to support your goal. Unleash your
creativity to create beautiful apps for smart jewelry!
65. 11/15/2018 6525h@USV | www.eed.usv.ro/25h
Contest theme
In a world of smart devices, homes, cars, and
clothes, design smart interactive jewelry.
Stage #1: capture the movement of the body, or body parts,
using the acceleration sensors that were provided to you.
Stage #2: design, engineer, and demonstrate an interactive
application for smart jewelry that makes use of the information
collected during stage #1 for a practical, useful purpose.
Stage #3 (not mandatory): include in your design other sensors,
devices, or software libraries to support your goal. Unleash your
creativity to create beautiful apps for smart jewelry!
66. 11/15/2018 6625h@USV | www.eed.usv.ro/25h
Contest theme
In a world of smart devices, homes, cars, and
clothes, design smart interactive jewelry.
Stage #1: capture the movement of the body, or body parts,
using the acceleration sensors that were provided to you.
Stage #2: design, engineer, and demonstrate an interactive
application for smart jewelry that makes use of the information
collected during stage #1 for a practical, useful purpose.
Stage #3 (not mandatory): include in your design other sensors,
devices, or online APIs to support your goal. Unleash your
creativity to create beautiful apps for smart jewelry!
67. 11/15/2018 6725h@USV | www.eed.usv.ro/25h
Contest theme
In a world of smart devices, homes, cars, and
clothes, design smart interactive jewelry.
Stage #1: capture the movement of the body, or body parts,
using the acceleration sensors that were provided to you.
Stage #2: design, engineer, and demonstrate an interactive
application for smart jewelry that makes use of the information
collected during stage #1 for a practical, useful purpose.
Stage #3 (not mandatory): include in your design other sensors,
devices, or online APIs to support your goal. Unleash your
creativity to create beautiful apps for smart jewelry!
This is a minimum requirement for you to qualify
for further evaluation and the awards!
68. 11/15/2018 6825h@USV | www.eed.usv.ro/25h
Contest theme
In a world of smart devices, homes, cars, and
clothes, design smart interactive jewelry.
Stage #1: capture the movement of the body, or body parts,
using the acceleration sensors that were provided to you.
Stage #2: design, engineer, and demonstrate an interactive
application for smart jewelry that makes use of the information
collected during stage #1 for a practical, useful purpose.
Stage #3 (not mandatory): include in your design other sensors,
devices, or software libraries to support your goal. Unleash your
creativity to create beautiful apps for smart jewelry!
• Total freedom to envision smart jewelry (e.g.,
ring, earrings, bracelet, necklace, brooch, etc.)
• Be creative! Keep in mind that one criteria is how
creative you were in your idea, design,
implementation, and demonstration
69. 11/15/2018 6925h@USV | www.eed.usv.ro/25h
Contest theme
In a world of smart devices, homes, cars, and
clothes, design smart interactive jewelry.
Stage #1: capture the movement of the body, or body parts,
using the acceleration sensors that were provided to you.
Stage #2: design, engineer, and demonstrate an interactive
application for smart jewelry that makes use of the information
collected during stage #1 for a practical, useful purpose.
Stage #3 (not mandatory): include in your design other sensors,
devices, or online APIs to support your goal. Unleash your
creativity to create beautiful apps for smart jewelry!
• Total freedom to make your own electronics
• Be creative! You will be evaluated by how
innovative your extended design is.
70. During your evaluation:
11/15/2018 7025h@USV | www.eed.usv.ro/25h
You will demonstrate to the jury how well your project works
The jury will try out your prototype
Communicate your solution clearly to the jury, provide
strong arguments, and answer carefully jury questions
71. During your evaluation:
11/15/2018 7125h@USV | www.eed.usv.ro/25h
You will demonstrate to the jury how well your project works
The jury will try out your prototype
Communicate your solution clearly to the jury, provide
strong arguments, and answer carefully jury questions
Evaluation criteria:
The first stage (capture motion) is mandatory to qualify
for the awards
Criterion #1 - Functionality: your prototype works (50%
of your evaluation)
Criterion #2 - Originality and creativity: your prototype
brings a new perspective on smart jewelry
Criterion #3 - Presentation: you manage to deliver a
great presentation to our jury (20%)
72. During your evaluation:
11/15/2018 7225h@USV | www.eed.usv.ro/25h
You will demonstrate to the jury how well your project works
The jury will try out your prototype
Communicate your solution clearly to the jury, provide
strong arguments, and answer carefully jury questions
Evaluation criteria:
The first stage (capture motion) is mandatory to qualify
for the awards
Criterion #1 - Functionality: your prototype works (50%
of your evaluation)
Criterion #2 - Originality and creativity: your prototype
brings a new perspective on smart jewelry
Criterion #3 - Presentation: you manage to deliver a
great presentation to our jury (20%)
1st prize = 900 RON
2nd prize = 450 RON
3rd prize = 300 RON
73. During the contest:
11/15/2018 7325h@USV | www.eed.usv.ro/25h
Free access to your own documentation, development
equipment (laptops, tablets, phones, video projectors, etc.),
and electronic devices and prototypes that you made
Free access to internet
You are free to use any development environment
The solution you propose must use the Phidgets HUB and
sensors that your received