The document discusses different types of interactions for human-computer interfaces, including gestural, tangible, and tactile interactions. It provides details on gestural interaction using motions as input, tangible interaction using physical objects, and tactile output through vibration and force feedback. It notes when each type of interaction is appropriate, such as gestural for immersive games, and when it is less appropriate, such as gestural requiring precision. It also gives examples to illustrate different interaction techniques.

1. Gestural, Tangible, and Tactile Interaction
Types of Interaction 4
Yonggeun Kim

2. Announcement

3. Announcement
1. Temporary change of venue
a. 21st September 2018 (Friday) Week 12
i. DR0001 ⇢ FCM E-Theater
2. Teaching Evaluation Trim 1, 2018/2019 (Term ID 1811)
a. https://cms.mmu.edu.my/psp/csprd/?cmd=login&languageCd=ENG&
3. Quiz (due 20th September)
a. Only for new submissions ⇢
4. Assignment 🔗
a. Due; 16th September 2018
5. Check your carry marks from MMLS announcement.

4. TOC
1. Type of Interactions
a. Basics
i. Bridging Physical and Digital: Sensors and Actuators
ii. Interface and Interaction Design of Modalities
b. Uncommon Modalities 1; Taste
c. Uncommon Modalities 2; Smell
d. Types of Interaction
i. Physical Controls
ii. Audio and Voice Interface
iii. Visual and Screen Interfaces
iv. Gestural, Tangible, and Tactile Interaction

5. Revision

6. Types of Interaction 3
Visual and screen interfaces

7. 1. Light Output
▸ LED(Light Emitting Diode)
■ on/off/blinking (pattern)/color
■ Status
▸ Are great…
■ … for glanceable and nonintrusive
information output. whether it is about
seeing something across a room, or
integrating lights in a way where they don’t
distract, lights are great to give simple
feedback without asking a lot of attention.
▸ Are less appropriate…
■ … for conveying complex information.

8. 3. Screens and displays
1. Screens and displays
a. Custom segment displays
b. Character set display
c. Dynamic displays
d. Electronic ink displays
2. Is a screen better than no screen?
a. Are great …
i. … for making physical objects dynamic.
ii. … for keeping products flexible.
b. Are less appropriate...
i. … for keeping the user experience simple.

9. The structure of memory
How can we tell which type of interactions is
appropriate or inappropriate?
Norman, Don. The Design of Everyday Things. Revised Edition, Basic Books, 2013. P.110-119
Read it.

11. LG T2516SSAVT

13. 1. Two major classes of memory
1. Short-term memory / Working memory
a. Short-term or working memory (STM) retains the most recent experiences or material that is
currently being thought about. It is the memory of the just present. Information is retained
automatically and retrieved without effort; but the amount of information that can be
retained this way is severely limited.
▸ Test yours

14. 1. Two major classes of memory
1. Short-term memory / Working memory
a. Characteristics
i. letting us remember words, names, phrases, and parts of tasks (working memory)
ii. Get distracted by some other activity and, poof, the stuff in STM disappears.
iii. The capacity of STM is surprisingly difficult to measure, because how much can be retained depends
upon the familiarity of the material.
iv. Retention is affected by both time and the number of items.
v. 5 ~ 7 ⇢ 3 ~ 5

15. 1. Two major classes of memory
1. Short-term memory / Working memory
a. Guideline
i. To maximize efficiency of working memory it is best to present different information
over different modalities: sight, sound, touch (haptics), hearing, spatial location, and
gestures.

16. 1. Two major classes of memory
2. Long-term memory
a. Long-term memory (LTM) is memory for the past. As a rule, it takes time for information to
get into LTM and time and effort to get it out again.

17. 1. Two major classes of memory
2. Long-term memory
a. Characteristics
i. We do not remember our experiences as an exact recording; rather, as bits and pieces
that are reconstructed and interpreted each time we recover the memories, which
means they are subject to all the distortions and changes that the human explanatory
mechanism imposes upon life.
ii. It is highly dependent upon how the material was interpreted in the first place.
iii. We don’t know
1. Size
2. Mechanism / method of improvement

18. 1. Two major classes of memory
2. Long-term memory
a. Characteristics
i. Long-term memory is a reconstruction of the knowledge, so it is subject to biases and
distortions. Knowledge in memory is meaningful, and at the time of retrieval, a person
might subject it to a different meaningful interpretation than is wholly accurate.
ii. We may reconstruct events the way we would prefer to remember them, rather than the
way we experienced them.

19. 2. Two major categories of memory
1. Memory for arbitrary things.
a. The items to be retained seem arbitrary, with
no meaning and no particular relationship to
one another or to things already known.
2. Memory for meaningful things
a. The items to be retained form meaningful
relationships with themselves or with other
things already known.

20. 2. Two major categories of memory
1. Memory for arbitrary things.
a. The items to be retained seem arbitrary, with
no meaning and no particular relationship to
one another or to things already known.
b. Simple remembering of the things that have
no underlying meaning or structure (requires
rote learning.)
i. Difficult
ii. Problem comes without no hint of
what has gone wrong
2. Memory for meaningful things
a. The items to be retained form meaningful
relationships with themselves or with other
things already known.
b. We can use rules and constraints to help
understand what things go together.
Meaningful structure can organize apparent
chaos and arbitrariness.

21. 1. Memory for arbitrary things.

22. Calm down.Ring me up.
2. Memory for meaningful things

23. http://gscs-b2c.lge.com/downloadFile?fileId=ui0KGa2dRM3DduWJCfmrw

24. http://gscs-b2c.lge.com/downloadFile?fileId=ui0KGa2dRM3DduWJCfmrw

25. Break

26. Types of Interaction 4
Gestural, Tangible, and
Tactile Interaction

27. 1. Gestural Interaction
▸ Input modality
■ Sending information via human gestures
either on touchable surfaces or mid-air.
➜ Kinect
https://medium.com/helm-experience-design/minority-report-15-years-later-328b15a7845a

28. 1. Gestural Interaction
▸ Input modality
■ Sending information via human gestures
either on touchable surfaces or mid-air.
➜ E.g. Kinect
The now disabled “wave to hush” feature let users silence the smoke alarm
by waving their hand at it (image: Nest Labs)

29. 1. Gestural Interaction
▸ Input
■ Is great
➜ … for video games. Gestural control
makes games more immersive and
physically challenging.
➜ … for short interactions. The “gorilla
arm” becomes a problem with lengthy
use. If gestural interaction is kept
short this is less of an issue.
➜ … when it is obvious what gestures
are possible. Users need to know or
recall what gestures are possible, or
have a guide.
▸ Input
■ Is less appropriate…
➜ … when precision and lengthy
interaction is required. Giving precise
input is hard with gestural interfaces.
Fatigue and muscle pains too.
➜ … when false positive recognition can
have serious consequences.
recognition is still unreliable and false
positives can occur.
➜ … when there is no time to learn.

30. https://www.google.com/mail/help/motion.html
https://www.youtube.com/watch?v=Bu927_ul_X0

31. 1. Gestural Interaction
▸ Input
■ Is less appropriate…
➜ … when precision and lengthy
interaction is required. Giving precise
input is hard with gestural interfaces.
Fatigue and muscle pains too.
➜ … when false positive recognition can
have serious consequences.
recognition is still unreliable and false
positives can occur.
➜ … when there is no time to learn.

32. 2. Tangible and tactile interaction
▸ Tangible user interfaces
■ Input is given through manipulating
physical objects or tokens representing
intangible assets
■ Tangible interfaces offer a way to simplify
the control of a device . The direct and
physical nature of manipulation can be
easier to learn and understand than
abstracted menu structures or conceptual
mappings of controls .
Retractable - The Tangible User Interface Instrument - Improv session

33. 2. Tangible and tactile interaction
▸ Tangible user interfaces
■ Are great
➜ … for digitally enabled experiences
that don’t feel like interacting with a
computer. For example, museum
exhibits that let visitors explore and
experiment, or musical instruments
like the reactable that lets users
interact with sounds and music.
➜ … for educational products. Learning
through manipulatives is an
important aspect of early childhood
education and approaches such as
Montessori education. Tangible user
interfaces can keep learning
experiences tactile while enhancing
them with dynamic digital content.

34. 2. Tangible and tactile interaction
▸ Tangible user interfaces
■ Are inappropriate
➜ … when keeping all parts together is
critical. If lost parts mean lost data or
lost access, the drawbacks of
tangible user interfaces might
outweigh their benefits.
➜ … when there is no time to learn. If the
application doesn’t allow for a
learning phase to let users
understand how to use the interface,
other interface types might be more
appropriate.

35. 3. Tactile output: Vibration, force feedback, and shape shifting
▸ Delivers information through tactile output
■ Vibration ⇢ phone
■ Force Feedback
➜ Touch 3D Stylus
■ Shape shifting
➜ Proverbial Wallets
➜ Materiable@MIT Tangible Media
Group

36. 3. Tactile output: Vibration, force feedback, and shape shifting
▸ Tactile output
■ Are inappropriate
➜ … for creating interfaces that demand
less attention. Tactile output can
reduce cognitive load. Instead of
interpreting visual output, it can be
faster (but less precise) to feel the
information.
■ Is less appropriate
➜ … (currently) for affordable, reliable
systems. Mechanical systems involve
friction, wear and tear, and they have
a limited lifespan.
https://help.apple.com/watch/en.lproj/static.html
https://www.theverge.com/2013/1/2/3827752/hyundai-kia-bringing-google-maps-to-car-navigation-systems

37. Project