An introduction to NUI

An Introduction
to NUI
NUI programming seminar
Jun Iio, Ph.D.
Chuo University

An introduction to NUI Copyright © Jun Iio 2
Self-introduction
●
Prof. Jun Iio, Ph.D.
Dept. of Socio-Informatics, Faculty of
Letters, Chuo University
●
1994-2013, Working for Mitsubishi Research
Institute
●
2013-2014, Assoc. Prof. of Chuo University
●
2014-Current, Prof. of Chuo University
●
Also, 2009-, Guest Assoc. Prof. of Tokyo University
of Agriculture and Technology
●
My interests are
– Software Engineering, Image Processing,
Open-Source Software, User Interface, etc.

What is NUI?
●
NUI stands for Natural User Interface
●
Improvement of the User Interface?
– CUI -> GUI -> NUI？
●
Let's look back on the history of UI...

The first step of data input methods
for computers
source: Wikimedia

Emerging personal computers
source: Wikimedia (CC BY-SA 2.0 fr)
Commodore PET 2001 computer. On display at the Musée Bolo, EPFL, Lausanne.
The making of this document was supported by Wikimedia CH.

CUI（Character User Interface）

GUI（Graphical User Interface）

WIMP
Ｗ … Windows
Ｉ … Icons
Ｍ … Menus
Ｐ … Pointers

Physical peripherals
●
Keyboard
●
Pointing devices
– Mouse
– Track-point
– Track-ball
– Touch pad
– Touch panel

Novel operations
●
Tap
– Double tap
– Tap and hold
●
Flick, drag
●
Pinch
– Pinch in
– Pinch out
●
Shake, bump(!)
– (Grab) source: http://bump.softonic.jp/iphone

Elemental technologies
●
Contact-type
– Touch interface
●
Multi touch, touch display, pressure-sensitive input
●
(Image input, acceleration sensor)
●
Contactless-type
– Space devices
●
Image input, three dimensional measurement
– Gesture interface
●
Human body modeling, inverse kinematics
●
Acceleration sensor (motion, direction, inclination)
●
Muscle potential measurement

An impact of Windows 8
source: http://fesoku.net/archives/6029501.html

Multi-touch
●
http://kivy.org/

TUIO
●
Tangible User Interface Objects

CCV（Community Core Vision）
●
Poorman's multi-touch device? Video

3D mouse

Kinect

OpenNI

Leap Motion
Video

Myo
Video

Ring
Video

Gesture interface
●
Maximum benefit
– “Direct manipulation”
– It enables to control the target directly
source: ”Make It So”

Note for the NUI designs
●
Natural User Interface development is still
in progress
●
Considerable features:
1. How to realize input methods which can not
make users tired?
2. “Mode” problem of gesture input
3. Haptic information (feedbacks of user's
operation)
●
Learning from SciFi movies
– What are applications using NUI efficiently?

References
“Make It So” (Japanese Translation)

“Minority Report”
●
A “prototype meter” for the gesture
interface
●
Lessons learned:
– A grate demo hide many flaws
– A gesture interface should understand intent

Enthusiasts for keyboards
source: http://www.pfu.fujitsu.com/hhkeyboard/hhkbpro/black.html
source: http://www.kmecs.com/products/detail_267.html

Haptic devices
●
An input was completed? or not?
●
Why does this keyboard make user feel
unpleasant?
source: http://www.itmedia.co.jp/pcuser/articles/1204/06/news033.html

Haptic devices
●
It enables the
feedbacks of user's
input by touch
information such as
vibration, resistant
force, and/or
motion.
●
Cf. “Myo uses a
pattern of vibration
as the notice for
user's input”
source: http://lab.sekaimon.com/index/index/c/58058/ym/201201/p/2/ajax/1

Grammar for the gesture input
source: Leap Motion SDK API document

Grammar for the gesture input
source: https://support.getmyo.com/hc/en-us/articles/201135025-Controlling-media-with-your-Myo-armband

Hollywood's pidgin
●
Wave to activate
●
Push to move
●
Turn to rotate
●
Swipe to dismiss
●
Point or touch to select
●
Extend the hand to shoot
●
Pinch and spread to scale

Wave to activate

Push to move

Turn to rotate

Swipe to dismiss

Point or touch to select

Extend the hand to shoot

Pinch and spread to scale

Be careful if you propose a novel
gesture out of the seven patterns
●
Another patter can
lead users
misunderstanding
or confusing
●
Considering global
rules and local rule
– Gestures in
Japanese culture
– Gestures in another
country's culture
source: Fuji television

Dependency of local cultures
source: http://matome.naver.jp/odai/2133536386030621501

An application of Kinect
●
Y. Nakano, T.
Tohyama, and S.
Kanemune,
“Comparison of
Japanese Input
Methods for Kinect” In
proc. of IPSJ-
SSS2013, 173-175.
– https://www.faceboo
k.com/media/set/?
set=a.18358989182
3257.1073741846.1
72555706260009

Applications using NUI
●
It should be considered what scene
requires NUI fundamentally?
●
In the case of promotion videos for Myo:
– Controlling drone, sphero, and UMV...
“It would be enough to use a controller!”
●
It is suitable for the situations where a
user cannot use his/her own hands
– Controlling recipe video when cooking
– tele-surgery

Future of interfaces
●
NUI does not replace {CUI, GUI}
– Programming using NUI? -> a distant idea
Practica
l
Abstrac
t
Professional
use
General
purpose
CUI
GUI
NUI

Conclusions
●
Progress of interacting user and computer:
CUI -> GUI -> NUI
– NUI study is the blue ocean and in progress
●
Elemental technologies for NUI
– multi-touch, 3D measurement, acceleration
sensors, muscle potential measurement
●
How to solve some problems in gesture UI
– Tiredness, mode-problem, haptic interface
●
How to make a grammar for gesture UI
– Not only hardware but also cultural constraints
should be considered!

An introduction to NUI

Recommended

Recommended

More Related Content

Similar to An introduction to NUI

Similar to An introduction to NUI (20)

More from Jun Iio

More from Jun Iio (20)

Recently uploaded

Recently uploaded (20)

An introduction to NUI