Multimodal Interfaces November, 2008 Thomas Winningham

November, 2008

Thomas Winningham

Multi ah-wha? Computer vision, mostly The Wiimote 100hz camera Capacitive surfaces (or lots of buttons)‏ IPhone & Mac touchpad HP TouchSmart Tin Foil & an Arduino Force transducers, the Wii Fit Teaching new dogs old tricks

Computer vision, mostly

The Wiimote 100hz camera

Capacitive surfaces (or lots of buttons)‏

IPhone & Mac touchpad

HP TouchSmart

Tin Foil & an Arduino

Force transducers, the Wii Fit

Teaching new dogs old tricks

Regions of interest At 30 fps or faster: Find a finger smudge (blobs)‏ Dots of light Triangulate based on closest data Color keying LEDs as two-way Use IR for spectrum separation Isolate features with laser Map in 3D using Lidar

At 30 fps or faster:

Find a finger smudge (blobs)‏

Dots of light

Triangulate based on closest data

Color keying

LEDs as two-way

Use IR for spectrum separation

Isolate features with laser

Map in 3D using Lidar

Better Software & Hardware DARPA Challenge Intel OpenCV Multiprocessor Machines Home clusters of machines GPU processors on videocards Better web cams Cheaper, more widely available materials Easier access to information

DARPA Challenge

Intel OpenCV

Multiprocessor Machines

Home clusters of machines

GPU processors on videocards

Better web cams

Cheaper, more widely available materials

Easier access to information

Fresh Ways to Interface with Computers It is not a panacea, but at least everyone is mostly invited.

It is not a panacea, but at least everyone is mostly invited.

Leave it to the big boys? IPhone's And knock-offs Microsoft Surface & Windows 7 & Tablets Microsoft Research HP TouchSmart Perceptive Pixel NUI Group step 1. cut a hole in a box

IPhone's

And knock-offs

Microsoft Surface & Windows 7 & Tablets

Microsoft Research

HP TouchSmart

Perceptive Pixel

NUI Group

step 1. cut a hole in a box

Gesturing Combos! Changes over time Inferring about the placement of input in orientation to the real world Surface Wii fit You're holding a nail near tinfoil

Combos!

Changes over time

Inferring about the placement of input in orientation to the real world

Surface

Wii fit

You're holding a nail near tinfoil

Simple! Or at least, reasonably simple given the complexity of what is going on. Small blocks of code in repition Again, faster computers

Or at least, reasonably simple given the complexity of what is going on.

Small blocks of code in repition

Again, faster computers

Web Cam in a Box Step #2 Put your junk in that box

Step #2

Put your junk in that box

Double true! Stackable cubes Large walls Small hobby projects Multiple cameras GPUs Networked machines Store Windows, floors , buildings

Stackable cubes

Large walls

Small hobby projects

Multiple cameras

GPUs

Networked machines

Store Windows, floors , buildings

Mostly flash (needs a lot of help from c)‏ But also: Python Processing Max VVVV Pure Data .NET Firefox / Javascript (Lily)‏

(needs a lot of help from c)‏

But also:

Python

Processing

Max

VVVV

Pure Data

.NET

Firefox / Javascript (Lily)‏

Rethinking a lot of things Gesture navigation User Interface Feedback

Gesture navigation

User Interface

Feedback

Guerrilla Arm

All kinds of things I tell ya! Sphere Gesturing with your hands above the home row Or just using your hands Or moving your head, or eyes, or thoughts This is an extra bullet point. Ha!

Sphere

Gesturing with your hands above the home row

Or just using your hands

Or moving your head, or eyes, or thoughts

This is an extra bullet point. Ha!

Meanwhile, back at the bat cave... Home clustering Semistructured web and public data HDTV's, projectors, err... iphones Social connectivity Except in the workplace where we'd generally like everyone to politely sod off Advanced modeling, free software for fluid dynamics and physics simulation

Home clustering

Semistructured web and public data

HDTV's, projectors, err... iphones

Social connectivity

Except in the workplace where we'd generally like everyone to politely sod off

Advanced modeling, free software for fluid dynamics and physics simulation

Lack of standards? lack of patent free ideas, maybe TUIO / OSC for ROI or Fiducials

lack of patent free ideas, maybe

TUIO / OSC for ROI or Fiducials

Romper room I see: NUI Group Open Frameworks Processing Apple Developer forums IPhone jailbreakers Multi-pointer X-Windows, Compiz, and the whole open-source world.

I see:

NUI Group

Open Frameworks

Processing

Apple Developer forums

IPhone jailbreakers

Multi-pointer X-Windows, Compiz, and the whole open-source world.

Feedback loop Proliferation? Adoption? Social implications This will all spawn something else Everybody's coming to the net Voluntary mass hallucination (Gibson)‏

Proliferation? Adoption?

Social implications

This will all spawn something else

Everybody's coming to the net

Voluntary mass hallucination (Gibson)‏

Come out and play! If yer under eighteen ya won't be doin any ty-ee-eye-ee-ime

If yer under eighteen ya won't be doin any ty-ee-eye-ee-ime

There's a place for us! More people playing! == better software Not as hard as it sounds, just steal code ;p Who's getting laid anyway? It is amazing what you can get done if you don't care who gets the credit. (jets ftw!)‏

More people playing!

== better software

Not as hard as it sounds, just steal code ;p

Who's getting laid anyway?

It is amazing what you can get done if you don't care who gets the credit. (jets ftw!)‏