Midas: A Declarative Multi-Touch Interaction Framework

A Declarative Multi-Touch MIDAS Interaction Framework CHRISTOPHE SCHOLLIERS LODE HOSTE BEAT SIGNER WOLFGANG DE MEUTERVRIJE UNIVERSITEIT BRUSSEL #MidasFW

MULTI-TOUCH IS EVERYWHERE!

HARDWARE CAPABILITIES VSGESTURE RECOGNITION SOFTWARE Pinch St ret c h D ra g Rot a te Fl i c k

WHAT’S THE FUNDAMENTAL PROBLEM? add 0 0,633750 0,293750 update 0 0,565000 0,335417 update 1 0,251250 0,620833 update 1 0,166250 0,343750 update 0 0,635000 0,293750 update 1 0,387500 0,450000 update 0 0,656250 0,666667 update 0 0,728750 0,306250 update 0 0,633750 0,293750 update 0 0,560000 0,345833 update 1 0,242500 0,604167 update 1 0,172500 0,331250 update 0 0,633750 0,295833 update 1 0,390000 0,462500 update 0 0,663750 0,662500 update 0 0,725000 0,300000 update 0 0,633750 0,297917 update 0 0,558750 0,352083 update 1 0,237500 0,591667 update 1 0,177500 0,318750 update 0 0,632500 0,300000 update 1 0,391250 0,479167 update 0 0,667500 0,658333 update 0 0,723750 0,295833 update 0 0,631250 0,300000 update 0 0,556250 0,358333 update 1 0,232500 0,583333 update 1 0,181250 0,314583 update 0 0,630000 0,300000 update 1 0,391250 0,489583 update 0 0,672500 0,656250 update 0 0,720000 0,291667 update 0 0,628750 0,300000 update 0 0,553750 0,366667 update 1 0,215000 0,558333 update 1 0,185000 0,310417 update 0 0,627500 0,300000 update 1 0,392500 0,502083 update 0 0,691250 0,637500 update 0 0,718750 0,289583 add 1 0,246250 0,289583 update 0 0,550000 0,377083 update 1 0,206250 0,543750 update 1 0,186250 0,304167 update 1 0,247500 0,289583 update 1 0,392500 0,516667 update 0 0,702500 0,612500 update 0 0,716250 0,287500 update 1 0,248750 0,291667 update 0 0,548750 0,385417 update 1 0,187500 0,510417 update 1 0,190000 0,302083 update 1 0,250000 0,291667 update 1 0,392500 0,531250 update 0 0,711250 0,589583 update 0 0,713750 0,281250 update 1 0,252500 0,291667 update 0 0,546250 0,395833 update 1 0,167500 0,485417 update 1 0,192500 0,300000 update 1 0,256250 0,291667 update 1 0,390000 0,543750 update 0 0,716250 0,568750 update 0 0,708750 0,277083 update 1 0,258750 0,291667 update 0 0,543750 0,408333 update 1 0,161250 0,477083 update 1 0,198750 0,293750 update 1 0,263750 0,291667 update 1 0,388750 0,556250 update 0 0,720000 0,556250 update 0 0,702500 0,270833 update 0 0,626250 0,300000 update 0 0,542500 0,422917 update 1 0,152500 0,466667 update 1 0,205000 0,289583 update 1 0,276250 0,291667 update 1 0,386250 0,566667 update 0 0,725000 0,539583 update 0 0,692500 0,266667 update 0 0,625000 0,300000 update 0 0,543750 0,429167 update 1 0,146250 0,460417 update 1 0,208750 0,289583 update 1 0,285000 0,295833 update 1 0,385000 0,577083 update 0 0,727500 0,527083 update 0 0,688750 0,264583 update 0 0,622500 0,300000 update 0 0,543750 0,441667 update 1 0,143750 0,456250 update 1 0,212500 0,287500 update 1 0,293750 0,300000 update 1 0,378750 0,595833 update 0 0,728750 0,510417 update 0 0,685000 0,264583 update 0 0,621250 0,302083 update 0 0,543750 0,450000 update 1 0,138750 0,447917 update 1 0,216250 0,285417 update 1 0,300000 0,304167 update 1 0,367500 0,633333 update 0 0,731250 0,502083 update 0 0,678750 0,256250 update 1 0,310000 0,310417 update 0 0,547500 0,475000 update 1 0,136250 0,445833 update 1 0,221250 0,283333 update 0 0,618750 0,302083 update 1 0,348750 0,660417 update 0 0,733750 0,485417 update 0 0,675000 0,252083 update 1 0,318750 0,316667 update 0 0,551250 0,504167 update 1 0,133750 0,441667 update 1 0,227500 0,283333 update 0 0,616250 0,302083 update 1 0,341250 0,666667 update 0 0,736250 0,472917 update 0 0,671250 0,250000 update 1 0,325000 0,320833 update 0 0,553750 0,514583 update 1 0,132500 0,439583 update 1 0,231250 0,281250 update 0 0,613750 0,302083 update 1 0,336250 0,675000 update 0 0,740000 0,458333 update 0 0,667500 0,250000 update 1 0,330000 0,322917 update 0 0,557500 0,527083 update 1 0,131250 0,435417 update 1 0,233750 0,281250 update 0 0,611250 0,302083 update 1 0,331250 0,679167 update 0 0,741250 0,447917 update 0 0,662500 0,247917 update 1 0,335000 0,325000 update 0 0,561250 0,541667 update 0 0,741250 0,439583 update 1 0,238750 0,279167 update 0 0,610000 0,302083 update 1 0,326250 0,683333 update 1 0,130000 0,431250 update 0 0,658750 0,247917 update 1 0,338750 0,329167 update 0 0,565000 0,552083 update 0 0,742500 0,422917 update 1 0,245000 0,279167 update 0 0,607500 0,304167 update 1 0,321250 0,685417 update 1 0,130000 0,429167 update 0 0,655000 0,245833 update 1 0,341250 0,333333 update 0 0,570000 0,572917 update 0 0,742500 0,416667 update 1 0,248750 0,277083 update 0 0,605000 0,304167 update 1 0,316250 0,685417 update 1 0,130000 0,427083 update 0 0,651250 0,245833 update 1 0,345000 0,335417 update 0 0,575000 0,583333 update 0 0,743750 0,402083 update 1 0,253750 0,277083 update 0 0,602500 0,304167 update 1 0,311250 0,685417 update 1 0,130000 0,425000 update 0 0,646250 0,250000 update 1 0,347500 0,337500 update 0 0,581250 0,595833 update 0 0,745000 0,395833 update 1 0,258750 0,275000 update 0 0,600000 0,304167 update 1 0,307500 0,683333 update 1 0,131250 0,422917 update 0 0,642500 0,250000 update 1 0,351250 0,341667 update 0 0,590000 0,612500 update 0 0,745000 0,387500 update 1 0,263750 0,272917 update 0 0,596250 0,304167 update 1 0,302500 0,681250 update 1 0,132500 0,418750 update 0 0,638750 0,254167 update 1 0,355000 0,347917 update 0 0,595000 0,618750 update 0 0,745000 0,383333 update 1 0,267500 0,270833 update 0 0,593750 0,306250 update 1 0,296250 0,679167 update 1 0,133750 0,416667 update 0 0,635000 0,256250 update 1 0,358750 0,352083 update 0 0,606250 0,633333 update 0 0,743750 0,377083 update 1 0,271250 0,270833 update 0 0,590000 0,308333 update 1 0,290000 0,675000 update 1 0,135000 0,412500 update 0 0,631250 0,258333 update 1 0,361250 0,358333 update 0 0,612500 0,639583 update 0 0,743750 0,360417 update 1 0,278750 0,270833 update 0 0,586250 0,312500 update 1 0,281250 0,664583 update 1 0,136250 0,408333 update 0 0,625000 0,262500 update 1 0,365000 0,368750 update 0 0,622500 0,654167 update 0 0,741250 0,354167 update 1 0,282500 0,270833 update 0 0,583750 0,314583 update 1 0,272500 0,656250 update 1 0,141250 0,400000 update 0 0,622500 0,262500 update 1 0,368750 0,379167 update 0 0,631250 0,660417 update 0 0,738750 0,337500 update 1 0,286250 0,270833 update 0 0,580000 0,316667 update 1 0,268750 0,647917 update 1 0,145000 0,389583 update 0 0,621250 0,264583 update 1 0,372500 0,389583 update 0 0,636250 0,662500 update 0 0,738750 0,333333 update 1 0,288750 0,268750 update 0 0,577500 0,318750 update 1 0,262500 0,639583 update 1 0,147500 0,383333 update 0 0,620000 0,266667 update 1 0,378750 0,410417 update 0 0,643750 0,664583 update 0 0,735000 0,325000 update 1 0,291250 0,268750 update 0 0,571250 0,329167 update 1 0,257500 0,629167 update 1 0,151250 0,372917 update 0 0,618750 0,268750 update 1 0,382500 0,420833 update 0 0,647500 0,664583 update 0 0,731250 0,316667 update 1 0,292500 0,268750 update 0 0,568750 0,333333 update 1 0,255000 0,627083 update 1 0,157500 0,360417 update 0 0,617500 0,268750 update 1 0,383750 0,435417 update 0 0,653750 0,666667 update 0 0,728750 0,308333 update 1 0,293750 0,268750

WHAT’S THE FUNDAMENTAL PROBLEM? Sequential code for the processing of concurrent events Find spatial and temporal relations in an immense pool of x/y coordinates  ~25 events / second / finger Approximative detection GUI-Event correlation Modularisation and composition  implementing gestures is hard  combining gestures is even harder

THE MIDAS APPROACH SELECT * FROM Audience WHERE interest > 0Facts Fact Base Rules

MIDAS ARCHITECTURE Infrastructure layer Midas  Transform low level Application Layer data into facts GUI Shadows Model Core layer  temporal and spatial Core Layer operators Fact Inference Rule  continuous queries via Base Engine Base Rete algorithm Infrastructure Layer Application layer Hardware * Bridge Translator  regular program augmented with rules

MIDAS USES(defrule PrintCursor ?c <- (Cursor)=> (printout t ‘‘A cursor is moving at: ’’ ?c.x ‘‘,’’ ?c.y))

MIDAS FILTERS + (defrule Tap ?appear <- (Cursor (finger ?f) (state ?*APPEAR*)) ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*)) (test (tShortlyAfter ?disappear ?appear)) => (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y))only a few lines of declarative code

MIDAS FINDS +(defrule Tap ?appear <- (Cursor (finger ?f) (state ?*APPEAR*)) ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*)) (test (tShortlyAfter ?disappear ?appear)) (test (sVeryNear ?disappear ?appear))=> (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y) (assert (Tap (x ?appear.x) (y ?appear.y) (on ?appear.on)) (retract ?appear ?disappear))

MIDAS ENCOURAGES(defrule DoubleTap ?tap1 <- (Tap) ?tap2 <- (Tap) (test (tShortlyAfter ?tap2 ?tap1)) (test (sVeryNear ?tap2 ?tap1))=> (printout t “A double tap detected at: “ ?tap1.x“,” ?tap2.y) (retract ?tap1 ?tap2))small and reusable building blocks

MIDAS ENCOURAGES(defrule FixedUpUp ?fixed <- (Fixed) ?up1 <- (Up) ?up2 <- (Up) (test (tEqual3 ?fixed ?up1 ?up2)) (test (sLeftOf ?fixed ?up1)) (test (sLeftOf ?up1 ?up2))=> (printout t “Complex scroll detected!”) (retract ?fixed ?up1 ?up2))

MIDAS CORRELATES add 0 0,633750 0,293750 update 0 0,565000 0,335417 update 0 0,635000 0,293750 update 1 0,387500 0,450000(defrule Gui update 0 0,633750 0,293750 update 0 0,560000 0,345833 update 0 0,633750 0,295833 update 1 0,390000 0,462500 update 0 0,633750 0,297917 update 0 0,558750 0,352083 update 0 0,632500 0,300000 update 1 0,391250 0,479167 ?tap<- (Tap) update 0 0,631250 0,300000 update 0 0,630000 0,300000 update 0 0,628750 0,300000 update 0 0,627500 0,300000 update 0 0,556250 0,358333 update 1 0,391250 0,489583 update 0 0,553750 0,366667 update 1 0,392500 0,502083 ?planet <- (Planet) add 1 0,246250 0,289583 update 1 0,247500 0,289583 update 1 0,248750 0,291667 update 0 0,550000 0,377083 update 1 0,392500 0,516667 update 0 0,548750 0,385417 (test (sInside ?tap ?planet)) update 1 0,250000 0,291667 update 1 0,392500 0,531250 update 1 0,252500 0,291667 update 0 0,546250 0,395833 update 1 0,256250 0,291667 update 1 0,390000 0,543750=> (printout t “Tap inside planet!”)) Fact Base Shadows (GUI elements)

MIDAS SUPPORTS Templates (defrule nextSlide ?goRight1 <- (FlickRight (tuioId ?*RH*)) ?goLeft <- (FlickLeft (tuioId ?*LH*)) ?goRight2 <- (FlickRight (tuioId ?*RH*)) (test (tMeets ?goRight1 ?goLeft ?goRight2)) => (call PowerPoint showNextSlide) (retract ?goRight1 ?goLeft ?goRight2))Implemented online gestures Kinect example

CONCLUSION Midas multi-touch interaction framework  Declarative description of gestures (fact base and rule engine)  extensibility and reusability of gestures  correlation of GUI elements and events via shadow facts  JMidas Java integration Ongoing work  integration of new devices: Kinect, Digital Pen & Paper, ...  extend Midas to support multimodal gestures Rapid prototyping of multi-touch gestures Midas Demo, Tuesday @ 15.00-18.30 -- ID: 251Research: http://goo.gl/OFpej - Youtube: http://goo.gl/MNDkT - Twitter: #MidasFW

Midas: A Declarative Multi-Touch Interaction Framework

by Beat Signer

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 1,555

Statistics

Midas: A Declarative Multi-Touch Interaction Framework Presentation Transcript

http://soft.vub.ac.be	1553
http://translate.googleusercontent.com	2