Midas: A Declarative Multi-Touch Interaction Framework

4,825
-1

Published on

Presentation given at TEI 2011, 5th International Conference on Tangible, Embedded and Embodied Interaction, Madeira, Portugal, January 2011

ABSTRACT: Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple’s iPhone or Microsoft’s Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hardcoded
procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.

Published in: Science
2 Comments
1 Like
Statistics
Notes
  • The Midas Multi-Touch Interaction Framework will be released to the public within a few months. We are currently replacing proprietary code which we cannot (re)distribute.

    The name ’Midas Multi-Touch’ originates from the greek story about King Midas’ ability to turn everything he touched into gold. :)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • JMidas seems to have a very interesting approach by encapsulating the gesture functionality into the core layer and below. Especially the rule-based approach seems to me very promising. But I couldn't find a way to test this framework; no download exists.

    Also, the name Midas is used for other projects. There is a MIDAS 2 CLIENT FOR JAVA and X-MIDAS for the defence industry exists. Even JMidas exists: JMidas XBRL is used to extract information from XML-base SEC reports. Of course, one could say that there are also many Gilberts all over the World. Why should there be not several JMidas products? :->
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
4,825
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
60
Comments
2
Likes
1
Embeds 0
No embeds

No notes for slide

Midas: A Declarative Multi-Touch Interaction Framework

  1. 1. A Declarative Multi-Touch MIDAS Interaction Framework CHRISTOPHE SCHOLLIERS LODE HOSTE BEAT SIGNER WOLFGANG DE MEUTERVRIJE UNIVERSITEIT BRUSSEL #MidasFW
  2. 2. MULTI-TOUCH IS EVERYWHERE!
  3. 3. HARDWARE CAPABILITIES VSGESTURE RECOGNITION SOFTWARE Pinch St ret c h D ra g Rot a te Fl i c k
  4. 4. WHAT’S THE FUNDAMENTAL PROBLEM? add 0 0,633750 0,293750 update 0 0,565000 0,335417 update 1 0,251250 0,620833 update 1 0,166250 0,343750 update 0 0,635000 0,293750 update 1 0,387500 0,450000 update 0 0,656250 0,666667 update 0 0,728750 0,306250 update 0 0,633750 0,293750 update 0 0,560000 0,345833 update 1 0,242500 0,604167 update 1 0,172500 0,331250 update 0 0,633750 0,295833 update 1 0,390000 0,462500 update 0 0,663750 0,662500 update 0 0,725000 0,300000 update 0 0,633750 0,297917 update 0 0,558750 0,352083 update 1 0,237500 0,591667 update 1 0,177500 0,318750 update 0 0,632500 0,300000 update 1 0,391250 0,479167 update 0 0,667500 0,658333 update 0 0,723750 0,295833 update 0 0,631250 0,300000 update 0 0,556250 0,358333 update 1 0,232500 0,583333 update 1 0,181250 0,314583 update 0 0,630000 0,300000 update 1 0,391250 0,489583 update 0 0,672500 0,656250 update 0 0,720000 0,291667 update 0 0,628750 0,300000 update 0 0,553750 0,366667 update 1 0,215000 0,558333 update 1 0,185000 0,310417 update 0 0,627500 0,300000 update 1 0,392500 0,502083 update 0 0,691250 0,637500 update 0 0,718750 0,289583 add 1 0,246250 0,289583 update 0 0,550000 0,377083 update 1 0,206250 0,543750 update 1 0,186250 0,304167 update 1 0,247500 0,289583 update 1 0,392500 0,516667 update 0 0,702500 0,612500 update 0 0,716250 0,287500 update 1 0,248750 0,291667 update 0 0,548750 0,385417 update 1 0,187500 0,510417 update 1 0,190000 0,302083 update 1 0,250000 0,291667 update 1 0,392500 0,531250 update 0 0,711250 0,589583 update 0 0,713750 0,281250 update 1 0,252500 0,291667 update 0 0,546250 0,395833 update 1 0,167500 0,485417 update 1 0,192500 0,300000 update 1 0,256250 0,291667 update 1 0,390000 0,543750 update 0 0,716250 0,568750 update 0 0,708750 0,277083 update 1 0,258750 0,291667 update 0 0,543750 0,408333 update 1 0,161250 0,477083 update 1 0,198750 0,293750 update 1 0,263750 0,291667 update 1 0,388750 0,556250 update 0 0,720000 0,556250 update 0 0,702500 0,270833 update 0 0,626250 0,300000 update 0 0,542500 0,422917 update 1 0,152500 0,466667 update 1 0,205000 0,289583 update 1 0,276250 0,291667 update 1 0,386250 0,566667 update 0 0,725000 0,539583 update 0 0,692500 0,266667 update 0 0,625000 0,300000 update 0 0,543750 0,429167 update 1 0,146250 0,460417 update 1 0,208750 0,289583 update 1 0,285000 0,295833 update 1 0,385000 0,577083 update 0 0,727500 0,527083 update 0 0,688750 0,264583 update 0 0,622500 0,300000 update 0 0,543750 0,441667 update 1 0,143750 0,456250 update 1 0,212500 0,287500 update 1 0,293750 0,300000 update 1 0,378750 0,595833 update 0 0,728750 0,510417 update 0 0,685000 0,264583 update 0 0,621250 0,302083 update 0 0,543750 0,450000 update 1 0,138750 0,447917 update 1 0,216250 0,285417 update 1 0,300000 0,304167 update 1 0,367500 0,633333 update 0 0,731250 0,502083 update 0 0,678750 0,256250 update 1 0,310000 0,310417 update 0 0,547500 0,475000 update 1 0,136250 0,445833 update 1 0,221250 0,283333 update 0 0,618750 0,302083 update 1 0,348750 0,660417 update 0 0,733750 0,485417 update 0 0,675000 0,252083 update 1 0,318750 0,316667 update 0 0,551250 0,504167 update 1 0,133750 0,441667 update 1 0,227500 0,283333 update 0 0,616250 0,302083 update 1 0,341250 0,666667 update 0 0,736250 0,472917 update 0 0,671250 0,250000 update 1 0,325000 0,320833 update 0 0,553750 0,514583 update 1 0,132500 0,439583 update 1 0,231250 0,281250 update 0 0,613750 0,302083 update 1 0,336250 0,675000 update 0 0,740000 0,458333 update 0 0,667500 0,250000 update 1 0,330000 0,322917 update 0 0,557500 0,527083 update 1 0,131250 0,435417 update 1 0,233750 0,281250 update 0 0,611250 0,302083 update 1 0,331250 0,679167 update 0 0,741250 0,447917 update 0 0,662500 0,247917 update 1 0,335000 0,325000 update 0 0,561250 0,541667 update 0 0,741250 0,439583 update 1 0,238750 0,279167 update 0 0,610000 0,302083 update 1 0,326250 0,683333 update 1 0,130000 0,431250 update 0 0,658750 0,247917 update 1 0,338750 0,329167 update 0 0,565000 0,552083 update 0 0,742500 0,422917 update 1 0,245000 0,279167 update 0 0,607500 0,304167 update 1 0,321250 0,685417 update 1 0,130000 0,429167 update 0 0,655000 0,245833 update 1 0,341250 0,333333 update 0 0,570000 0,572917 update 0 0,742500 0,416667 update 1 0,248750 0,277083 update 0 0,605000 0,304167 update 1 0,316250 0,685417 update 1 0,130000 0,427083 update 0 0,651250 0,245833 update 1 0,345000 0,335417 update 0 0,575000 0,583333 update 0 0,743750 0,402083 update 1 0,253750 0,277083 update 0 0,602500 0,304167 update 1 0,311250 0,685417 update 1 0,130000 0,425000 update 0 0,646250 0,250000 update 1 0,347500 0,337500 update 0 0,581250 0,595833 update 0 0,745000 0,395833 update 1 0,258750 0,275000 update 0 0,600000 0,304167 update 1 0,307500 0,683333 update 1 0,131250 0,422917 update 0 0,642500 0,250000 update 1 0,351250 0,341667 update 0 0,590000 0,612500 update 0 0,745000 0,387500 update 1 0,263750 0,272917 update 0 0,596250 0,304167 update 1 0,302500 0,681250 update 1 0,132500 0,418750 update 0 0,638750 0,254167 update 1 0,355000 0,347917 update 0 0,595000 0,618750 update 0 0,745000 0,383333 update 1 0,267500 0,270833 update 0 0,593750 0,306250 update 1 0,296250 0,679167 update 1 0,133750 0,416667 update 0 0,635000 0,256250 update 1 0,358750 0,352083 update 0 0,606250 0,633333 update 0 0,743750 0,377083 update 1 0,271250 0,270833 update 0 0,590000 0,308333 update 1 0,290000 0,675000 update 1 0,135000 0,412500 update 0 0,631250 0,258333 update 1 0,361250 0,358333 update 0 0,612500 0,639583 update 0 0,743750 0,360417 update 1 0,278750 0,270833 update 0 0,586250 0,312500 update 1 0,281250 0,664583 update 1 0,136250 0,408333 update 0 0,625000 0,262500 update 1 0,365000 0,368750 update 0 0,622500 0,654167 update 0 0,741250 0,354167 update 1 0,282500 0,270833 update 0 0,583750 0,314583 update 1 0,272500 0,656250 update 1 0,141250 0,400000 update 0 0,622500 0,262500 update 1 0,368750 0,379167 update 0 0,631250 0,660417 update 0 0,738750 0,337500 update 1 0,286250 0,270833 update 0 0,580000 0,316667 update 1 0,268750 0,647917 update 1 0,145000 0,389583 update 0 0,621250 0,264583 update 1 0,372500 0,389583 update 0 0,636250 0,662500 update 0 0,738750 0,333333 update 1 0,288750 0,268750 update 0 0,577500 0,318750 update 1 0,262500 0,639583 update 1 0,147500 0,383333 update 0 0,620000 0,266667 update 1 0,378750 0,410417 update 0 0,643750 0,664583 update 0 0,735000 0,325000 update 1 0,291250 0,268750 update 0 0,571250 0,329167 update 1 0,257500 0,629167 update 1 0,151250 0,372917 update 0 0,618750 0,268750 update 1 0,382500 0,420833 update 0 0,647500 0,664583 update 0 0,731250 0,316667 update 1 0,292500 0,268750 update 0 0,568750 0,333333 update 1 0,255000 0,627083 update 1 0,157500 0,360417 update 0 0,617500 0,268750 update 1 0,383750 0,435417 update 0 0,653750 0,666667 update 0 0,728750 0,308333 update 1 0,293750 0,268750
  5. 5. WHAT’S THE FUNDAMENTAL PROBLEM? Sequential code for the processing of concurrent events Find spatial and temporal relations in an immense pool of x/y coordinates  ~25 events / second / finger Approximative detection GUI-Event correlation Modularisation and composition  implementing gestures is hard  combining gestures is even harder
  6. 6. THE MIDAS APPROACH SELECT * FROM Audience WHERE interest > 0Facts Fact Base Rules
  7. 7. MIDAS ARCHITECTURE Infrastructure layer Midas  Transform low level Application Layer data into facts GUI Shadows Model Core layer  temporal and spatial Core Layer operators Fact Inference Rule  continuous queries via Base Engine Base Rete algorithm Infrastructure Layer Application layer Hardware * Bridge Translator  regular program augmented with rules
  8. 8. MIDAS USES(defrule PrintCursor ?c <- (Cursor)=> (printout t ‘‘A cursor is moving at: ’’ ?c.x ‘‘,’’ ?c.y))
  9. 9. MIDAS FILTERS + (defrule Tap ?appear <- (Cursor (finger ?f) (state ?*APPEAR*)) ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*)) (test (tShortlyAfter ?disappear ?appear)) => (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y))only a few lines of declarative code
  10. 10. MIDAS FINDS +(defrule Tap ?appear <- (Cursor (finger ?f) (state ?*APPEAR*)) ?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*)) (test (tShortlyAfter ?disappear ?appear)) (test (sVeryNear ?disappear ?appear))=> (printout t “A single tap detected at: “ ?appear.x “,” ?appear.y) (assert (Tap (x ?appear.x) (y ?appear.y) (on ?appear.on)) (retract ?appear ?disappear))
  11. 11. MIDAS ENCOURAGES(defrule DoubleTap ?tap1 <- (Tap) ?tap2 <- (Tap) (test (tShortlyAfter ?tap2 ?tap1)) (test (sVeryNear ?tap2 ?tap1))=> (printout t “A double tap detected at: “ ?tap1.x“,” ?tap2.y) (retract ?tap1 ?tap2))small and reusable building blocks
  12. 12. MIDAS ENCOURAGES(defrule FixedUpUp ?fixed <- (Fixed) ?up1 <- (Up) ?up2 <- (Up) (test (tEqual3 ?fixed ?up1 ?up2)) (test (sLeftOf ?fixed ?up1)) (test (sLeftOf ?up1 ?up2))=> (printout t “Complex scroll detected!”) (retract ?fixed ?up1 ?up2))
  13. 13. MIDAS CORRELATES add 0 0,633750 0,293750 update 0 0,565000 0,335417 update 0 0,635000 0,293750 update 1 0,387500 0,450000(defrule Gui update 0 0,633750 0,293750 update 0 0,560000 0,345833 update 0 0,633750 0,295833 update 1 0,390000 0,462500 update 0 0,633750 0,297917 update 0 0,558750 0,352083 update 0 0,632500 0,300000 update 1 0,391250 0,479167 ?tap<- (Tap) update 0 0,631250 0,300000 update 0 0,630000 0,300000 update 0 0,628750 0,300000 update 0 0,627500 0,300000 update 0 0,556250 0,358333 update 1 0,391250 0,489583 update 0 0,553750 0,366667 update 1 0,392500 0,502083 ?planet <- (Planet) add 1 0,246250 0,289583 update 1 0,247500 0,289583 update 1 0,248750 0,291667 update 0 0,550000 0,377083 update 1 0,392500 0,516667 update 0 0,548750 0,385417 (test (sInside ?tap ?planet)) update 1 0,250000 0,291667 update 1 0,392500 0,531250 update 1 0,252500 0,291667 update 0 0,546250 0,395833 update 1 0,256250 0,291667 update 1 0,390000 0,543750=> (printout t “Tap inside planet!”)) Fact Base Shadows (GUI elements)
  14. 14. MIDAS SUPPORTS Templates (defrule nextSlide ?goRight1 <- (FlickRight (tuioId ?*RH*)) ?goLeft <- (FlickLeft (tuioId ?*LH*)) ?goRight2 <- (FlickRight (tuioId ?*RH*)) (test (tMeets ?goRight1 ?goLeft ?goRight2)) => (call PowerPoint showNextSlide) (retract ?goRight1 ?goLeft ?goRight2))Implemented online gestures Kinect example
  15. 15. CONCLUSION Midas multi-touch interaction framework  Declarative description of gestures (fact base and rule engine)  extensibility and reusability of gestures  correlation of GUI elements and events via shadow facts  JMidas Java integration Ongoing work  integration of new devices: Kinect, Digital Pen & Paper, ...  extend Midas to support multimodal gestures Rapid prototyping of multi-touch gestures Midas Demo, Tuesday @ 15.00-18.30 -- ID: 251Research: http://goo.gl/OFpej - Youtube: http://goo.gl/MNDkT - Twitter: #MidasFW
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×