QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion Controller
•Download as PPTX, PDF•
0 likes•3 views
Presentation of a full paper @ EICS 2022 The paper is available at https://dl.acm.org/doi/10.1145/3532211
Recommended
Recommended
More Related Content
Similar to QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion Controller
Similar to QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion Controller (20)
Recently uploaded
Recently uploaded (20)
QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion Controller
- 1. QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion Controller Arthur Sluÿters, Mehdi Ousmer, Paolo Roselli, and Jean Vanderdonckt 1
- 2. 2
- 3. What if, as a developer, you want to create a gesture-based application? 3
- 4. Select a sensor Design a gesture set Implement gesture recognition 4
- 7. Implement gesture recognition: Drivers & Libraries LMC driver LMC API Dialog IF gesture = circle THEN action1() IF gesture = swipe THEN action2() Functional adapter Application core Using built-in gesture recognition 7
- 8. Implement gesture recognition: Drivers & Libraries LMC driver LMC API Dialog IF gesture = circle THEN action1() IF gesture = swipe THEN action2() Functional adapter Application core Using built-in gesture recognition Easy, but… • Usually very limited • No always available 8
- 9. Implement gesture recognition: Drivers & Libraries LMC driver LMC API Dialog IF gesture = circle THEN action1() IF gesture = swipe THEN action2() Functional adapter Application core Global interaction technique Using custom gestures 9
- 10. Implement gesture recognition: Drivers & Libraries LMC driver LMC API Dialog IF gesture = circle THEN action1() IF gesture = swipe THEN action2() Functional adapter Application core Global interaction technique Using custom gestures Time-consuming! • Search & compare • Understand • Implement • Test 10
- 11. Implement gesture recognition: Drivers & Libraries LMC driver LMC API Dialog IF gesture = circle THEN action1() IF gesture = swipe THEN action2() Functional adapter Application core Global interaction technique What if we add gestures? 11
- 12. We designed QuantumLeap to simplify the work of developpers 13
- 13. What is QuantumLeap? Drivers & Libraries LMC driver LMC API Dialog IF gesture = circle THEN action1() IF gesture = swipe THEN action2() Functional adapter Application core Global interaction technique 14
- 14. What is QuantumLeap? Drivers & Libraries LMC driver LMC API Dialog IF gesture = circle THEN action1() IF gesture = swipe THEN action2() Functional adapter Application core Global interaction technique QuantumLeap backend 15
- 15. What is QuantumLeap? Drivers & Libraries LMC driver LMC API Dialog IF gesture = circle THEN action1() IF gesture = swipe THEN action2() Functional adapter Application core Global interaction technique QuantumLeap backend Application code & QuantumLeap API 16
- 16. In the rest of the presentation: 1. Architecture (backend, UI, API) 2. Applications 3. Evaluation 4. Conclusion 17
- 18. QL architecture • Dataflow with 8 types of modules • Configuration UI • API 19
- 19. Sensor interface Retrieve frames from the sensor and convert them into a standardized format 20
- 20. Filter Reduce user tremors and/or noise 21
- 21. Static recognizer Recognize static gestures (e.g., hand poses) at each frame 22
- 22. Analyzer Extract additional data from the sensor frames for use in an application 23
- 23. Segmenter & Dynamic recognizer • Identify intentional dynamic gestures from the stream of data • Recognize dynamic gestures (e.g., left swipe) from segments 26
- 24. Modules implementations For each type of module, implementations based on the literature + custom modules : • Recognizers: Jackknife, 3 cent,… • Filters: 1 euro filter, moving average,… • … 27
- 25. QuantumLeap API Interface between an application and QuantumLeap • connect([address]) • registerGestures(type, names) • addListener(eventName, listener) • … 28
- 26. Configuration UI • Built with ReactJS • Overview of the dataflow • Configuration of the modules
- 27. Configuration UI • Built with ReactJS • Dataflow overview • Module configuration Dataflow overview & controls 30
- 28. Configuration UI • Built with ReactJS • Dataflow overview • Module configuration Module configuration (1) 31
- 29. Configuration UI • Built with ReactJS • Dataflow overview • Module configuration Module configuration (2) 32
- 30. Applications
- 31. DICOM Viewer Manipulation of DICOM images by surgeons • 14 possible actions https://cdn.technologyadvice.com/wp-content/uploads/2015/05/dicom-viewers.jpg 34
- 32. 35
- 33. Large User Interface Manipulation of multimedia content: • Photos • Videos • Documents • Maps 37
- 34. Large User Interface Collaboration with MIT: • They provide the functional core of the app • We implement gesture recognition in 4 steps: 1. Gesture set design 2. Evaluation of recognizers 3. Configuration of QuantumLeap 4. Integration of gestures 38
- 35. 39
- 36. 40
- 37. Evaluation
- 38. Procedure • 7 users • Two tasks: • Configuration • Development (integration of gestures into a small application) 44
- 39. Procedure Two tasks: • Configuration • Development (integration of gestures into a small application) 45
- 40. Before During After Data Collection • Demographic data (age, experience,…) • Task completion rate • Task completion time • Video recording • SUS (usability) • NASA-TLX (workload) • Open-ended questions 46
- 41. Results The configuration process is still complex • Lack of guidance • Many modules to configure 47
- 42. Conclusion
- 43. Conclusion • A framework for the development of gesture-based applications • Modular dataflow • Simple API • Configuration UI • 2 applications • An evaluation with 7 users 49
- 44. Limitations • Lack of guidance in the configuration • Rigid architecture • Only instantiated to the LMC 50
- 45. Future works • Simplifying the configuration process • Guidance • Suggestions • Integration of new modules • Sensors (radars,…) • Gesture segmentation & recognition techniques • … • Evaluation • Bigger development projects • Comparisons with/without QuantumLeap 51
- 46. Thank you for your attention!
Editor's Notes
- Despite the tremendous progress made for recognizing gestures acquired by various devices, such as the Leap Motion Controller, developing a gestural user interface based on such devices still induces a significant programming and software engineering effort before obtaining a running interactive application As an example, when you look up algorithms for gesture recognition on google scholar, you get 100s of thousands of results, which can be intimidating for developers that are less experienced with gesture recognition in general.
- Despite the tremendous progress made for recognizing gestures acquired by various devices, such as the Leap Motion Controller, developing a gestural user interface based on such devices still induces a significant programming and software engineering effort before obtaining a running interactive application
- Very few / no gestures
- Very few / no gestures
- Need to find/create a gesture recognizer. If existing recognizer: source code not always available, difficult to compare,…
- Need to find/create a gesture recognizer. If existing recognizer: source code not always available, difficult to compare,…
- Changing the sensor/gesture recognizer requires a lot of changes to the application Sensors have different APIs, different data structures Gesture recognizer algorithms have different interfaces
- Pour vous donner un peu de contexte, on va analyser une situation où un développeur souhaite créer une application controlée par de gestes
- In the rest of the presentation, I’ll give more details about the architecture of QuantumLeap Then, I’ll present two different applications built using QuantumLeap Discuss the limitations of QuantumLeap Conclude and discuss future works
- Pour vous donner un peu de contexte, on va analyser une situation où un développeur souhaite créer une application controlée par de gestes