The Tactus Project
Novel Multi-Touch Task Management


Ryan Pavlik
The Project Definition
The Project Definition

 Overall objective: Do something neat and new.
The Project Definition

 Overall objective: Do something neat and new.
 Build hardware to enable multi-touch (MT) input to ...
The Project Definition

 Overall objective: Do something neat and new.
 Build hardware to enable multi-touch (MT) input to ...
The Project Definition

 Overall objective: Do something neat and new.
 Build hardware to enable multi-touch (MT) input to ...
Getting it done...
Getting it done...
Lots of sawdust, plenty of error messages
Hardware Deliverables
Hardware Deliverables

Multi-touch displays capable of also supporting a
“standard” workstation
Hardware Deliverables

Multi-touch displays capable of also supporting a
“standard” workstation
  “MT Mini” temporary test...
Hardware Deliverables

Multi-touch displays capable of also supporting a
“standard” workstation
  “MT Mini” temporary test...
Hardware Deliverables

Multi-touch displays capable of also supporting a
“standard” workstation
  “MT Mini” temporary test...
Software Deliverables
Software Deliverables
 rp-mt-scripts - Multi-touch software system
Software Deliverables
 rp-mt-scripts - Multi-touch software system
   Linux-based (Ubuntu 9.04) system for automatic
   se...
Software Deliverables
 rp-mt-scripts - Multi-touch software system
   Linux-based (Ubuntu 9.04) system for automatic
   se...
Software Deliverables
 rp-mt-scripts - Multi-touch software system
   Linux-based (Ubuntu 9.04) system for automatic
   se...
Software Deliverables
 rp-mt-scripts - Multi-touch software system
   Linux-based (Ubuntu 9.04) system for automatic
   se...
Software Deliverables
 rp-mt-scripts - Multi-touch software system
   Linux-based (Ubuntu 9.04) system for automatic
   se...
What’s up with multi-touch?
What’s up with multi-touch?

Keyboard
What’s up with multi-touch?

Keyboard
Mouse
What’s up with multi-touch?

Keyboard
Mouse
Touchscreen
What’s up with multi-touch?

Keyboard
Mouse
Touchscreen
Multi-touch tablets/screens
Optical multi-touch sensing
Optical multi-touch sensing
Blob tracking through computer vision
Infrared
sensing
Infrared
sensing
Permits multi-touch
displays, not just
tablets
Table design principles
Table design principles
1001 ways to make a multi-touch screen
Revision 0: “MT Mini”
Revision 0: “MT Mini”
Software test platform, works by “front DI”
Rear diffused illumination
Rear diffused illumination
“Projected rear DI” chosen for first hardware
revision
Laser light plane (LLP)
Laser light plane (LLP)
LCD LLP: Slow shipping is disappointing...
Software
Software
Doing work with multi-touch, without throwing
out your keyboard or mouse
Navigation display
Navigation display


 Multi-touch surface as secondary display
Navigation display


 Multi-touch surface as secondary display
 Enhance interaction with existing software
Navigation display


 Multi-touch surface as secondary display
 Enhance interaction with existing software
 Requires a cus...
Normal
                                            Display



Physical        Multi-touch
                Nav. Display
   ...
Normal
                                                          Display



Physical                      Multi-touch
    ...
Navigation display
Navigation display

 Replaces your taskbar,
 dock, etc.
Navigation display

 Replaces your taskbar,
 dock, etc.
 Horizontal multi-touch
 screen
Navigation display

 Replaces your taskbar,
 dock, etc.
 Horizontal multi-touch
 screen
 Supports a variety of
 gestures
Minimize all current windows
Minimize all current windows
Drag from “current display” bar into “pile canvas”
Minimize all current windows
Drag from “current display” bar into “pile canvas”
Restore a “window pile”
Restore a “window pile”
Drag from “pile canvas” into “current display” bar
Restore a “window pile”
Drag from “pile canvas” into “current display” bar
Move window to new “pile”
Move window to new “pile”
Drag a single window between piles, possibly
expanding first if you find it helpful
Move window to new “pile”
Drag a single window between piles, possibly
expanding first if you find it helpful
on ...
  t en si
Ex




Spread out a “pile”
on ...
  t en si
Ex




Spread out a “pile”
Drag a single window between piles, possibly
expanding first if you find it help...
on ...
  t en si
Ex




Spread out a “pile”
Drag a single window between piles, possibly
expanding first if you find it help...
Demo time
Demo time
Please feel free to ask questions, etc...
How it got done
How it got done
What I used, what I learned
Process
Process
Research/Comparison
Process
Research/Comparison
Interaction Design
Process
Research/Comparison
Interaction Design
Parallel Platform Development:
Process
Research/Comparison
Interaction Design
Parallel Platform Development:
  Hardware Construction
Process
Research/Comparison
Interaction Design
Parallel Platform Development:
  Hardware Construction
  Software platforms...
Process
Research/Comparison
Interaction Design
Parallel Platform Development:
  Hardware Construction
  Software platforms...
Strategies
Strategies
 Internet resources: nuigroup.com forums, etc
Strategies
 Internet resources: nuigroup.com forums, etc
   DIY MT is popular right now
Strategies
 Internet resources: nuigroup.com forums, etc
   DIY MT is popular right now
 Use existing components where pos...
Strategies
 Internet resources: nuigroup.com forums, etc
   DIY MT is popular right now
 Use existing components where pos...
Strategies
 Internet resources: nuigroup.com forums, etc
   DIY MT is popular right now
 Use existing components where pos...
Knowledge
Knowledge
Object-oriented programming - CS core
Knowledge
Object-oriented programming - CS core
  Lots of polymorphism and multiple inheritance
Knowledge
Object-oriented programming - CS core
  Lots of polymorphism and multiple inheritance
Linux skills, shell script...
Knowledge
Object-oriented programming - CS core
  Lots of polymorphism and multiple inheritance
Linux skills, shell script...
Knowledge
Object-oriented programming - CS core
  Lots of polymorphism and multiple inheritance
Linux skills, shell script...
Knowledge
Object-oriented programming - CS core
  Lots of polymorphism and multiple inheritance
Linux skills, shell script...
Advice
Advice

Hardware always takes time
Advice

Hardware always takes time
Make bold decisions - be wary of the paradox of
choice
Advice

Hardware always takes time
Make bold decisions - be wary of the paradox of
choice
Use version control software - g...
Advice

Hardware always takes time
Make bold decisions - be wary of the paradox of
choice
Use version control software - g...
Advice

Hardware always takes time
Make bold decisions - be wary of the paradox of
choice
Use version control software - g...
Difficulties and scope
control...
Difficulties and scope
control...

 Pile interaction there but disabled for demo
Difficulties and scope
control...

 Pile interaction there but disabled for demo
 Auto-clustering algorithms designed but n...
Difficulties and scope
control...

 Pile interaction there but disabled for demo
 Auto-clustering algorithms designed but n...
Difficulties and scope
control...

 Pile interaction there but disabled for demo
 Auto-clustering algorithms designed but n...
Extensions
Extensions


Implement other gestures planned and more
Extensions


Implement other gestures planned and more
User interaction/efficiency studies
Extensions


Implement other gestures planned and more
User interaction/efficiency studies
Add launcher functionality
Acknowledgements
Acknowledgements
It takes a college, a city, and an Internet to
create this technology...
Acknowledgements:
Presentation
Acknowledgements:
Presentation
3d images of multi-touch techniques: Nolan, from PeauProductions.org
and NUI Group Forums

...
Acknowledgements:
Project Support
Acknowledgements:
Project Support
SNC IT and Tech Support

Mr. John Friedrich, Division Manager at Midland Plastics Inc. -...
Acknowledgements -
Software
Acknowledgements -
Software
Ubuntu Linux and GNOME
Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
libwnck and python-gnome for window management
Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
libwnck and python-gnome for window management
setpwc for webcam...
Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
libwnck and python-gnome for window management
setpwc for webcam...
Acknowledgements -
Software
Ubuntu Linux and GNOME
Python
libwnck and python-gnome for window management
setpwc for webcam...
The Tactus Project
Hope you enjoyed this walkthrough!
The Tactus Project
Hope you enjoyed this walkthrough!
Multi-touch Interaction and Task Management - CS460 Spring 09 Capstone Project
Multi-touch Interaction and Task Management - CS460 Spring 09 Capstone Project
Upcoming SlideShare
Loading in …5
×

Multi-touch Interaction and Task Management - CS460 Spring 09 Capstone Project

2,154 views
2,084 views

Published on

This is the slide deck from my senior project presentation. More information is available at http://compsci.snc.edu/cs460/pavlra/

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,154
On SlideShare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
































































































































































































  • Multi-touch Interaction and Task Management - CS460 Spring 09 Capstone Project

    1. 1. The Tactus Project Novel Multi-Touch Task Management Ryan Pavlik
    2. 2. The Project Definition
    3. 3. The Project Definition Overall objective: Do something neat and new.
    4. 4. The Project Definition Overall objective: Do something neat and new. Build hardware to enable multi-touch (MT) input to a computer system.
    5. 5. The Project Definition Overall objective: Do something neat and new. Build hardware to enable multi-touch (MT) input to a computer system. Develop software to use this hardware in a novel way.
    6. 6. The Project Definition Overall objective: Do something neat and new. Build hardware to enable multi-touch (MT) input to a computer system. Develop software to use this hardware in a novel way. Refinement: Create task-management software to use multi-touch
    7. 7. Getting it done...
    8. 8. Getting it done... Lots of sawdust, plenty of error messages
    9. 9. Hardware Deliverables
    10. 10. Hardware Deliverables Multi-touch displays capable of also supporting a “standard” workstation
    11. 11. Hardware Deliverables Multi-touch displays capable of also supporting a “standard” workstation “MT Mini” temporary test device
    12. 12. Hardware Deliverables Multi-touch displays capable of also supporting a “standard” workstation “MT Mini” temporary test device Desk #1 - Projected Rear DI - complete
    13. 13. Hardware Deliverables Multi-touch displays capable of also supporting a “standard” workstation “MT Mini” temporary test device Desk #1 - Projected Rear DI - complete Desk #2 - LCD Laser Light Plane - lasers arrived on Monday...
    14. 14. Software Deliverables
    15. 15. Software Deliverables rp-mt-scripts - Multi-touch software system
    16. 16. Software Deliverables rp-mt-scripts - Multi-touch software system Linux-based (Ubuntu 9.04) system for automatic setup and management of multi-touch software
    17. 17. Software Deliverables rp-mt-scripts - Multi-touch software system Linux-based (Ubuntu 9.04) system for automatic setup and management of multi-touch software Now installing required software a one-step process
    18. 18. Software Deliverables rp-mt-scripts - Multi-touch software system Linux-based (Ubuntu 9.04) system for automatic setup and management of multi-touch software Now installing required software a one-step process tactus-navigator - “Nav display” task manager
    19. 19. Software Deliverables rp-mt-scripts - Multi-touch software system Linux-based (Ubuntu 9.04) system for automatic setup and management of multi-touch software Now installing required software a one-step process tactus-navigator - “Nav display” task manager Replacement for task bar
    20. 20. Software Deliverables rp-mt-scripts - Multi-touch software system Linux-based (Ubuntu 9.04) system for automatic setup and management of multi-touch software Now installing required software a one-step process tactus-navigator - “Nav display” task manager Replacement for task bar Places icons for current applications on MT display
    21. 21. What’s up with multi-touch?
    22. 22. What’s up with multi-touch? Keyboard
    23. 23. What’s up with multi-touch? Keyboard Mouse
    24. 24. What’s up with multi-touch? Keyboard Mouse Touchscreen
    25. 25. What’s up with multi-touch? Keyboard Mouse Touchscreen Multi-touch tablets/screens
    26. 26. Optical multi-touch sensing
    27. 27. Optical multi-touch sensing Blob tracking through computer vision
    28. 28. Infrared sensing
    29. 29. Infrared sensing Permits multi-touch displays, not just tablets
    30. 30. Table design principles
    31. 31. Table design principles 1001 ways to make a multi-touch screen
    32. 32. Revision 0: “MT Mini”
    33. 33. Revision 0: “MT Mini” Software test platform, works by “front DI”
    34. 34. Rear diffused illumination
    35. 35. Rear diffused illumination “Projected rear DI” chosen for first hardware revision
    36. 36. Laser light plane (LLP)
    37. 37. Laser light plane (LLP) LCD LLP: Slow shipping is disappointing...
    38. 38. Software
    39. 39. Software Doing work with multi-touch, without throwing out your keyboard or mouse
    40. 40. Navigation display
    41. 41. Navigation display Multi-touch surface as secondary display
    42. 42. Navigation display Multi-touch surface as secondary display Enhance interaction with existing software
    43. 43. Navigation display Multi-touch surface as secondary display Enhance interaction with existing software Requires a custom desk for both multi-touch I/O and normal I/O
    44. 44. Normal Display Physical Multi-touch Nav. Display Keyb. Layout LCD and Laser Light Plane LCD and light plane sandwiched in there Mouse IR camera Wheels to be installed for portability
    45. 45. Normal Display Physical Multi-touch Nav. Display Keyb. Layout LCD and Laser Light Plane LCD and light plane sandwiched in there “Normal” desk with Mouse IR camera the multitouch surface to the left of Wheels to be installed the keyboard for portability
    46. 46. Navigation display
    47. 47. Navigation display Replaces your taskbar, dock, etc.
    48. 48. Navigation display Replaces your taskbar, dock, etc. Horizontal multi-touch screen
    49. 49. Navigation display Replaces your taskbar, dock, etc. Horizontal multi-touch screen Supports a variety of gestures
    50. 50. Minimize all current windows
    51. 51. Minimize all current windows Drag from “current display” bar into “pile canvas”
    52. 52. Minimize all current windows Drag from “current display” bar into “pile canvas”
    53. 53. Restore a “window pile”
    54. 54. Restore a “window pile” Drag from “pile canvas” into “current display” bar
    55. 55. Restore a “window pile” Drag from “pile canvas” into “current display” bar
    56. 56. Move window to new “pile”
    57. 57. Move window to new “pile” Drag a single window between piles, possibly expanding first if you find it helpful
    58. 58. Move window to new “pile” Drag a single window between piles, possibly expanding first if you find it helpful
    59. 59. on ... t en si Ex Spread out a “pile”
    60. 60. on ... t en si Ex Spread out a “pile” Drag a single window between piles, possibly expanding first if you find it helpful
    61. 61. on ... t en si Ex Spread out a “pile” Drag a single window between piles, possibly expanding first if you find it helpful
    62. 62. Demo time
    63. 63. Demo time Please feel free to ask questions, etc...
    64. 64. How it got done
    65. 65. How it got done What I used, what I learned
    66. 66. Process
    67. 67. Process Research/Comparison
    68. 68. Process Research/Comparison Interaction Design
    69. 69. Process Research/Comparison Interaction Design Parallel Platform Development:
    70. 70. Process Research/Comparison Interaction Design Parallel Platform Development: Hardware Construction
    71. 71. Process Research/Comparison Interaction Design Parallel Platform Development: Hardware Construction Software platforms (task nav. with no interface, rp- mt-scripts)
    72. 72. Process Research/Comparison Interaction Design Parallel Platform Development: Hardware Construction Software platforms (task nav. with no interface, rp- mt-scripts) Integration
    73. 73. Strategies
    74. 74. Strategies Internet resources: nuigroup.com forums, etc
    75. 75. Strategies Internet resources: nuigroup.com forums, etc DIY MT is popular right now
    76. 76. Strategies Internet resources: nuigroup.com forums, etc DIY MT is popular right now Use existing components where possible
    77. 77. Strategies Internet resources: nuigroup.com forums, etc DIY MT is popular right now Use existing components where possible Avoid re-inventing the wheel when you can do just as well without doing so.
    78. 78. Strategies Internet resources: nuigroup.com forums, etc DIY MT is popular right now Use existing components where possible Avoid re-inventing the wheel when you can do just as well without doing so. Careful design consideration and comparison of alternatives
    79. 79. Knowledge
    80. 80. Knowledge Object-oriented programming - CS core
    81. 81. Knowledge Object-oriented programming - CS core Lots of polymorphism and multiple inheritance
    82. 82. Knowledge Object-oriented programming - CS core Lots of polymorphism and multiple inheritance Linux skills, shell scripting - CS370 (Operating Systems)
    83. 83. Knowledge Object-oriented programming - CS core Lots of polymorphism and multiple inheritance Linux skills, shell scripting - CS370 (Operating Systems) Python programming language
    84. 84. Knowledge Object-oriented programming - CS core Lots of polymorphism and multiple inheritance Linux skills, shell scripting - CS370 (Operating Systems) Python programming language Event programming - PyMT/Pyglet and GTK+ (to build test interface to window manager) event systems
    85. 85. Knowledge Object-oriented programming - CS core Lots of polymorphism and multiple inheritance Linux skills, shell scripting - CS370 (Operating Systems) Python programming language Event programming - PyMT/Pyglet and GTK+ (to build test interface to window manager) event systems Open-source dev techniques - version control, etc.
    86. 86. Advice
    87. 87. Advice Hardware always takes time
    88. 88. Advice Hardware always takes time Make bold decisions - be wary of the paradox of choice
    89. 89. Advice Hardware always takes time Make bold decisions - be wary of the paradox of choice Use version control software - git and github.com
    90. 90. Advice Hardware always takes time Make bold decisions - be wary of the paradox of choice Use version control software - git and github.com Build on open platforms when possible
    91. 91. Advice Hardware always takes time Make bold decisions - be wary of the paradox of choice Use version control software - git and github.com Build on open platforms when possible the “shoulders of giants” principle
    92. 92. Difficulties and scope control...
    93. 93. Difficulties and scope control... Pile interaction there but disabled for demo
    94. 94. Difficulties and scope control... Pile interaction there but disabled for demo Auto-clustering algorithms designed but not coded
    95. 95. Difficulties and scope control... Pile interaction there but disabled for demo Auto-clustering algorithms designed but not coded Ordering and shipping delays slowed construction
    96. 96. Difficulties and scope control... Pile interaction there but disabled for demo Auto-clustering algorithms designed but not coded Ordering and shipping delays slowed construction Transportation to demos led to hardware damage
    97. 97. Extensions
    98. 98. Extensions Implement other gestures planned and more
    99. 99. Extensions Implement other gestures planned and more User interaction/efficiency studies
    100. 100. Extensions Implement other gestures planned and more User interaction/efficiency studies Add launcher functionality
    101. 101. Acknowledgements
    102. 102. Acknowledgements It takes a college, a city, and an Internet to create this technology...
    103. 103. Acknowledgements: Presentation
    104. 104. Acknowledgements: Presentation 3d images of multi-touch techniques: Nolan, from PeauProductions.org and NUI Group Forums Schematics and samples for techniques: Seth (cerupcat) from NUI Group Forums Touch gesture images, from the Designing Gestural Interfaces book by Dan Saffer, drawings by Rachel Glaves: DesigningGesturalInterfaces.com
    105. 105. Acknowledgements: Project Support
    106. 106. Acknowledgements: Project Support SNC IT and Tech Support Mr. John Friedrich, Division Manager at Midland Plastics Inc. - Contributed acrylic sheets and materials expertise Dr. Michael Olson (Physics) - Laser information, lasers and safety equipment, laser assembly space SNC Computer Science Department - hard drive for Linux on test machine, camera for IR sensing conversion, construction and management advice, assembly space My family - construction assistance
    107. 107. Acknowledgements - Software
    108. 108. Acknowledgements - Software Ubuntu Linux and GNOME
    109. 109. Acknowledgements - Software Ubuntu Linux and GNOME Python
    110. 110. Acknowledgements - Software Ubuntu Linux and GNOME Python libwnck and python-gnome for window management
    111. 111. Acknowledgements - Software Ubuntu Linux and GNOME Python libwnck and python-gnome for window management setpwc for webcam adjustments
    112. 112. Acknowledgements - Software Ubuntu Linux and GNOME Python libwnck and python-gnome for window management setpwc for webcam adjustments NUI Group’s tBeta (using OpenCV) for blob tracking
    113. 113. Acknowledgements - Software Ubuntu Linux and GNOME Python libwnck and python-gnome for window management setpwc for webcam adjustments NUI Group’s tBeta (using OpenCV) for blob tracking PyMT for interface development
    114. 114. The Tactus Project Hope you enjoyed this walkthrough!
    115. 115. The Tactus Project Hope you enjoyed this walkthrough!

    ×