Your SlideShare is downloading. ×
0
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Build Your Own Multi-Touch Interface with Java and JavaFX Technology
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Build Your Own Multi-Touch Interface with Java and JavaFX Technology

12,366

Published on

Build Your Own Multi-Touch Interface with …

Build Your Own Multi-Touch Interface with
Java and JavaFX Technology
Simon Ritter, Sun Microsystems
Angela Caicedo, Sun Microsystems
TS-6127

Published in: Technology, Education
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
12,366
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
449
Comments
0
Likes
9
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Build Your Own Multi-Touch Interface with Java and JavaFX Technology Simon Ritter, Sun Microsystems Angela Caicedo, Sun Microsystems TS-6127
  • 2. TM Learn how to build a Java technology-powered touch screen that recognises multiple touch TM points. Also see how easy it is to use JavaFX technology to build user interfaces that work with multi-touch input. 2008 JavaOneSM Conference | java.sun.com/javaone | 2
  • 3. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 3
  • 4. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 4
  • 5. Touch Screen Basics Several different approaches • Special surface coating - capacitive, resistive • Surface acoustic wave, acoustic pulse • Optical – Frustrated total internal reflection Most screens only respond to single touch point • Touch replaces use of mouse Multi-touch is becoming more popular • Apple iPhone • Samsung multi-touch display • Microsoft table 2008 JavaOneSM Conference | java.sun.com/javaone | 5
  • 6. Frustrated Total Internal Reflection Similar concept to fibre optic cable Touching screen causes IR to appear where touched Finger Acrylic Sheet Infra-Red Source (LED) Screen Web-cam Projector 2008 JavaOneSM Conference | java.sun.com/javaone | 6
  • 7. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 7
  • 8. Touch Surface Thick acrylic sheet • We used 6mm thick, 50x40cm Edges need to be polished We used a three stage process • Back of a hacksaw blade • Very fine sandpaper • Car polish Screen for display needs to be separate • Coating the underside of the touch surface was a complete disaster • Used a separate 3mm thick acrylic sheet covered in tracing paper 2008 JavaOneSM Conference | java.sun.com/javaone | 8
  • 9. Infra-Red Source/Detection Infra-red LEDs mounted along edge of screen • Initially mounted one LED every 7mm • This was reduced to one LED every 3cm (lower power consumption) Webcam mounted under screen to observe touch points • Webcams can see infra-red, unlike humans • Most webcams have an infra-red filter which must be removed • This can be difficult (we destroyed several webcams!) • Replace this with a filter that only lets IR through • Use the filter from a remote control • Tricky bit is getting the webcam far enough away to see whole screen • Wide angle lens makes this easier 2008 JavaOneSM Conference | java.sun.com/javaone | 9
  • 10. Infra-Red LED Wiring LEDs wired in parallel 100 Ohm resistor for each LED • Good, bright LED • Not too much power consumption • Not too hot 2008 JavaOneSM Conference | java.sun.com/javaone | 10
  • 11. Infra-Red LED Mounting Each LED machined flat Better optical connectivity to touch surface 2008 JavaOneSM Conference | java.sun.com/javaone | 11
  • 12. Touch Surface IR LEDs 2008 JavaOneSM Conference | java.sun.com/javaone | 12
  • 13. Touch Surface with Screen IR LEDs Display screen Touch surface 2008 JavaOneSM Conference | java.sun.com/javaone | 13
  • 14. Webcam Without filter With IR bandpass filter 2008 JavaOneSM Conference | java.sun.com/javaone | 14
  • 15. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 15
  • 16. Getting an Image From a Webcam Java Media Framework (JMF) API • The forgotten Java API, last update 2003 Version 1 only for playback, version 2 introduced capture Still works well for SolarisTM (SPARC®) technology, Linux and Windows • Pure Java technology version available • Performance packs for specific OS • Decision was to use Ubuntu Linux for driver support Performance was not an issue • Able to grab and process images at suitably high frame rate 2008 JavaOneSM Conference | java.sun.com/javaone | 16
  • 17. JMF API Frame Grabbing Locate device with CaptureDataManager Get DataSource through MediaLocator Set Format Create and Realize Processor Create a PushBufferDataSource • We really want a PullBufferDataSource, but can't have one Get BufferStream through DataSource Read Buffer Convert Buffer to BufferedImage with BufferToImage Convert BufferedImage to RGB array with PixelGrabber 2008 JavaOneSM Conference | java.sun.com/javaone | 17
  • 18. Image Processing: Stage 1 Image from camera was not good for processing • Camera had automatic adjustment of white balance, contrast, etc. Use webcam device driver ioctls • Turn off automatic adjustment • Retrieve current settings • Manually change settings Driver needed modification • Yay open source! 2008 JavaOneSM Conference | java.sun.com/javaone | 18
  • 19. Image Processing: Stage 2 Java 2DTM and Java Advanced Imaging API Convert to 8-bit greyscale for simplified processing • ColorConvolveOp BufferedImage is really useful for this kind of processing BufferedImage ColorModel Raster SampleModel ColorSpace DataBuffer 2008 JavaOneSM Conference | java.sun.com/javaone | 19
  • 20. Brightness and Contrast Adjustment Filtered Pixel Values Window / 2 Window Level Image Pixel Values Create a ByteLookupTable Apply LookupOp filter LookupOp lop = new LookupOp(byteTable, null); filteredImage = lop.filter(oldImage, null); 2008 JavaOneSM Conference | java.sun.com/javaone | 20
  • 21. Image Processing: Stage 3 Identify where bright spots are in image Algorithm is basically simple • Although certain cases make life a bit harder Ensure image is mono-chrome, each pixel is white or black • 8-bit greyscale -> 1 bit mono-chrome • Tunable level for change from black to white (0-255) Sum pixels in each row and column (white = 1, black = 0) Rows or columns with high values indicate touch point Generate a set of x, y co-ordinates for all touch points 2008 JavaOneSM Conference | java.sun.com/javaone | 21
  • 22. Image Processing: Stage 3 2 3 3 y point 1 3 3 y point 2 3 x point 1 242 333 x point 2 Points can be reversed Test possible locations to get accurate result 2008 JavaOneSM Conference | java.sun.com/javaone | 22
  • 23. Generating Touch Events We now have a set of touch point co-ordinates Simple events are somewhat like mouse events • Change of position • New touch point • Touch point disappeared Touch point code is more complex than mouse • Must track points and figure out when a new one appears • Or an existing one disappears Gesture recognition is even more complex • Swipe movement • expand/shrink/rotate using two points • Requires time-based analysis We defined an extendable touch event interface 2008 JavaOneSM Conference | java.sun.com/javaone | 23
  • 24. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 24
  • 25. JavaFX Technology Basics Programming Language for the Java Platform Object-oriented Declarative Syntax Statically-typed with type-inference Automatic data binding Extensive Widget library encompassing Swing and Java 2D API Development tools including NetBeansTM and Eclipse IDE plugins 2008 JavaOneSM Conference | java.sun.com/javaone | 25
  • 26. Java/JavaFX Technology and Multi-Touch Software Integration Why JavaFX technology: • Great for image manipulation • Data binding simplifies the software implementation • Simple and easy to design and implement the user interface Why Java technology: • Heavy multi-threading implemented in Java technology • Java based model bound to the interface with JavaFX technology capabilities • Model change <-> view change Why Multi-touch: • Intuitive user interaction • Multiple point of interactions • Allows multiple users interacting with the software • Cool and interesting technology 2008 JavaOneSM Conference | java.sun.com/javaone | 26
  • 27. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 27
  • 28. Multi-Touch Events They capture: • Initial position (finger down) • Direction (finger trajectory) • Speed (finger speed) • Final position (finger up) Dynamically associated with SmartObject Dynamically released from SmartObjects Universe vs. individual interaction Direction = 0 and speed = 0 then locking events Speed > threshold then animation triggered 2008 JavaOneSM Conference | java.sun.com/javaone | 28
  • 29. Smart Objects Register multi-touch events Object's behavior based on multi-touch events and areas of interactions rotating & rotating, rotating & stretching stretching & stretching flipping rotating, rotating, stretching & rotating, stretching & flipping flipping rotating & rotating, rotating & stretching stretching & stretching flipping 2008 JavaOneSM Conference | java.sun.com/javaone | 29
  • 30. Stretching with Locking Action 2008 JavaOneSM Conference | java.sun.com/javaone | 30
  • 31. Both Sides Stretching 2008 JavaOneSM Conference | java.sun.com/javaone | 31
  • 32. Flipping 2008 JavaOneSM Conference | java.sun.com/javaone | 32
  • 33. Locking and Rotating 2008 JavaOneSM Conference | java.sun.com/javaone | 33
  • 34. Free Rotation 2008 JavaOneSM Conference | java.sun.com/javaone | 34
  • 35. Putting Things Together 2008 JavaOneSM Conference | java.sun.com/javaone | 35
  • 36. Universe Interaction (1 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 36
  • 37. Universe Interaction (2 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 37
  • 38. Smart Objects Still Active (1 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 38
  • 39. Smart Objects Still Active (2 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 39
  • 40. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 40
  • 41. Where Next? Infra-red is very useful as we can't see it • No interference with rest of display Wiimote from Nintendo has IR detector • Can track up to four points simultaneously • Reports positions via bluetooth interface • Used to figure out where the Wiimote is relative to the display Why not keep the Wiimote stationary, move the IR? Great examples of this • Johnny Chung Lee of CMU • 3D position interpolation Working on combining this with touch screen • Using wiiremotej open source project 2008 JavaOneSM Conference | java.sun.com/javaone | 41
  • 42. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 42
  • 43. Summary Multi-touch screens are simple and cheap to build • Most expensive bit is the projector Java technology makes the software easy JavaFX technology is easy to integrate with new types of event JavaFX technology makes building multi-touch user interfaces simple This is just the beginning... 2008 JavaOneSM Conference | java.sun.com/javaone | 43
  • 44. For More Information Java Media Framework • java.sun.com/products/java-media/jmf/ Java Advanced Imaging • java.sun.com/javase/technologies/desktop/media/jai/ JavaFX technology • openjfx.org Jeff Hahn (FTIR multi-touch screen) • cs.nyu.edu/~jhan/ftirtouch/ • www.perceptivepixel.com/ Building Imaging Applications With Java Technology • Lawrence H. Rodrigues 2008 JavaOneSM Conference | java.sun.com/javaone | 44
  • 45. Multi-touch screen in action with JavaFX technology 2008 JavaOneSM Conference | java.sun.com/javaone | 45
  • 46. Simon Ritter, Sun Microsystems Angela Caicedo, Sun Microsystems TS-6127 2008 JavaOneSM Conference | java.sun.com/javaone | 46

×