Build Your Own Multi-Touch Interface with Java and JavaFX Technology

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

1 comments

Comments 1 - 1 of 1 previous next Post a comment

  • + guestc8a091 guestc8a091 11 months ago
    Very good ideas here both for fun and real tasks.
    Is JavaFX currently used for medical imaging such as digital x-ray and MRI presentation?
Post a comment
Embed Video
Edit your comment Cancel

3 Favorites

Build Your Own Multi-Touch Interface with Java and JavaFX Technology - Presentation Transcript

  1. Build Your Own Multi-Touch Interface with Java and JavaFX Technology Simon Ritter, Sun Microsystems Angela Caicedo, Sun Microsystems TS-6127
  2. TM Learn how to build a Java technology-powered touch screen that recognises multiple touch TM points. Also see how easy it is to use JavaFX technology to build user interfaces that work with multi-touch input. 2008 JavaOneSM Conference | java.sun.com/javaone | 2
  3. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 3
  4. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 4
  5. Touch Screen Basics Several different approaches • Special surface coating - capacitive, resistive • Surface acoustic wave, acoustic pulse • Optical – Frustrated total internal reflection Most screens only respond to single touch point • Touch replaces use of mouse Multi-touch is becoming more popular • Apple iPhone • Samsung multi-touch display • Microsoft table 2008 JavaOneSM Conference | java.sun.com/javaone | 5
  6. Frustrated Total Internal Reflection Similar concept to fibre optic cable Touching screen causes IR to appear where touched Finger Acrylic Sheet Infra-Red Source (LED) Screen Web-cam Projector 2008 JavaOneSM Conference | java.sun.com/javaone | 6
  7. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 7
  8. Touch Surface Thick acrylic sheet • We used 6mm thick, 50x40cm Edges need to be polished We used a three stage process • Back of a hacksaw blade • Very fine sandpaper • Car polish Screen for display needs to be separate • Coating the underside of the touch surface was a complete disaster • Used a separate 3mm thick acrylic sheet covered in tracing paper 2008 JavaOneSM Conference | java.sun.com/javaone | 8
  9. Infra-Red Source/Detection Infra-red LEDs mounted along edge of screen • Initially mounted one LED every 7mm • This was reduced to one LED every 3cm (lower power consumption) Webcam mounted under screen to observe touch points • Webcams can see infra-red, unlike humans • Most webcams have an infra-red filter which must be removed • This can be difficult (we destroyed several webcams!) • Replace this with a filter that only lets IR through • Use the filter from a remote control • Tricky bit is getting the webcam far enough away to see whole screen • Wide angle lens makes this easier 2008 JavaOneSM Conference | java.sun.com/javaone | 9
  10. Infra-Red LED Wiring LEDs wired in parallel 100 Ohm resistor for each LED • Good, bright LED • Not too much power consumption • Not too hot 2008 JavaOneSM Conference | java.sun.com/javaone | 10
  11. Infra-Red LED Mounting Each LED machined flat Better optical connectivity to touch surface 2008 JavaOneSM Conference | java.sun.com/javaone | 11
  12. Touch Surface IR LEDs 2008 JavaOneSM Conference | java.sun.com/javaone | 12
  13. Touch Surface with Screen IR LEDs Display screen Touch surface 2008 JavaOneSM Conference | java.sun.com/javaone | 13
  14. Webcam Without filter With IR bandpass filter 2008 JavaOneSM Conference | java.sun.com/javaone | 14
  15. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 15
  16. Getting an Image From a Webcam Java Media Framework (JMF) API • The forgotten Java API, last update 2003 Version 1 only for playback, version 2 introduced capture Still works well for SolarisTM (SPARC®) technology, Linux and Windows • Pure Java technology version available • Performance packs for specific OS • Decision was to use Ubuntu Linux for driver support Performance was not an issue • Able to grab and process images at suitably high frame rate 2008 JavaOneSM Conference | java.sun.com/javaone | 16
  17. JMF API Frame Grabbing Locate device with CaptureDataManager Get DataSource through MediaLocator Set Format Create and Realize Processor Create a PushBufferDataSource • We really want a PullBufferDataSource, but can't have one Get BufferStream through DataSource Read Buffer Convert Buffer to BufferedImage with BufferToImage Convert BufferedImage to RGB array with PixelGrabber 2008 JavaOneSM Conference | java.sun.com/javaone | 17
  18. Image Processing: Stage 1 Image from camera was not good for processing • Camera had automatic adjustment of white balance, contrast, etc. Use webcam device driver ioctls • Turn off automatic adjustment • Retrieve current settings • Manually change settings Driver needed modification • Yay open source! 2008 JavaOneSM Conference | java.sun.com/javaone | 18
  19. Image Processing: Stage 2 Java 2DTM and Java Advanced Imaging API Convert to 8-bit greyscale for simplified processing • ColorConvolveOp BufferedImage is really useful for this kind of processing BufferedImage ColorModel Raster SampleModel ColorSpace DataBuffer 2008 JavaOneSM Conference | java.sun.com/javaone | 19
  20. Brightness and Contrast Adjustment Filtered Pixel Values Window / 2 Window Level Image Pixel Values Create a ByteLookupTable Apply LookupOp filter LookupOp lop = new LookupOp(byteTable, null); filteredImage = lop.filter(oldImage, null); 2008 JavaOneSM Conference | java.sun.com/javaone | 20
  21. Image Processing: Stage 3 Identify where bright spots are in image Algorithm is basically simple • Although certain cases make life a bit harder Ensure image is mono-chrome, each pixel is white or black • 8-bit greyscale -> 1 bit mono-chrome • Tunable level for change from black to white (0-255) Sum pixels in each row and column (white = 1, black = 0) Rows or columns with high values indicate touch point Generate a set of x, y co-ordinates for all touch points 2008 JavaOneSM Conference | java.sun.com/javaone | 21
  22. Image Processing: Stage 3 2 3 3 y point 1 3 3 y point 2 3 x point 1 242 333 x point 2 Points can be reversed Test possible locations to get accurate result 2008 JavaOneSM Conference | java.sun.com/javaone | 22
  23. Generating Touch Events We now have a set of touch point co-ordinates Simple events are somewhat like mouse events • Change of position • New touch point • Touch point disappeared Touch point code is more complex than mouse • Must track points and figure out when a new one appears • Or an existing one disappears Gesture recognition is even more complex • Swipe movement • expand/shrink/rotate using two points • Requires time-based analysis We defined an extendable touch event interface 2008 JavaOneSM Conference | java.sun.com/javaone | 23
  24. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 24
  25. JavaFX Technology Basics Programming Language for the Java Platform Object-oriented Declarative Syntax Statically-typed with type-inference Automatic data binding Extensive Widget library encompassing Swing and Java 2D API Development tools including NetBeansTM and Eclipse IDE plugins 2008 JavaOneSM Conference | java.sun.com/javaone | 25
  26. Java/JavaFX Technology and Multi-Touch Software Integration Why JavaFX technology: • Great for image manipulation • Data binding simplifies the software implementation • Simple and easy to design and implement the user interface Why Java technology: • Heavy multi-threading implemented in Java technology • Java based model bound to the interface with JavaFX technology capabilities • Model change <-> view change Why Multi-touch: • Intuitive user interaction • Multiple point of interactions • Allows multiple users interacting with the software • Cool and interesting technology 2008 JavaOneSM Conference | java.sun.com/javaone | 26
  27. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 27
  28. Multi-Touch Events They capture: • Initial position (finger down) • Direction (finger trajectory) • Speed (finger speed) • Final position (finger up) Dynamically associated with SmartObject Dynamically released from SmartObjects Universe vs. individual interaction Direction = 0 and speed = 0 then locking events Speed > threshold then animation triggered 2008 JavaOneSM Conference | java.sun.com/javaone | 28
  29. Smart Objects Register multi-touch events Object's behavior based on multi-touch events and areas of interactions rotating & rotating, rotating & stretching stretching & stretching flipping rotating, rotating, stretching & rotating, stretching & flipping flipping rotating & rotating, rotating & stretching stretching & stretching flipping 2008 JavaOneSM Conference | java.sun.com/javaone | 29
  30. Stretching with Locking Action 2008 JavaOneSM Conference | java.sun.com/javaone | 30
  31. Both Sides Stretching 2008 JavaOneSM Conference | java.sun.com/javaone | 31
  32. Flipping 2008 JavaOneSM Conference | java.sun.com/javaone | 32
  33. Locking and Rotating 2008 JavaOneSM Conference | java.sun.com/javaone | 33
  34. Free Rotation 2008 JavaOneSM Conference | java.sun.com/javaone | 34
  35. Putting Things Together 2008 JavaOneSM Conference | java.sun.com/javaone | 35
  36. Universe Interaction (1 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 36
  37. Universe Interaction (2 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 37
  38. Smart Objects Still Active (1 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 38
  39. Smart Objects Still Active (2 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 39
  40. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 40
  41. Where Next? Infra-red is very useful as we can't see it • No interference with rest of display Wiimote from Nintendo has IR detector • Can track up to four points simultaneously • Reports positions via bluetooth interface • Used to figure out where the Wiimote is relative to the display Why not keep the Wiimote stationary, move the IR? Great examples of this • Johnny Chung Lee of CMU • 3D position interpolation Working on combining this with touch screen • Using wiiremotej open source project 2008 JavaOneSM Conference | java.sun.com/javaone | 41
  42. Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 42
  43. Summary Multi-touch screens are simple and cheap to build • Most expensive bit is the projector Java technology makes the software easy JavaFX technology is easy to integrate with new types of event JavaFX technology makes building multi-touch user interfaces simple This is just the beginning... 2008 JavaOneSM Conference | java.sun.com/javaone | 43
  44. For More Information Java Media Framework • java.sun.com/products/java-media/jmf/ Java Advanced Imaging • java.sun.com/javase/technologies/desktop/media/jai/ JavaFX technology • openjfx.org Jeff Hahn (FTIR multi-touch screen) • cs.nyu.edu/~jhan/ftirtouch/ • www.perceptivepixel.com/ Building Imaging Applications With Java Technology • Lawrence H. Rodrigues 2008 JavaOneSM Conference | java.sun.com/javaone | 44
  45. Multi-touch screen in action with JavaFX technology 2008 JavaOneSM Conference | java.sun.com/javaone | 45
  46. Simon Ritter, Sun Microsystems Angela Caicedo, Sun Microsystems TS-6127 2008 JavaOneSM Conference | java.sun.com/javaone | 46

+ adorepumpadorepump, 11 months ago

custom

2326 views, 3 favs, 0 embeds more stats

Build Your Own Multi-Touch Interface with
Java and more

More info about this document

© All Rights Reserved

Go to text version

  • Total Views 2326
    • 2326 on SlideShare
    • 0 from embeds
  • Comments 1
  • Favorites 3
  • Downloads 164
Most viewed embeds

more

All embeds

less

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

Cancel
File a copyright complaint
Having problems? Go to our helpdesk?

Categories