Build Your Own Multi-Touch Interface with Java and JavaFX Technology

Build Your Own Multi-Touch Interface with Java and JavaFX Technology Simon Ritter, Sun Microsystems Angela Caicedo, Sun Microsystems TS-6127

TM Learn how to build a Java technology-powered touch screen that recognises multiple touch TM points. Also see how easy it is to use JavaFX technology to build user interfaces that work with multi-touch input. 2008 JavaOneSM Conference | java.sun.com/javaone | 2

Agenda Basic ideas of the multi-touch display Construction of the display Using Java technology to recognise touch points JavaFX technology integration with multi-touch events JavaFX technology multi-touch user interfaces Future directions Summary and further information 2008 JavaOneSM Conference | java.sun.com/javaone | 3

Touch Screen Basics Several different approaches • Special surface coating - capacitive, resistive • Surface acoustic wave, acoustic pulse • Optical – Frustrated total internal reflection Most screens only respond to single touch point • Touch replaces use of mouse Multi-touch is becoming more popular • Apple iPhone • Samsung multi-touch display • Microsoft table 2008 JavaOneSM Conference | java.sun.com/javaone | 5

Frustrated Total Internal Reflection Similar concept to fibre optic cable Touching screen causes IR to appear where touched Finger Acrylic Sheet Infra-Red Source (LED) Screen Web-cam Projector 2008 JavaOneSM Conference | java.sun.com/javaone | 6

Touch Surface Thick acrylic sheet • We used 6mm thick, 50x40cm Edges need to be polished We used a three stage process • Back of a hacksaw blade • Very fine sandpaper • Car polish Screen for display needs to be separate • Coating the underside of the touch surface was a complete disaster • Used a separate 3mm thick acrylic sheet covered in tracing paper 2008 JavaOneSM Conference | java.sun.com/javaone | 8

Infra-Red Source/Detection Infra-red LEDs mounted along edge of screen • Initially mounted one LED every 7mm • This was reduced to one LED every 3cm (lower power consumption) Webcam mounted under screen to observe touch points • Webcams can see infra-red, unlike humans • Most webcams have an infra-red filter which must be removed • This can be difficult (we destroyed several webcams!) • Replace this with a filter that only lets IR through • Use the filter from a remote control • Tricky bit is getting the webcam far enough away to see whole screen • Wide angle lens makes this easier 2008 JavaOneSM Conference | java.sun.com/javaone | 9

Infra-Red LED Wiring LEDs wired in parallel 100 Ohm resistor for each LED • Good, bright LED • Not too much power consumption • Not too hot 2008 JavaOneSM Conference | java.sun.com/javaone | 10

Infra-Red LED Mounting Each LED machined flat Better optical connectivity to touch surface 2008 JavaOneSM Conference | java.sun.com/javaone | 11

Touch Surface IR LEDs 2008 JavaOneSM Conference | java.sun.com/javaone | 12

Touch Surface with Screen IR LEDs Display screen Touch surface 2008 JavaOneSM Conference | java.sun.com/javaone | 13

Webcam Without filter With IR bandpass filter 2008 JavaOneSM Conference | java.sun.com/javaone | 14

Getting an Image From a Webcam Java Media Framework (JMF) API • The forgotten Java API, last update 2003 Version 1 only for playback, version 2 introduced capture Still works well for SolarisTM (SPARC®) technology, Linux and Windows • Pure Java technology version available • Performance packs for specific OS • Decision was to use Ubuntu Linux for driver support Performance was not an issue • Able to grab and process images at suitably high frame rate 2008 JavaOneSM Conference | java.sun.com/javaone | 16

JMF API Frame Grabbing Locate device with CaptureDataManager Get DataSource through MediaLocator Set Format Create and Realize Processor Create a PushBufferDataSource • We really want a PullBufferDataSource, but can't have one Get BufferStream through DataSource Read Buffer Convert Buffer to BufferedImage with BufferToImage Convert BufferedImage to RGB array with PixelGrabber 2008 JavaOneSM Conference | java.sun.com/javaone | 17

Image Processing: Stage 1 Image from camera was not good for processing • Camera had automatic adjustment of white balance, contrast, etc. Use webcam device driver ioctls • Turn off automatic adjustment • Retrieve current settings • Manually change settings Driver needed modification • Yay open source! 2008 JavaOneSM Conference | java.sun.com/javaone | 18

Image Processing: Stage 2 Java 2DTM and Java Advanced Imaging API Convert to 8-bit greyscale for simplified processing • ColorConvolveOp BufferedImage is really useful for this kind of processing BufferedImage ColorModel Raster SampleModel ColorSpace DataBuffer 2008 JavaOneSM Conference | java.sun.com/javaone | 19

Brightness and Contrast Adjustment Filtered Pixel Values Window / 2 Window Level Image Pixel Values Create a ByteLookupTable Apply LookupOp filter LookupOp lop = new LookupOp(byteTable, null); filteredImage = lop.filter(oldImage, null); 2008 JavaOneSM Conference | java.sun.com/javaone | 20

Image Processing: Stage 3 Identify where bright spots are in image Algorithm is basically simple • Although certain cases make life a bit harder Ensure image is mono-chrome, each pixel is white or black • 8-bit greyscale -> 1 bit mono-chrome • Tunable level for change from black to white (0-255) Sum pixels in each row and column (white = 1, black = 0) Rows or columns with high values indicate touch point Generate a set of x, y co-ordinates for all touch points 2008 JavaOneSM Conference | java.sun.com/javaone | 21

Image Processing: Stage 3 2 3 3 y point 1 3 3 y point 2 3 x point 1 242 333 x point 2 Points can be reversed Test possible locations to get accurate result 2008 JavaOneSM Conference | java.sun.com/javaone | 22

Generating Touch Events We now have a set of touch point co-ordinates Simple events are somewhat like mouse events • Change of position • New touch point • Touch point disappeared Touch point code is more complex than mouse • Must track points and figure out when a new one appears • Or an existing one disappears Gesture recognition is even more complex • Swipe movement • expand/shrink/rotate using two points • Requires time-based analysis We defined an extendable touch event interface 2008 JavaOneSM Conference | java.sun.com/javaone | 23

JavaFX Technology Basics Programming Language for the Java Platform Object-oriented Declarative Syntax Statically-typed with type-inference Automatic data binding Extensive Widget library encompassing Swing and Java 2D API Development tools including NetBeansTM and Eclipse IDE plugins 2008 JavaOneSM Conference | java.sun.com/javaone | 25

Java/JavaFX Technology and Multi-Touch Software Integration Why JavaFX technology: • Great for image manipulation • Data binding simplifies the software implementation • Simple and easy to design and implement the user interface Why Java technology: • Heavy multi-threading implemented in Java technology • Java based model bound to the interface with JavaFX technology capabilities • Model change <-> view change Why Multi-touch: • Intuitive user interaction • Multiple point of interactions • Allows multiple users interacting with the software • Cool and interesting technology 2008 JavaOneSM Conference | java.sun.com/javaone | 26

Multi-Touch Events They capture: • Initial position (finger down) • Direction (finger trajectory) • Speed (finger speed) • Final position (finger up) Dynamically associated with SmartObject Dynamically released from SmartObjects Universe vs. individual interaction Direction = 0 and speed = 0 then locking events Speed > threshold then animation triggered 2008 JavaOneSM Conference | java.sun.com/javaone | 28

Smart Objects Register multi-touch events Object's behavior based on multi-touch events and areas of interactions rotating & rotating, rotating & stretching stretching & stretching flipping rotating, rotating, stretching & rotating, stretching & flipping flipping rotating & rotating, rotating & stretching stretching & stretching flipping 2008 JavaOneSM Conference | java.sun.com/javaone | 29

Stretching with Locking Action 2008 JavaOneSM Conference | java.sun.com/javaone | 30

Both Sides Stretching 2008 JavaOneSM Conference | java.sun.com/javaone | 31

Flipping 2008 JavaOneSM Conference | java.sun.com/javaone | 32

Locking and Rotating 2008 JavaOneSM Conference | java.sun.com/javaone | 33

Free Rotation 2008 JavaOneSM Conference | java.sun.com/javaone | 34

Putting Things Together 2008 JavaOneSM Conference | java.sun.com/javaone | 35

Universe Interaction (1 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 36

Universe Interaction (2 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 37

Smart Objects Still Active (1 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 38

Smart Objects Still Active (2 of 2) 2008 JavaOneSM Conference | java.sun.com/javaone | 39

Where Next? Infra-red is very useful as we can't see it • No interference with rest of display Wiimote from Nintendo has IR detector • Can track up to four points simultaneously • Reports positions via bluetooth interface • Used to figure out where the Wiimote is relative to the display Why not keep the Wiimote stationary, move the IR? Great examples of this • Johnny Chung Lee of CMU • 3D position interpolation Working on combining this with touch screen • Using wiiremotej open source project 2008 JavaOneSM Conference | java.sun.com/javaone | 41

Summary Multi-touch screens are simple and cheap to build • Most expensive bit is the projector Java technology makes the software easy JavaFX technology is easy to integrate with new types of event JavaFX technology makes building multi-touch user interfaces simple This is just the beginning... 2008 JavaOneSM Conference | java.sun.com/javaone | 43

For More Information Java Media Framework • java.sun.com/products/java-media/jmf/ Java Advanced Imaging • java.sun.com/javase/technologies/desktop/media/jai/ JavaFX technology • openjfx.org Jeff Hahn (FTIR multi-touch screen) • cs.nyu.edu/~jhan/ftirtouch/ • www.perceptivepixel.com/ Building Imaging Applications With Java Technology • Lawrence H. Rodrigues 2008 JavaOneSM Conference | java.sun.com/javaone | 44

Multi-touch screen in action with JavaFX technology 2008 JavaOneSM Conference | java.sun.com/javaone | 45

Simon Ritter, Sun Microsystems Angela Caicedo, Sun Microsystems TS-6127 2008 JavaOneSM Conference | java.sun.com/javaone | 46

Build Your Own Multi-Touch Interface with Java and JavaFX Technology

1 comments

Notes on slide 1

3 Favorites