Virtual Valipilla - Air Gesture Based Tool for Practicing Writing
•Download as PPTX, PDF•
1 like•781 views
Master of Computer Science Viva Presentation Air Writing Application using Leap Motion and $P Point Cloud Framework
More Related Content
Similar to Virtual Valipilla - Air Gesture Based Tool for Practicing Writing
Similar to Virtual Valipilla - Air Gesture Based Tool for Practicing Writing (20)
More from Upekha Vandebona
More from Upekha Vandebona (20)
Recently uploaded
Recently uploaded (20)
Virtual Valipilla - Air Gesture Based Tool for Practicing Writing
- 1. Virtual “Valipilla” : Air Gesture Based Tool for Practicing the English Alphabet Writing U.V Vandebona Reg No. : 2013/MCS/072 Index No. : 13440722 Supervisor : Dr. G.D.S.P. Wimalaratne Master of Computer Science - UCSC - Final Year Project - 2016Date : 2016-Feb-9
- 2. Air Writing • Yoshihiro Itaguchi et al, “Writing in the Air: Contributions of Finger Movement to Cognitive Processing”, Journal of Public Library of Science - PLoS One, June 2015 • Human Computer Interaction - Writing interaction with the computer • Machine Learning - Recognize what was written HCI ML
- 3. Popular Consumer Vision Sensor Devices Microsoft Kinect [1] Image Reference: [1] http://www.xbox.com/en-US/xbox-one/accessories/kinect-for-xbox-one [2] http://asia.creative.com/p/web-cameras/creative-senz3d [3] http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html [4] https://www.leapmotion.com/ Leap Motion [4] Normal Web Cams and 3D Cams Intel RealSense [3]Creative Senz3D [2]
- 4. Air Written Character Recognition Techniques Warping Methods Dynamic Time Warping (DTW) Hilbert Warping (HW) Statistical Methods Hidden Markov Model (HMM) Artificial Neural Network (ANN) Template Matching $P Cloud Point
- 5. Related Work • Kinect Sensor with DTW and SVM Approach – Chengzhang Qu et al, “Online Kinect Handwritten Digit Recognition Based on Dynamic Time Warping and Support Vector Machine”, Journal of Information & Computational Science, pp 413 - 422, January 2015 • Kinect Sensor with Neural Gas Network Approach – MRA Heidari et al, “Writing in the Air Using Kinect and Growing Neural Gas Network”, Jurnal Teknologi Universiti Teknologi Malaysia, vol. 72, no. 5, 2014 • Leap Motion Sensor with DTW Approach – Vikram Sharad et al, "Writing and sketching in the air, recognizing and controlling on the fly.," in ACM Conference on Human Factors in Computing Systems (CHI), 2013. • Leap Motion Sensor with HMM Approach – Mingyu Chen, "Universal Motion Based Control," School of Electrical and Computer Engineering, Georgia Institute of Technology, PhD. Dissertation 2013. • Webcam with Neural Network Approach – Aditya G. Joshi et al, "Touchless Writer a hand gesture recognizer for Englsih characters," in Proceedings of 22nd IRF International Conference, Pune, India, January 2015. • Webcam with HW Approach – Hiroyuki Ishida, Tomokazu Takahashi, Ide Ichiro, and Murase Hiroshi, "A Hilbert warping method for handwriting gesture recognition," Pattern Recognition, vol. 43, no. 0031-3203, pp. 2799-2806, August 2010.
- 6. Proposed Prototype Writing Capturing Recognition Feedback Left Handed Right Handed
- 7. Gesture Type Finger Tip Tool Tip • Pencil tip • Pen tip Left Handed Right Handed • Start • Clear • Check • Back • Sound • Next
- 8. Motion Segmentation Motion Segmentation Techniques Explicit Delimitation Key/Button Press Break by Gesture Virtual Touch Zone Automatic Detection Spotting Approach Sliding Window Approach • Vision input devices constantly streams the location of the fingers within its field of view.
- 9. Virtual Touch Input Zone +1 0 -1 Touch Zone Hover Zone Cursor Pointer
- 10. Proposed Approach Writing Capturing Recognition Feedback 2D Virtual Writing Interface
- 11. Proposed Approach Writing Capturing Recognition Feedback Normalization Process • Resample • Scale with Shape Preservation • Translation to Origin Measure Cloud Distance • Use Euclidean Distance Greedy Cloud Match • Recognized Template $P Point Cloud Framework
- 12. $P Point Cloud • “The $P Point-Cloud Recognizer is a 2-D gesture recognizer designed for rapid prototyping of gesture-based user interfaces. In machine learning terms, $P is an instance- based nearest-neighbor classifier with a Euclidean scoring function, i.e., a geometric template matcher [1]. ” Reference [1] (2016, January), MAD Lab - University of Washington, $P Point-Cloud Recognizer, [Online] http://depts.washington.edu/aimgroup/proj/dollar/pdollar.html
- 13. $P Point Cloud - Advantages • Independent of – Scale – Number of Strokes – Stroke Direction – Stroke Order – Stroke Type (Uni-Stroke & Multi Stroke)
- 14. $P Point Cloud - Normalization -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 -0.8 -0.3 0.2 0.7 0 50 100 150 200 250 0 50 100 150 200 250 Stroke 2 Template Defined for letter ‘R’ Normalized Template for Matching • Resample • Scale with Shape Preservation • Translation to Origin Stroke 1 Stroke 3
- 15. $P Point Cloud - Normalization Air Writing for letter ‘R’ Normalized Air Writing for Matching • Resample • Scale with Shape Preservation • Translation to Origin 0 100 200 300 400 500 600 700 800 900 1000 0 200 400 600 800 1000 -0.6 -0.4 -0.2 -1E-15 0.2 0.4 0.6 -0.6 -0.1 0.4
- 16. $P Point Cloud - Recognition Template Library (T) Candidate (C) (Tt1) (Tt2) (Tt3) d1 d2 d3 Cloud Distance : d1 < d2 < d3
- 17. -0.6 -0.4 -0.2 -1E-15 0.2 0.4 0.6 -0.6 -0.4 -0.2 -1E-15 0.2 0.4 0.6 Template Point Cloud (Normalized) Candidate Air Writing Point Cloud (Normalized) Matching as an Assignment Problem ?
- 18. $P Point Cloud - Greedy Cloud Matching Reference [1] (2016, January), MAD Lab - University of Washington, $P Point-Cloud Recognizer, [Online] http://depts.washington.edu/aimgroup/proj/dollar/pdollar.html
- 19. 12 3 $P Point Cloud - Cloud Distance Template Point Cloud (Normalized) Candidate Air Writing Point Cloud (Normalized) 3 4 21 12 3 3 4 21 44 Tj Ci Tj Ci Sum of Euclidean distances with a confidence weight Loop 1.1 Loop 1.1.2 12 3 3 4 21 4 Tj Ci Loop 1.2 12 3 3 4 21 4 Tj Ci Loop 2…. Minimum CloudDistance
- 20. $P Point Cloud - Cloud Distance Reference [1] (2016, January), MAD Lab - University of Washington, $P Point-Cloud Recognizer, [Online] http://depts.washington.edu/aimgroup/proj/dollar/pdollar.html
- 21. Writing Capturing Recognition Feedback • Cloud Distance is normalize as a score between 0 and 1 and presented as a percentage which is beneficial for a general user to understand. Proposed Approach Recognized As Feedback
- 22. Evaluation - Character Recognition No of English letters tested : 26 No of samples from each letter tested : 24 Path On Mode : 12 Left Handed : 6 Right Handed : 6 Path Off Mode : 12 Left Handed : 6 Right Handed : 6 Total tested samples for each gesture type : 26 24 = 624 [312 samples on each mode]
- 23. Results - Character Recognition • Accuracy on Tooltip Air Writing A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 0.00 20.00 40.00 60.00 80.00 100.00 120.00 0 5 10 15 20 25 Precision Sensitivity • Path On Mode = 100% • Path Off Mode = 90.7%
- 24. Ergonomics • Involve more muscles; As a result cause more arm fatigue • Educational Intervals in-between • Rest the elbow and writes with the movement of the upper arm and wrist. • Workstation/Leap Motion positioning adjustments. • Preferred tool based air writing
- 25. Conclusion • For Practice Writing : Touch Vs Vision – Wide Interaction Area – Invulnerable to Scratches • User doesn’t need to carry an expensive item which prone to accidental damages. – With improvements can be used as a touch based system • Using physical touch plane obtained from the environment (ex: a wall, a board, etc.)
Editor's Notes
- My project is about developing an air gesture based tool for practice English Alphabet Writing using the traditional teaching tool metaphor the Sri Lankan Culture have; valipilla, in a virtual space.
- This approach is generally called as Air Writing in the primary education system to improve cognitive processing. And those educationists use this as a technique to teach writing before using papers to write. So what I tried to do was provide a computerized platform for the writing interaction and the recognition process. This approach combines two popular research areas, HCI and ML.
- If I go the HCI part 1st, to capture the writing interaction in the air, we need a vision sensor device. Most popular consumer device is Microsoft Kinect, and it’s specifically focusing on full body motion capturing. Also we can use Web cameras; the newest trend 3D web cams such like creative’s Senz3D and Intel’s RealSense, and they are more focusing on web cam activities such like video conferencing or else facial activities. Well for our problem, we need something specifically for capture hand and finger motion. So the ideal solution is Leap Motion device. It can recognize each of the five fingers on each of the hand, and can provide each of their position coordinates and many other information relevant. So when compared to the cost and accessability of the other devices in the consumer market, Leap Motion device is affordable. Gloves
- To recognize writing interaction in the air, we found out several techniques in the research sphere. ………………………………………………….. And each of them have its own pros and cons. So if we take the analysis of our problem, it is basically for learning/practicing thing rather than solely recognize writing. There should be a learning outcome. Users need to follow/learn the correct standard of the letter. Recognition on one letter at a time. Userbase containing of learners rather than frequent writers. With this analysis information we chose a template matching approach. When compared to others it is easy to implement, no heavy processing needed and very much suitable for our conditions. And in the literature survey conducted, we identified the first three approaches are much appropriate for typical air writing recognizers.
- We’ve gone through several related works in the research sphere that uses previously mentioned consumer vision sensor devices, and recognition techniques. But they are focusing on developing a typical recognizer. But for us measuring the learning outcome is important. How much user get into the alphabetical character, its shape, how he follows the path. So template matching approach is ideal for that.
- So for the proposed approach, it has four phases………………………………………………. In the first phase user need to configure which handed user he is, the hand he use to perform writing interaction. For example if he is a right handed user, his left hand is taken as the operating hand.
- Operating hand in the sense, the hand to perform operations such as clear the canvas, checking the writing, etc. To trigger such kind of operations, static gestures with the operating hand is used. For the writing interaction in the prototype it provides two options. Either by the fingertip or by a tool tip, such as tip of a pencil or pen.
- In capturing the writing interaction, the problem we came across was, these vision input devices constantly streams the motion trajectories of the finger position. So it contained relevant motion coordinates as well as the irrelevant motion coordinates. So we need to indentify the actual motion coordinates which represent the character stroke. This is called as motion segmentation. The easiest way is to explicitly delimit the motion stroke and only the relevant data is captured. The effectiveness is depend on the user. But in the automatic detection approach, it is a complex and expensive technique, it captures all the data and through a process it identifies the garbage data. The effectiveness is depend on the process and the training. As this is a learning/practicing tool measuring users effectiveness, we went for the explicit delimitation. Key/Button press is the easiest one, holding a button when writing. But it breaks the mental model we’re going to establish, the natural interaction. For break by gesture, it goes along with the natural interaction but somewhat user training is needed on how to start a stroke and end a stroke. Virtual touch zone was the approach we selected.
- It is like this. Z axis goes like this and its 0th coordinates are in here. So we consider the area which is less than 0 as the touch zones. And we kept a little area infront for the hover zone to alert that user is moving towards touch zone. To alert we use three colors…………………………….. And blue if he’s not near to those zones.
- So it is exactly like a 2D virtual writing interface like in this picture.
- So for the recognition of the character written in the air, we use a template matching technique called as $P Point Cloud Framework. It contain 3 steps. 1st step is normalization of the captured trajectory, 2nd is measuring the distance, and its based on Euclidean distance approach. 3rd step is matching to find the character written.
- So first of all let me tell you what is the specialty of point cloud for gesture writing recognition. According to its founders it defines as 2D gesture recognizer, instance-based nearest-neighbor classifier with a Euclidean scoring function. $P uses simple geometric principles to match gestures
- Let me explain with an example. In the left side this is the template we provided for letter “R”. Actually it’s list of x and y coordinates, that we called as point cloud. For straight lines it only needed start and end coordinates. But curves need much detail. Each letter in the alphabet have a point cloud template stored. Before the matching process, all of them need to be normalized. 1st thing is resampling. So this normalized template it have exactly 52 coordinates whatever the size of the list provided. Next scale with shape preservation. Scale of 1 by 1 square preserving the shape. Next step is translation to origin. Centered the shape into (0,0) coordiantes. So each template is normalized prior to the matching.
- As well as the character written in the air.
- Now each point in the normalized air writing can be compared against the points in the normalized templates. search for the minimum matching distance (“goodness”) between C and Tt from all the possible alignments (also called as Assignment Problem
- greedy process need to run the algorithm multiple times with different starting points and return the minimum matching of all runs