Patent Landscape Report on Hand Gesture Recognition by PatSeer Pro

1. HAND GESTURE RECOGNITION PATENT LANDSCAPE REPORT BY PATSEER PRO

2. Page 2 of 24 © 2017 Gridlogics. All Rights Reserved. HAND GESTURE RECOGNITION - OVERVIEW Gesture recognition is the ability of a device to identify and respond to the different gestures of an individual. Most gesture recognition technology can be 2D-based or 3D-based, working with the help of a camera-enabled device, which is placed in front of the individual. The camera-enabled device beams an invisible infrared light on the individual, which is reflected back to the camera and onto a gesture recognition Integrated Chip (IC). Gestures can originate from any bodily motion or state but commonly originate from the face or hand. Current focuses in the field include emotion recognition from face and hand gesture recognition. Users can use simple gestures to control or interact with devices without physically touching them. Many approaches have been made using cameras and computer vision algorithms to interpret sign language. However, the identification and recognition of posture, gait, proxemics, and human behaviors is also the subject of gesture recognition techniques. Gesture recognition can be seen as a way for computers to begin to understand human body language, thus building a richer bridge between machines and humans than primitive text user interfaces or even GUIs (graphical user interfaces), which still limit the majority of input to keyboard and mouse. Touchless user interface is an emerging type of technology in relation to gesture control. Touchless user interface (TUI) is the process of commanding the computer via body motion and gestures without touching a keyboard, mouse, or screen. The ability to track a person's movements and determine what gestures they may be performing can be achieved through various tools. The kinetic user interfaces (KUIs) are an emerging type of user interfaces that allow users to interact with computing devices through the motion of objects and bodies. Examples of KUIs include tangible user interfaces and motion-aware games such as Wii and Microsoft's Kinect, and other interactive projects In order to interpret movements of the body, one has to classify them according to common properties and the message the movements may express. For example, in sign language each gesture represents a word or phrase. The taxonomy that seems very appropriate for Human- Computer Interaction has been proposed by Quek in “Toward a Vision-Based Hand Gesture Interface". He presents several interactive gesture systems in order to capture the whole space of the gestures:  Manipulative  Semaphoric  Conversational Some literature differentiates 2 different approaches in gesture recognition: a 3D model based and an appearance-based. The foremost method makes use of 3D information of key elements of the body parts in order to obtain several important parameters, like palm position or joint angles. On the other hand, Appearance-based systems use images or videos for direct interpretation.  3D model-based algorithms  Skeletal-based algorithms  Appearance-based models

3. Page 3 of 24 © 2017 Gridlogics. All Rights Reserved. PATENT SEARCH STRATEGY Using PatSeer, we searched individual publications and then collapsed them to one member per Family (Simple Families). Patent families are a group of one or more patent applications in multiple countries which represent the same invention. The publications included in the report are updated as of 5th September 2017. Summary of Search Results:  We started with a set of 5386 records  We then used deduplication as Simple Families to get 2953 Families  We browsed through some records to remove all the records which were irrelevant to the search topic  We flagged/rated each individual record using Search Recall™ option present in PatSeer  We then filtered, analyzed and came across some unwanted terms/Keywords  These records were then removed from the project by using the search within records option NOTE: All analysis in this report has been done on Simple Family (one member per family) and so the data in the charts should be construed accordingly.

4. Page 4 of 24 © 2017 Gridlogics. All Rights Reserved. TECHNICAL SEGMENTATION (PATENT CATEGORIZATION) To know more about the search strategy of this report and methodology used for patent categorization Contact us here:

5. Page 5 of 24 © 2017 Gridlogics. All Rights Reserved. FILING TREND The chart below shows number of filings for hand Gesture Recognition during the last 15 years. The chart was generated on overall records (5386 Records). Trend analysis based on filing of priority application indicates a gradual increase in number of applications being filed. Maximum number of patent applications (818) taking/having priority were filed during 2012. How we did it? Application trend chart was generated within Quick Stats on the overall record set. The generated chart was then converted to Earliest Priority Year.

6. Page 6 of 24 © 2017 Gridlogics. All Rights Reserved. TOP COMPANIES The chart below represents top companies active in Hand Gesture Recognition technology, with a single representation from each family. It is evident that Microsoft leads the patent count for Hand Gesture Recognition followed by Samsung and Intel.  The chart below shows assignee innovation time line for the top 15 Assignees active in hand gesture control  With the dots representing patent publications / filings and the blue lines indicating the timelines between the earliest and latest filings for different assignees, one can look into the relevance of filings for different assignees with respect to time  Intel appear to be one of the earliest assignees with filings for gesture recognition and one that innovators pursued the longest right till present date. Though there are long gaps in the timeline from the early days, the recent years have continued to see a lot of IP activity. Similarly, Audi has visibly shorter timeline with no new patents since 2015

7. Page 7 of 24 © 2017 Gridlogics. All Rights Reserved. How we did it? Once the patents were added to PatSeer Projects, top assignees and innovation time line was generated within the Quick Stats Tab. Note: Assignees used for all the charts below were generated with the help of custom fields, i.e. similar assignees with name variations were filtered, and then these were assigned to a particular custom field (assignee).This was repeated for the top assignees

8. Page 8 of 24 © 2017 Gridlogics. All Rights Reserved. RESEARCH ACTIVITY AROUND THE WORLD The below map represents the geographical filing relating to Gesture Recognition. The map helps provide an indication of where innovation in this area is originating. China is the leading country in this field with 1131 families followed by United States (919) and Korea (321). The strength of the coloring represents the proportion of patent applications. Country Total CN 1131 US 919 KR 321 JP 214 TW 208 How we did it? The map was generated using the Priority country tab under the Quick Stats option.

9. Page 9 of 24 © 2017 Gridlogics. All Rights Reserved. COMPANIES ACTIVITY ACROSS APPLICATIONS  The table below shows research activity of companies across different applications of hand gesture recognition  Samsung has the maximum number of records for gaming with approximately 50% of its records concentrating on gaming  Leap Motion which is primarily into gesture recognition focuses mainly on automobile and gaming applications How we did it? First records were classified under different application areas. These were then compared using the co- occurrence tool with the companies. The resulted matrix/table chart was then exported to a CSV format

10. Page 10 of 24 © 2017 Gridlogics. All Rights Reserved. COMPANIES ACROSS DIFFERENT TYPES SENSORS  The table below shows research activity of companies across different sensors used for hand gesture recognition  Samsung leads the records count for Proximity, infrared and capacitive sensors  Microsoft has research activity across all the type of sensors except speed sensor How we did it? First records were classified under different types of sensors used for hand gesture recognition, and then a co-occurrence matrix for assignees and sensors was generated using the co-occurrence tool. The resulted matrix was then converted to Row Heat Map and exported.

11. Page 11 of 24 © 2017 Gridlogics. All Rights Reserved. COMPANIES ACROSS DIFFERENT TYPES OF GESTURES  The table below shows research activity of companies across different types of gestures classified relating hand gesture recognition  Most of the top assignees namely Microsoft , Intel and Samsung are using dynamic gestures How we did it? First records were classified under different types of gestures and then using the co-occurrence tool a matrix was generated for the top assignees and types of gestures. The resulted matrix was then converted to a Tree Map and exported as an image.

12. Page 12 of 24 © 2017 Gridlogics. All Rights Reserved. COMPANIES ACROSS DIFFERENT INPUT DEVICES  The table below shows research activity of companies across different types input devices relating hand gesture recognition  Intel and Microsoft are more into Stereoscopic and depth cameras  Hyundai motors has research activity for only stereoscopic and infrared camera which is relative to its applications in the automobile sector How we did it? First records were classified under different types of input devices used for hand gesture control; matrix was then generated for the top assignees and different input devices using the co-occurrence tool. The resulted matrix was then converted to a Column Chart and exported as an image.

13. Page 13 of 24 © 2017 Gridlogics. All Rights Reserved. COMPANIES ACROSS GESTURE CLASSIFICATION ALGORITHMS  The table below shows research activity of companies across different classification algorithms used for hand gesture recognition  Continuous Hidden Markov algorithm is the most commonly used algorithm  Qualcomm uses all the algorithms except conditional random field for its applications relation to gesture control How we did it? First records were classified under different classification algorithms used for hand gesture recognition, and then a co-occurrence matrix for assignees and those algorithms was generated using the co- occurrence tool. The resulted matrix was then converted to Column Heat Map and exported.

14. Page 14 of 24 © 2017 Gridlogics. All Rights Reserved. APPLICATIONS ACROSS DIFFERENT TYPES OF SENSORS  The Heat Map below shows the different types of sensors used across various application of hand gesture recognition  Gaming require detection of vibration, acceleration, inclination and orientation which is possible with the help of accelerometer sensors; which are widely used for gaming applications  Accelerometer and infrared sensors are widely used for home automation How we did it? First records were classified under different application areas and various sensors and then a co- occurrence matrix for the same was generated using the co-occurrence tool. The resulted matrix was then converted to Heat Map and exported.

15. Page 15 of 24 © 2017 Gridlogics. All Rights Reserved. APPLICATIONS ACROSS INPUT DEVICES  The Heat Map below shows the different types of input devices used across various application of hand gesture recognition  Stereoscopic cameras and infrared cameras are widely used across all the application areas  Depth Cameras are used in applications like gaming, automobile and television control to detect the relative position of users from the device. How we did it? First records were classified under different application areas and input devices and then a co-occurrence matrix for the same was generated using the co-occurrence tool. The resulted matrix was then converted to Column Chart and exported as image.

16. Page 16 of 24 © 2017 Gridlogics. All Rights Reserved. APPLICATIONS ACROSS GESTURE CLASSIFICATION ALOGRITHM  The chart below shows the different Cryptographic methods used for various applications  Hidden Markov and Support Vector Machine(SVM) algorithms are widely used for all the applications  Finite State Machine (FSM) have been used for many years in video games to model the AI of Non Playing Characters; the chart below confirms this fact How we did it? First records were classified under different application areas and classification algorithms and then a co- occurrence matrix for the same was generated using the co-occurrence tool. The resulted matrix was then converted to a column chart and exported as an image.

17. Page 17 of 24 © 2017 Gridlogics. All Rights Reserved. TECHNOLOGY LANDSCAPE FOR HAND GESTURE RECOGNITION The landscape map below represents key concepts for different companies generated across title, abstract and claims. Themes are collection of prominent topics extracted from the patent data and grouped under relevant parent tags. Clusters for Touch screen and virtual reality touch screen are close to each other as there is high degree of relevance between the records present in those themes. How we did it? The VizMAP tool in PatSeer Pro was used for this analysis. First the clusters were generated on title, abstract and claims using the Advanced Clustering option. Once the clusters were generated these were loaded using the Landscape mode and then the themes were colored by assignee.

18. Page 18 of 24 © 2017 Gridlogics. All Rights Reserved. CITATIONS ANALYSIS FOR LEAP MOTION Leap Motion, Inc. is an American company that manufactures and markets a computer hardware sensor device that supports hand and finger motions as input, analogous to a mouse, but requires no hand contact or touching. In 2016, the company released new software designed for hand tracking in virtual reality. Leap Motion is a hardware sensor device that tracks hand and finger motions and translates it into input. The Leap Motion controller is a small USB powered device compatible with Windows and Macintosh that uses two monochromatic IR cameras and three IR LEDs to track movements and motion made by hands and fingers in a roughly 1m hemispherical 3D space. The cameras reportedly generate 300 frames per second of data, which is analyzed and interpreted by proprietary software The filings by Leap Motion relate to 3Dimensional space, infrared cameras mapping hand gestures to wearable devices and computer readable storage mediums. Many multinationals (patenting authorities) are referring/citing these patents to evolve the technology further. The forward citation chart below reiterates the fact. Companies like Google, Microsoft, Acer, Samsung, Sony etc. all are citing the Leap Motion patents. The following chart shows the forward citations for Leap Motion

19. Page 19 of 24 © 2017 Gridlogics. All Rights Reserved. The chart below shows the backward citations for Leap Motion How we did it? For citation analysis, first a group was created for all the records of Leap Motion. The created group was selected and analyzed using the Citation Analysis option.

20. Page 20 of 24 © 2017 Gridlogics. All Rights Reserved. KEY COMPANY ANALYSIS The key companies considered for analysis are: 1. Leap Motion 2. Amazon 3. Oblong 4. Pixart 5. Nokia INNOVATION TIMELINE  The chart below shows the innovation timeline representing patenting activity for hand gesture recognition  Leap Motion had its first filing in the year 2014 and has shown consistent growth in the subsequent years with maximum filings in 2015 How we did it? Once the patents were added to PatSeer Projects, key assignees were filtered from the apply filters option and innovation time line was generated within the Quick Stats Tab.

21. Page 21 of 24 © 2017 Gridlogics. All Rights Reserved. INNOVATIONS ACROSS DIFFERENT SEGMENTS OF GESTURE RECOGNITION  The chart below shows number of patents held by key companies across various technology segments and applications of hand gesture recognition How we did it? First key companies in Hand Gesture Recognition were identified by online review and these were analyzed with respect to the different technology segmentations for gesture recognition. The resulting matrix was the converted to a bar chart and the chart was exported as an image.

22. Page 22 of 24 © 2017 Gridlogics. All Rights Reserved. SUMMARY OF FILINGS BY KEY COMPANIES  The chart below shows summary report for key companies with respect to the tech domain, Publication country and the CPC classes they are filing in How we did it? First key companies in hand gesture recognition were identified by online review and these were filtered from the Apply Filter option present. Once this was done then Summary Report option was selected. Then under “generate report for” option companies (under custom fields) option were selected. Parameters required for analysis were selected from the option “Patent Fields” and the resulting chart was generated and then exported as an image.

23. Page 23 of 24 © 2017 Gridlogics. All Rights Reserved. SUMMARY This report analyzes research trends of hand gesture recognition with a focus on Types of Gestures, Input Devices, Sensors, Algorithms and Applications and also highlights the key companies involved in this space. Gesture is a symbol of physical behavior or emotional expression. It includes body gesture and hand gesture. It falls into two categories namely static gesture and dynamic gesture. For the former, the posture of the body or the gesture of the hand denotes a sign. For the latter, the movement of the body or the hand conveys some messages. Gesture can be used as a tool of communication between computer and human. It is greatly different from the traditional hardware based methods and can accomplish human-computer interaction through gesture recognition. Gesture recognition determines the user intent through the recognition of the gesture or movement of the body or body parts. In the past decades, many researchers have strived to improve the hand gesture recognition technology Gestures involving hand act as an interface between a computer/system and the users. It works in a way that Instead of typing with keys or tapping on a touch screen, a motion sensor perceives and interprets movements as the primary source of data input. This is what happens between the time a gesture is made and the computer reacts. A camera feeds image data into a sensing device that is connected to a computer. Kinect is Microsoft's motion sensor add-on for the Xbox 360 gaming console. The device provides a natural user interface (NUI) that allows users to interact intuitively and without any intermediary device, such as a controller. The evolution of gesture technology can be useful to physically challenge and can improve immersive gaming technology. To get this project shared with you via PatSeer’s Patent Dashlet and to obtain access of dataset behind this report, Contact us here: Source: https://giphy.com/gifs/ Double click on the image to see hand gesture recognition technology example live in action.

24. Page 24 of 24 © 2017 Gridlogics. All Rights Reserved. ABOUT PATSEER PRO PatSeer Pro is a powerful and flexible web-based Patent Search, Analytics and Landscaping Platform. It includes a comprehensive set of analytical tools needed for precise technology landscaping and competitive intelligence projects such as Assignee cleanup tools, Co-occurrence Matrices, configurable Chart Dashboards, multi-generation Citation Analysis, Text Clustering and VizMAP – a spatial exploration and contour mapping visualization engine. PatSeer Pro is developed by Gridlogics, a leading provider of products and custom software solutions for patent research, management, data analysis and project management. For more information: Visit us at: www.patseer.com/pro- edition/  Create and configure dashboards as per your business charting requirements  A large set of powerful visualizations  Share dashboards across the organization  Render multiple visualizations in a single view  Add interactive filters to your dashboard  Collaborative decision making experience via sharable dashboards  Advanced 2D spatial visualizations for semantic exploration  Fully HTML5 compliant SaaS that works across all browsers  Powerful Customization Options  Normalize Assignees as per your business context  Competitor Trend Analysis  Natural Language / Similarity Searching  Out-licensing Research beyond citations  Powerful web based co- occurrence matrix with integrated real-time filters  Generate heat-maps on the fly  A variety of Matrix Export options  Leverage custom data points (Custom Fields, Categories) in Matrix  Automated Cleanup and Grouping using Fuzzy, Regex, Thesaurus Matching  Clustering / Topic Identification  One Click Summary Reports  Multi-generation Citation Analyzer  Thesaurus Creation/Edit and taxonomy management  Analyze networks of relationships between Companies, inventors and technologies using VizMAP  Contour Maps that highlights the peaks and troughs of the technology landscape  Co-Citation and Self/Non-Self Citation Analysis  Map the landscape around your hierarchical categories (buckets)  Powerful slide-dice options giving full-flexibility to the analyst  Fast response times even for large set of records  Multi-generation Citation Analyzer  Thesaurus Creation/Edit and taxonomy management One-stop platform for Technology Mining, White Space Analysis, Tech/Economic Forecasting and Competitive Intelligence Powerful and flexible analytics tools required for precision analysis Seamless collaboration and workflow capabilities allow for creating and sharing dashboards and analytical outputs Integrated Global Patent Content of PatSeer Database PatSeer Pro PATSEER PRO CAPABILITIES AT-A-GLANCE…