Real time emotion recognition during presentations - Artemii Frolov
•
0 likes•201 views
This document discusses real-time emotion recognition during presentations using machine learning. It outlines using Amazon's DeepLens camera to detect faces in real-time, recognize emotions using Amazon Rekognition and a mini-Xception model, and visualize results. Challenges with DeepLens like limited processing power and setup issues are discussed. Potential extensions like head pose detection and mobile deployment are mentioned.
Recommended
More Related Content
Similar to Real time emotion recognition during presentations - Artemii Frolov
Similar to Real time emotion recognition during presentations - Artemii Frolov (20)
Recently uploaded
Recently uploaded (20)
Real time emotion recognition during presentations - Artemii Frolov
- 1. REAL-TIME EMOTION RECOGNITION DURING PRESENTATION 17 October 2019
- 2. 2 Jogging may be boring RUNKEEPER FOR PRESENTATIONS
- 3. Jogging with Runkeeper is much more interesting 3 Jogging may be boring RUNKEEPER FOR PRESENTATIONS
- 4. Presentations may be boring 4 Jogging may be boring RUNKEEPER FOR PRESENTATIONS Jogging with Runkeeper is much more interesting
- 5. Presentations may be boring How about presentations with Runkeeper?5 Jogging may be boring RUNKEEPER FOR PRESENTATIONS Jogging with Runkeeper is much more interesting
- 6. • Task description: what are we doing and what do we need • Face detection: Amazon solution, MTCNN and collecting data • Emotion recognition: Amazon Rekogniton and mini-Xception • DeepLens: working pipeline and what’s wrong with it • Realization: splitting tasks and watchdogs • Results visualization: Flask and Bokeh • Data analysis: what one can get from results • Plans: what else I can do AGENDA Machine learning part Engineering part Data part
- 7. 7 HOW CAN WE EVALUATE PRESENTATION? 1. Record a video of presenter during presentation and evaluate/measure performance later 2. Recognise pose and gesture of presenter during presentation with machine learning 3. Take information from audience, not from presenter
- 8. 8 1. Photos are uploaded in the cloud 2. Photos are stored locally 3. Personal information WHAT WERE THE MAIN PROBLEMS?
- 9. 9 1. No photos are uploaded in the cloud 2. No photos are stored locally 3. Only aggregated information PRIVACY CONCERN
- 10. 10 AMAZON DEEPLENS HD camera with GPU and CPU inside
- 11. FACE DETECTION ALGORITHM 11 WHAT DO WE NEED?
- 12. FACE DETECTION ALGORITHM EMOTION RECOGNITION ALGORITHM 12 WHAT DO WE NEED?
- 13. • Preconfigured project in Amazon library 13 FACE DETECTION Amazon solution
- 14. • Preconfigured project in Amazon library • MXNet model • 300*300 input reshape 14 FACE DETECTION Amazon solution
- 15. • Preconfigured project in Amazon library • MXNet model • 300*300 input reshape • 4-5 faces maximum 15 FACE DETECTION Amazon solution
- 16. • Preconfigured project in Amazon library • MXNet model • 300*300 input reshape • 4-5 faces maximum 16 FACE DETECTION Amazon solution CAN WE DO SOMETHING?
- 17. 17 CROWD DETECTION PROBLEM Resize -> Window -> Resize back
- 18. 18 CROWD DETECTION PROBLEM No resize -> Frame -> Face
- 19. 19 CROWD DETECTION PROBLEM Resize or not resize?
- 20. 20 CROWD DETECTION PROBLEM Why not both?
- 21. 1. Building “Image pyramid” 2. Proposal Network (P-Net) 3. Refinement Network (R-Net) 4. Output Network (O-Net) + Non Maximum Supression 21 MTCNN… ..to the rescue! https://arxiv.org/ftp/arxiv/papers/1604/1604.02878.pdf
- 22. 22 MTCNN RESULTS 1 face Image size Total pixels Process time FPS Image size Total pixels Process time FPS 460x259 119,140 0.118 seconds 8.5 474x224 106,176 0.185 seconds 5.4 561x561 314,721 0.227 seconds 4.5 736x348 256,128 0.290 seconds 3.4 667x1000 667,000 0.456 seconds 2.2 2100x994 2,087,400 1.286 seconds 0.7 1920x1200 2,304,000 1.093 seconds 0.9 4799x3599 17,271,601 8.798 seconds 0.1 10 faces
- 23. 23 MTCNN RESULTS 3872 x 2592
- 24. 24 MTCNN RESULTS 3872 x 2592
- 25. 25 MTCNN RESULTS 1280 x 1040
- 26. 26 MTCNN RESULTS 1280 x 1040 300 x 300
- 27. 27 AMAZON RECOGNITION API Test launch Crop faces Recognize emotions • SAD: 0.01 • HAPPY: 0.34 • CONFUSED: 0.14 • DISGUSTED: 0.05 • CALM: 0.22 • ANGRY: 0.11 • SURPRISED: 0.05
- 28. 28 AMAZON REKOGNITION Test launch Crop faces Recognize emotions • SAD: 0.01 • HAPPY: 0.34 • CONFUSED: 0.14 • DISGUSTED: 0.05 • CALM: 0.22 • ANGRY: 0.11 • SURPRISED: 0.05 What can be wrong here?
- 30. 30 • Privacy reasons • Time consumption PROBLEMS
- 31. 31 • Privacy reasons • Time consumption • Calm bias PROBLEMS
- 32. 32 FER+ DATASET Even better than FER
- 33. 33 MINI-XCEPTION… Fast and small https://arxiv.org/pdf/1710.07557.pdf Resnet50: ~25 million weights Inception: ~56 million weights Xception: ~23 million weights Mini-Xception: ~60k weights Munich -> Airport ~ 40 km Length of equator ~40k km
- 34. 34 MINI-XCEPTION… Fast and small https://arxiv.org/pdf/1710.07557.pdf Resnet50: ~25 million weights Inception: ~56 million weights Xception: ~23 million weights Mini-Xception: ~60k weights
- 35. 35 DEPTHWISE SEPARABLE CONVOLUTION https://towardsdatascience.com/review-xception-with-depthwise-separable-convolution-better-than-inception-v3-image-dc967dd42568 Why is it so fast? Regular convolution: Input: M*M*C1 Output: (M-K)*(M-K)*C2 Weights: C1*C2*K*K Separable convolution: Input: M*M*C1 Output: M*M*C2 Weights: C1*C2*1*1 + C2*K*K Example: Input: 100*100*16 Output: 97*97*32 Kernel: 3*3 Regular: 16*32*3*3 = 4608 Separable: 16*32*1*1 + 32*3*3 = 288
- 36. 36 MINI-XCEPTION Results COMPARISON WITH AMAZON REKOGNITION
- 37. 37 MINI-XCEPTION Results COMPARISON WITH AMAZON REKOGNITION LOSS AND VAL-LOSS
- 38. ONE MODEL AT A TIME Fix: save the second model in device memory 38 CHALLENGES WITH DEEPLENS Really, why?
- 39. ONE MODEL AT A TIME Fix: save the second model in device memory Fix: check in a code if there is a folder -> create this folder via ssh 39 CAN’T TURN ON / TURN OFF CHALLENGES WITH DEEPLENS Really, why?
- 40. ONE MODEL AT A TIME Fix: save the second model in device memory Fix: check in a code if there is a folder -> create this folder via ssh Fix: improvise, adapt, overcome 40 CAN’T TURN ON / TURN OFF CAN’T STACK OVERFLOWING CHALLENGES WITH DEEPLENS Really, why?
- 41. ONE MODEL AT A TIME Fix: save the second model in device memory Fix: check in a code if there is a folder -> create this folder via ssh Fix: improvise, adapt, overcome 41 CAN’T TURN ON / TURN OFF CAN’T STACK OVERFLOWING SETUP (INITIAL AND EVERYTIME WIFI) Fix: suffer CHALLENGES WITH DEEPLENS Really, why?
- 42. ONE MODEL AT A TIME Fix: save the second model in device memory Fix: check in a code if there is a folder -> create this folder via ssh Fix: improvise, adapt, overcome 42 CAN’T TURN ON / TURN OFF CAN’T STACK OVERFLOWING SETUP (INITIAL AND EVERYTIME WIFI) Fix: suffer PYTHON3 IS NOT SUPPORTED Fix: micro-architecture CHALLENGES WITH DEEPLENS Really, why?
- 43. ALGORITHM 1 Send result to step 2 Send result to step 3 Send results to output Send confirmation to step 1 Do something 43 MICRO-ARCHITECTURE APPROACH ALGORITHM 2 Do something ALGORITHM 3 Do something
- 44. 44 WATCHDOG Inotify and threads New file created Function
- 45. • Four independent modules • Unified watchdog algorithm • Easy to modify 45 REALIZATION MTCNN Mini Xception Photo Visualization
- 46. • Running on device • Can be reach by Wi-Fi from laptops and mobile devices • Update graphics every 3 seconds on-the-go via Ajax 46 ONLINE VISUALIZATION Flask and Bokeh GIF
- 49. 49 AFTERWARD DATA ANALYSIS People interested
- 50. • Headpose detection • Runnable on a mobile device • Psychology research 50 POSSIBLE EXTENSIONS
- 51. Data Reply: Dmitrii Azarnykh Wadim Pessin Gavin Perrie 51 THANKS TUM: Prof. Dr. Christoph Ungemach
- 52. BTW, I AM LOOKING FOR А THESIS Artemii Frolov arfr@live.ru https://www.linkedin.com/in/artemii-frolov/
- 53. THANK YOU