Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0

Share

Download to read offline

Development of a lossy online mouse tracking method for capturing user interaction with web content.

Download to read offline

Video: https://youtu.be/GfuAGH3EGE8

Dissertation: https://www.publish0x.com/fajar-purnama-academics/development-of-a-lossy-online-mouse-tracking-method-for-capt-xjjlwxd?a=4oeEw0Yb0B&tid=slideshare

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Development of a lossy online mouse tracking method for capturing user interaction with web content.

  1. 1. 1/66 Doctoral Thesis Open Defense Development of a Lossy Online Mouse Tracking Method for Capturing User Interaction with Web Browser Content Fajar Purnama D3 1 1 Human Interface Cyber Communication Laboratory (Usagawa Lab), Computer Science and Electrical Engineering, Kumamoto University fajar@hicc.cs.kumamoto-u.ac.jp July 28, 2020
  2. 2. 2/66 Doctoral Thesis Open Defense Chapter 1 Introduction Chapter 1 Introduction
  3. 3. 3/66 Doctoral Thesis Open Defense Chapter 1 Introduction Background Due to Covid-19 pandemic, all activities are online.
  4. 4. 4/66 Doctoral Thesis Open Defense Chapter 1 Introduction Problem Problem 1: How to monitor/follow these activities online?
  5. 5. 5/66 Doctoral Thesis Open Defense Chapter 1 Introduction Problem Can conventional web logs / educational data only monitor complicated online activities? Answer: Cannot
  6. 6. 6/66 Doctoral Thesis Open Defense Chapter 1 Introduction Problem Human interface that can generate deeper logs? (Solution) conventional web logs / educational data +
  7. 7. 7/66 Doctoral Thesis Open Defense Chapter 1 Introduction Problem Why eye tracking cannot be implemented today? Although the best monitoring quality is to use every logging and tracking there is, eye tracking cannot be implemented today. Eye tracking is inconvenient: Intrusive: see previous slide where an additional goggle is necessary. Expensive: JPY 10,000 - 1,000,000. Mouse tracking is convenient: No additional hardware: in default available in every computer. Non-intrusive: users will not be aware that mouse tracking is active.
  8. 8. 8/66 Doctoral Thesis Open Defense Chapter 1 Introduction Problem Problem 2: mouse tracking applications mentioned in academia does not suit public implementation. Offline Mouse Tracking Application in Laboratories: (1) Must be installed in each client and cursor position fixed to desktop. (2) Data are stored in each client and must be retrieved individually. Solution: Online Mouse Tracking Client Application: (1) Must be installed in each client but (2) data are sent to server. Or Online Mouse Tracking Server Application: (1) Browser only / no additional installation and (2) data are sent to server.
  9. 9. 9/66 Doctoral Thesis Open Defense Chapter 1 Introduction Problem Problem 3: Mouse Tracking Generates Big Data Rumors Volume: a swipe from left to right can produced 1 MB of data. Velocity: large data generated per second (data rate). Veracity: transmission may result in data loss due to limited connectivity. Variety: many types of mouse and other events. Value: the data are valuable. Rumors are not widely investigated: if mouse tracking generates huge data, how huge?.
  10. 10. 10/66 Doctoral Thesis Open Defense Chapter 1 Introduction Objective Objective: build application and implement. Write an application that can capture mouse and other browser human interfaces called mouse tracking. Make the mouse tracking application online that can be implemented in public. Investigate its data generation and resource usage. Reduce its data generation and resource usage.
  11. 11. 11/66 Doctoral Thesis Open Defense Chapter 1 Introduction Objective Objective: dig deeper into users’ online activities.
  12. 12. 12/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Chapter 2 New Online Mouse Tracking Based on publication list, the publication that is related to this chapter is: 1* Implementation of real-time online mouse tracking on overseas quiz session From server administrator point of view, Educational and Information Technologies, Online First, Springer, March 2020.
  13. 13. 13/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking System Overview
  14. 14. 14/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Mouse Tracking Desktop Recording: clicks, movements, scrolls, zooms, keylogs, copies, and other events.
  15. 15. 15/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Touch Tracking Mobile https://hicc.cs.kumamoto-u.ac.jp/ fajar/doctoral/mouse-tracking-keyboard- mouse-scroll-zoom-demo-and-mobile.html
  16. 16. 16/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Online Mouse Tracking System
  17. 17. 17/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Privacy Policy
  18. 18. 18/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Implementation Implementation
  19. 19. 19/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Implementation Client Application: Tracks All Browser’s Activities https://github.com/0fajarpurnama0/Real-Time-Online-Mouse-Tracking-
  20. 20. 20/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Implementation Server Application: Tracks Only Website’s Activities https://md.hicc.cs.kumamoto-u.ac.jp
  21. 21. 21/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Implementation Moodle Plugin Install https://github.com/0fajarpurnama0/Real-Time-Online-Mouse-Tracking-
  22. 22. 22/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Implementation Mouse Tracking Quiz Session Between Mongolia and Japan
  23. 23. 23/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Investigation Investigation
  24. 24. 24/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Investigation Obtained Mouse Tracking Data Data available: Mendeley repository titled ’Data for: Implementation of Real-Time Online Mouse Tracking Case Study in a Small Online Quiz’.
  25. 25. 25/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Investigation Rumor and Proof of Huge Data Generation Log File Duration Students Size Daily Pageview City Archive 2 month - 13 kB Moodle Log and Grades 3h 30min 41 191 kB Moodle Log Myanmmar 6 month 41 5 MB Mouse Tracking Mongolia 3h 30min 41 122 MB Mouse Tracking Afganistan 3h 42 114 MB Nasa Server Log 1995 23 days - 153 MB Open University Learning Analytics 1 year 32593 442 MB HarvardX Person-Course 2013 1 year 301609 33.8 MB
  26. 26. 26/66 Doctoral Thesis Open Defense Chapter 2 Online Mouse Tracking Investigation Students’ Events and Estimated Data Amount Generated
  27. 27. 27/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Chapter 3 Resource Saving Methods Based on publication list, the publication that are related to this chapter are: 1* Implementation of real-time online mouse tracking on overseas quiz session From server administrator point of view, Educational and Information Technologies, Online First, Springer, March 2020. 2* Using Realtime Online Mouse Tracking Based on Demand Pre-processing for Lower Storage and Transmission Cost, Journal of Big Data, Vol.7, no.27, pp.1-22, Springer, April, 2020. 3* Demonstration on Extending The Pageview Feature to Page Section Based: Towards Identifying Reading Patterns of Users, ICSE, November, 2016. .
  28. 28. 28/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Existing Solutions Redundant data reduction for example: Long URL: https://www.publish0x.com/0fajarpurnama0/can-i-share-my- work-after-copyright-transfer-xroyrle?a=4oeEw0Yb0B&tid=twitter Into Short URL: https://t.co/Hl62T1llTV Sampling rate (event / second) reduction for example delay capture interval for 50 milliseconds = 20 max events per second. Adaptive sampling: do not record during idle activities (eye tracking record during idle activities). Compression methods.
  29. 29. 29/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Not Realtime vs Realtime in Mouse Tracking Traditionally: accumulate > wait until session ends > send all. Realtime: send events immediately on occurrence.
  30. 30. 30/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Lossy Based on Demand Coordinate Compression Default: full mouse tracking log. ROI: summarize coordinates into areas. Summarized: summarize coordinates into amounts. Events Summarized ROI Default Amounts Areas X Coordinates X X
  31. 31. 31/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Default Mouse Tracking Data Transmission Send data immediately / real time on each coordinate passed:
  32. 32. 32/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Region of Interest Mouse Tracking Data Transmission Send data after leaving areas:
  33. 33. 33/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Summarized Event Amounts of Mouse Tracking Send data after closing webpage:
  34. 34. 34/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Three Mouse Tracking Simulation
  35. 35. 35/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Storage Reduction Type Queries Data Size Default Mouse Tracking 286510 ∼100 MB ROI Mouse Tracking 1 28048 ∼7.7 MB ROI Mouse Tracking 2 19061 ∼5.3 MB ROI Mouse Tracking 3 17880 ∼4.9 MB Summarized Event Amount 23 ∼16 kB
  36. 36. 36/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods Data Rate: Reduced > 50% storage and transmission
  37. 37. 37/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods CPU Usage: RP3 cannot handle the default MS
  38. 38. 38/66 Doctoral Thesis Open Defense Chapter 3 Resource Saving Methods RAM Usage: abnormally high for default mouse tracking
  39. 39. 39/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Chapter 4 The depth Levels of Logs Concept Based on form 3, the publication that are related to this chapter are: 1* Implementation of real-time online mouse tracking on overseas quiz session From server administrator point of view, Educational and Information Technologies, Online First, Springer, March 2020. 2* Using Realtime Online Mouse Tracking Based on Demand Pre-processing for Lower Storage and Transmission Cost, Journal of Big Data, Vol.7, no.27, pp.1-22, Springer, April, 2020. 3* Demonstration on Extending The Pageview Feature to Page Section Based: Towards Identifying Reading Patterns of Users, ICSE, November, 2016. .
  40. 40. 40/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Conceptual Explaination: Log depth Level Concept
  41. 41. 41/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Web Page / Course Content Level Log Web Page / Course Content Level Log
  42. 42. 42/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Web Page / Course Content Level Log Mouse tracking is able to identify students leaving quiz page (address problem 1: potential cheating).
  43. 43. 43/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Web Page / Course Content Level Log Online Examination Leave Detector (Possible Software) https://hicc.cs.kumamoto-u.ac.jp/ fajar/doctoral/mouse-tracking/exam-leave- detect-demo.html
  44. 44. 44/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Area / Section Level Log Area / Section Level Log
  45. 45. 45/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Area / Section Level Log Area Heatmaps are Able to Address Problem 1: Required Reading Passage (Proof of Read)
  46. 46. 46/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Area / Section Level Log Area Heatmaps in Detail
  47. 47. 47/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Area / Section Level Log Terms of Services Read Detector (Possible Software) https://hicc.cs.kumamoto-u.ac.jp/ fajar/doctoral/mouse-tracking/eproof-of- read.html
  48. 48. 48/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Coordinate / Point Level Log Coordinate / point level log
  49. 49. 49/66 Doctoral Thesis Open Defense Chapter 4 The depth Levels of Logs Concept Coordinate / Point Level Log Clicks, Left: Triangle, Middle: Square, Right: Pentagon.
  50. 50. 50/66 Doctoral Thesis Open Defense conclusion and Future Work Conclusion Demonstrated mouse tracking as solution to problem 1 regarding to monitoring demand for human-online interaction.
  51. 51. 51/66 Doctoral Thesis Open Defense conclusion and Future Work Conclusion Answered problem 2 regarding public mouse tracking application by building new real-time online mouse tracking application that does not require additional application for the client and implemented.
  52. 52. 52/66 Doctoral Thesis Open Defense conclusion and Future Work Conclusion Answered problem 3 regarding huge resource rumors by investigating real-time online mouse tracking resource usage and reducing it over 50% by summarizing coordinates into areas.
  53. 53. 53/66 Doctoral Thesis Open Defense conclusion and Future Work Thank you
  54. 54. 54/66 Doctoral Thesis Open Defense Supplementary Materials Almost All Online, Anywhere at Anytime, Face to Face
  55. 55. 55/66 Doctoral Thesis Open Defense Supplementary Materials Benefit and Significance Mouse tracking is one of the missing keys of anything fully online. The code is open source that can freely implemented and developed by other parties. The data and resource usage investigation result can help parties to plan in implementing mouse tracking. Reduced data and resource usage can help those with limited connectivity.
  56. 56. 56/66 Doctoral Thesis Open Defense Supplementary Materials Obtained Moodle Log Data
  57. 57. 57/66 Doctoral Thesis Open Defense Supplementary Materials Obtained Grades
  58. 58. 58/66 Doctoral Thesis Open Defense Supplementary Materials Truth: Users Actually Have Full Control but Unaware
  59. 59. 59/66 Doctoral Thesis Open Defense Supplementary Materials Rumor and Proof of Huge Data Generation Moodle Log Myanmar: (CP: monmon@hicc.cs.kumamoto-u.ac.jp) 6 months of entire course with 40 students. 21098 rows of table, 5MB. Can a swipe generate a megabyte of data? Yes, if the width of a webpage from left to right is 1000 pixels, then a swipe from left to right can generate at large 1000 rows of table. Mouse Tracking Session Mongolia: 2 hours of quiz session with 22 students each hour. 286511 rows of table, 100 MB. Mouse Tracking Session Afganistan: 2 hours of quiz session with 23 students each hour. 282601 rows of table, 92 MB. Problem: Huge Storage and Transmission Cost
  60. 60. 60/66 Doctoral Thesis Open Defense Supplementary Materials Server Storage Side: Query Amount and Data Size Default Mouse Tracking: 100 MB ID Name Date 2019/3/01 Duration Left Click Right Click Middle Click Mouse X Mouse Y Scroll X Scroll Y 1 Student 1 11:06:39 13.674 false false false 0 0 0 0 2 Student 1 11:06:39 0.002 false false false 1197 317 0 0 ... ... ... ... ... ... ... ... ... ... ... 286511 Student 23 14:44:09 0.002 false false false 1009 179 0 0 ROI Mouse Tracking: 5.4 MB ID Name Date 2019/3/01 Duration (second) Area (x1,x2,y1,y2) Left Clicks Right Clicks Middle Clicks Mouse Moves Scrolls 1 Student 1 11:06:39 14.148 {"header":[0,1920,0,64]} 0 0 0 1 1 2 Student 1 11:06:40 1.179 {"quiz1":[529,1900,291,570]} 0 0 0 86 0 ... ... ... ... ... ... ... ... ... ... 19062 Student 23 14:44:09 0.002 {"title":[16,1904,150,270]} 0 0 0 1 0 Whole Page Summarized Mouse Tracking: 16 kB Name Date 2019/3/01 Duration (seconds) Left Clicks Right Clicks Middle Clicks Mouse Moves Scrolls Student 1 14:12:29 41 3 0 0 629 114 Student 2 14:44:09 90 7 0 0 1176 137 ... ... ... ... ... ... ... ... Student 22 11:55:14 2188 157 5 0 20912 6626 Student 23 11:57:37 2236 323 0 0 17982 6930
  61. 61. 61/66 Doctoral Thesis Open Defense Supplementary Materials Client Side: Browser Speed
  62. 62. 62/66 Doctoral Thesis Open Defense Supplementary Materials Amount of mouse event logs Most analyzers only mined up to this level logs: page view, quiz, grades, and other activity attempts. Even in this level, there are logs that most analyzers are not aware of: Amounts of events: name duration leftclick rightclick middleclick mousemove scroll highlight average 1368.16 175.78 0.29 7.98 11004.73 4158.12 0.61 total 56094.53 7207.00 12.00 327.00 451194.00 170483.00 25.00 minimum 41.26 3.00 0.00 0.00 633.00 116.00 0.00 maximum 2748.21 1179.00 3.00 179.00 20724.00 9521.00 4.00 stdev 728.82 295.33 0.72 39.05 4977.88 2291.82 1.16
  63. 63. 63/66 Doctoral Thesis Open Defense Supplementary Materials heatmap 10 minutes interval
  64. 64. 64/66 Doctoral Thesis Open Defense Supplementary Materials Grade heatmap
  65. 65. 65/66 Doctoral Thesis Open Defense Supplementary Materials Why RP3? To Facilitate Those With Limited Connectivity Hand Carry Data Collecting Through Questionnaire and Quiz Alike Using Mini-computer Raspberry Pi, IMLF Proceeding, June, 2017.
  66. 66. 66/66 Doctoral Thesis Open Defense Supplementary Materials Regular Synchronization is Necessary From 6 Incremental Synchronization Implementation on Survey using Hand Carry Server Raspberry Pi, IEICE Tech Report, vol.117, no.65, pp.21-24, June, 2017. and 7* Rsync and Rdiff implementation on Moodle’s backup and restore feature for course synchronization over the network, (TENSYMP), pp.24-29, IEEE, Mei, 2016.

Video: https://youtu.be/GfuAGH3EGE8 Dissertation: https://www.publish0x.com/fajar-purnama-academics/development-of-a-lossy-online-mouse-tracking-method-for-capt-xjjlwxd?a=4oeEw0Yb0B&tid=slideshare

Views

Total views

414

On Slideshare

0

From embeds

0

Number of embeds

323

Actions

Downloads

1

Shares

0

Comments

0

Likes

0

×