Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Using Crowdsourcing, Automated Methods and Google Street View to Collect Sidewalk Accessibility Data

887 views

Published on

In this presentation, I describe a system that uses crowdsourcing, computer vision, machine learning, and Google Street View to collect sidewalk accessibility data.

Published in: Technology
  • Be the first to comment

Using Crowdsourcing, Automated Methods and Google Street View to Collect Sidewalk Accessibility Data

  1. 1. makeability lab クラウドソーシング・コンピュータビジョン・ ストリートビューを用いた歩道の アクセシビリティデータの収集手法 原航太郎 | Project Sidewalk (PI: Jon E. Froehlich)
  2. 2. A B C
  3. 3. D A B C
  4. 4. Human-Computer Interaction Lab
  5. 5. Characterizing Sidewalk Accessibility at Scale using Google Street View, Crowdsourcing, and Automated Methods Kotaro Hara | Project Sidewalk (PI: Prof. Jon Froehlich) makeability lab
  6. 6. I want to start with a story…
  7. 7. You Your Friend
  8. 8. 30.6million U.S. adults with mobility impairment
  9. 9. 15.2million use an assistive aid
  10. 10. Incomplete Sidewalks Physical Obstacles Surface Problems No Curb Ramps Stairs/Businesses
  11. 11. The lack of street-level accessibility information can have a significant impact on the independence and mobility of citizens cf. Nuernberger, 2008; Thapar et al., 2004
  12. 12. Accessibility-aware Navigation
  13. 13. Visualizing Accessibility of a City
  14. 14. Our goal is to collect and deliver data for the accessibility of every city in the world
  15. 15. Physical Street Audits
  16. 16. Time-consuming and expensive
  17. 17. Mobile Crowdsourcing SeeClickFix.com
  18. 18. These mobile tools require people to be on-site Mobile Crowdsourcing SeeClickFix.com
  19. 19. Use Google Street View (GSV) as a massive data source for scalably finding and characterizing street-level accessibility
  20. 20. AutomationCrowdsourcing How can we efficiently collect accurate accessibility data with…
  21. 21. Amazon Mechanical Turk is an online labor market where you can hire workers to complete small tasks
  22. 22. Task: Find the company name from an email domain $0.02 per task Task interface
  23. 23. Timer: 00:07:00 of 3 hours University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps Kotaro Hara 10 3 hours Crowdsourcing Data Collection Hara K., Le V., and Froehlich J.E [ASSETS2012, CHI2013] Crowdsourcing | Image Labeling
  24. 24. Manual labeling is accurate, but labor intensive
  25. 25. Manual labeling is accurate, but labor intensive
  26. 26. Computer Vision
  27. 27. Computer vision automatically finds curb ramps Automatic Curb Ramp Detection
  28. 28. Automatic Curb Ramp Detection Curb Ramp Labels Detected with Computer Vision
  29. 29. Automatic Curb Ramp Detection Curb Ramp Labels Detected with Computer Vision
  30. 30. Some curb ramps never get detected False detections Automatic Curb Ramp Detection
  31. 31. 2x Manual Label Verification
  32. 32. Computer vision + verification is cheaper but less accurate compared to manual labeling
  33. 33. Automatic Task Allocation Research Question How can we combine manual labeling and computer vision to achieve high accuracy and low cost?
  34. 34. Tohme遠目 Remote Eye・
  35. 35. Computer vision + verification is cheaper but less accurate Manual labeling is accurate, but labor intensive Design Principles
  36. 36. Computer vision + verification is cheaper but less accurate (not true for easy tasks) Manual labeling is accurate, but labor intensive Design Principles
  37. 37. Dataset svDetect Automatic Curb Ramp Detection svCrawl Web Scraper Tohme 遠目 Remote Eye・
  38. 38. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation Tohme 遠目 Remote Eye・
  39. 39. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification Tohme 遠目 Remote Eye・
  40. 40. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  41. 41. Tohme 遠目 Remote Eye・ .
  42. 42. Tohme 遠目 Remote Eye・
  43. 43. Tohme 遠目 Remote Eye・ Complexity: Cardinality: Depth: CV: 0.14 0.33 0.21 0.22
  44. 44. Tohme 遠目 Remote Eye・ Complexity: Cardinality: Depth: CV: 0.14 0.33 0.21 0.22 Predict computer vision performance
  45. 45. Tohme 遠目 Remote Eye・ Complexity: Cardinality: Depth: CV: 0.14 0.33 0.21 0.22 The easy task is passed to the cheaper verification workflow.
  46. 46. Tohme 遠目 Remote Eye・ .
  47. 47. Tohme 遠目 Remote Eye・
  48. 48. Tohme 遠目 Remote Eye・ Complexity: Cardinality: Depth: CV: 0.82 0.25 0.96 0.54
  49. 49. Tohme 遠目 Remote Eye・ Complexity: Cardinality: Depth: CV: 0.82 0.25 0.96 0.54
  50. 50. Tohme 遠目 Remote Eye・ Complexity: Cardinality: Depth: CV: 0.82 0.25 0.96 0.54The difficult task is passed to the more accurate labeling workflow.
  51. 51. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  52. 52. Google Street View Panoramas and Metadata 3D Point-cloud Data Top-down Google Maps Imagery Scraper
  53. 53. Saskatoon Los Angeles Baltimore Washington D.C. Washington D.C. Baltimore Los Angeles Saskatoon
  54. 54. D.C. | Downtown D.C. | Residential Scraper | Areas of Study
  55. 55. Washington D.C. Dense urban area Semi-urban residential areas Scraper
  56. 56. Washington D.C. Baltimore Los Angeles Saskatoon Total Area:11.3 km2 Intersections: 1,086 Curb Ramps: 2,877 Missing Curb Ramps:647 Avg. GSV Data Age:2.2 yr* * At the time of downloading data in summer 2013 Scraper
  57. 57. How well does GSV data reflect the current state of the physical world?
  58. 58. Vs.Vs.
  59. 59. Washington D.C. Baltimore Physical Audit Areas GSV and Physical World > 97.7% agreement 273 Intersections Dataset | Validating Dataset Small disagreement due to construction.
  60. 60. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  61. 61. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  62. 62. Dataset
  63. 63. Ground Truth Curb Ramp Dataset 2 researchers labeled curb ramps in our dataset 2,877 curb ramp labels (M=2.6 per intersection) Dataset
  64. 64. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  65. 65. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  66. 66. Deformable Part Models Felzenszwalb et al. 2008 Automatic Curb Ramp Detection http://www.cs.berkeley.edu/~rbg/latent/
  67. 67. Deformable Part Models Felzenszwalb et al. 2008 Automatic Curb Ramp Detection http://www.cs.berkeley.edu/~rbg/latent/ Root filter Parts filter Displacement cost
  68. 68. Automatic Curb Ramp Detection Multiple redundant detection boxes Detected Labels Stage 1: Deformable Part Model Correct 1 False Positive 12 Miss 0
  69. 69. Automatic Curb Ramp Detection Curb ramps shouldn’t be in the sky or on roofs Correct 1 False Positive 12 Miss 0 Detected Labels Stage 1: Deformable Part Model
  70. 70. Automatic Curb Ramp Detection Detected Labels Stage 2: Post-processing
  71. 71. Automatic Curb Ramp Detection Detected Labels Stage 3: SVM-based Refinement Filter out labels based on their size, color, and position. Correct 1 False Positive 5 Miss 0
  72. 72. Automatic Curb Ramp Detection Correct 1 False Positive 3 Miss 0 Detected Labels Stage 3: SVM-based Refinement
  73. 73. Google Street View Panoramic Image Curb Ramp Labels Detected by Computer Vision Automatic Curb Ramp Detection
  74. 74. Good example!
  75. 75. Bad Example :(
  76. 76. Used two-fold cross validation to evaluate CV sub-system
  77. 77. 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Precision(%) Recall (%) Automatic Curb Ramp Detection COMPUTER VISION SUB-SYSTEM RESULTS Precision Higher, less false positives Recall Higher, less false negatives
  78. 78. 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Precision(%) Recall (%) Automatic Curb Ramp Detection COMPUTER VISION SUB-SYSTEM RESULTS Goal: maximize area under curve
  79. 79. 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Precision(%) Recall (%) Stage 1: DPM Stage 2: Post-Processing Stage 3: SVM Automatic Curb Ramp Detection COMPUTER VISION SUB-SYSTEM RESULTS More than 20% of curb ramps were missed
  80. 80. 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Precision(%) Recall (%) Stage 1: DPM Stage 2: Post-Processing Stage 3: SVM Automatic Curb Ramp Detection COMPUTER VISION SUB-SYSTEM RESULTS Confidence threshold of - 0.99, which results in 26% precision and 67% recall
  81. 81. Occlusion Illumination Scale Viewpoint Variation Structures Similar to Curb Ramps Curb Ramp Design Variation Automatic Curb Ramp Detection CURB RAMP DETECTION IS A HARD PROBLEM
  82. 82. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  83. 83. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  84. 84. Automatic Task Allocation | Features to Assess Scene Difficulty for CV A number of streets connected in an intersection Depth information for a road width and variance in distance Top-down images to assess complexity of an intersection A number of detections and confidence values
  85. 85. Automatic Task Allocation | Features to Assess Scene Difficulty for CV A number of street from metadata Depth information to assess a road width and variance in distance Top-down images to assess complexity of an intersection A number of detections and confidence values
  86. 86. Depth information for a road width and variance in distance Automatic Task Allocation | Features to Assess Scene Difficulty for CV
  87. 87. Automatic Task Allocation | Features to Assess Scene Difficulty for CV A number of streets from metadata Depth information for a road width and variance in distance Top-down images to assess complexity of an intersection A number of detections and confidence values
  88. 88. Google Maps Styled Maps Top-down images to assess complexity of an intersection Automatic Task Allocation | Features to Assess Scene Difficulty for CV
  89. 89. Automatic Task Allocation | Features to Assess Scene Difficulty for CV A number of streets from metadata Depth information for a road width and variance in distance Top-down images to assess complexity of an intersection CV Output: A number of detections and confidence values
  90. 90. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  91. 91. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  92. 92. 3x Manual Labeling | Labeling Interface
  93. 93. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  94. 94. svCrawl Web Scraper Dataset svDetect Automatic Curb Ramp Detection svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Tohme 遠目 Remote Eye・
  95. 95. 2x Manual Label Verification
  96. 96. Automatic Task Allocation Can we combine manual labeling and computer vision to achieve high accuracy and low cost?
  97. 97. STUDY METHOD: CONDITIONS Manual labeling without smart task allocation &vs. CV + Verification without smart task allocation Tohme遠目 Remote Eye・ vs. Evaluation
  98. 98. Accuracy Task Completion Time Evaluation STUDY METHOD: MEASURES
  99. 99. Recruited workers from Mturk Used 1,046 GSV images (40 used for golden insertion) Evaluation STUDY METHOD: APPROACH
  100. 100. RESULTS Labeling Tasks Verification Tasks # of distinct turkers: 242 161 1,270 582# of HITs completed: # of tasks completed: 6,350 4,820 # of tasks allocated: 769 277 Evaluation We used Monte Carlo simulations for evaluation
  101. 101. 84% 68% 83% 88% 58% 86%86% 63% 84% 0% 20% 40% 60% 80% 100% AccuracyMeasures(%) Precision Recall F-measure 94 42 81 0 20 40 60 80 100 TaskCompletionTime/Scene(s) Accuracy measures Task completion time per scene Manual Labeling CV and Manual Verification & Tohme 遠目 Remote Eye・ Manual Labeling CV and Manual Verification & Tohme 遠目 Remote Eye・ Evaluation | Labeling Accuracy and Time Cost Error bars are standard deviations. ACCURACY COST (TIME)
  102. 102. 84% 68% 83% 88% 58% 86%86% 63% 84% 0% 20% 40% 60% 80% 100% AccuracyMeasures(%) Precision Recall F-measure Error bars are standard deviations. Manual Labeling CV and Manual Verification & 94 42 81 0 20 40 60 80 100 TaskCompletionTime/Scene(s) Manual Labeling CV and Manual Verification & Accuracy measures Task completion time per scene Tohme 遠目 Remote Eye・ Tohme 遠目 Remote Eye・ Evaluation | Labeling Accuracy and Time Cost 13% reduction in cost ACCURACY COST (TIME)
  103. 103. svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Evaluation | Smart Task Allocator ~80% of svVerify tasks were correctly routed ~50% of svLabel tasks were correctly routed
  104. 104. svControl Automatic Task Allocation svVerify Manual Label Verification svLabel Manual Labeling Evaluation | Smart Task Allocator If svControl worked perfectly, Tohme’s cost would drop to 28% of a manually labelling approach alone.
  105. 105. Example Labels from Manual Labeling
  106. 106. Evaluation | Example Labels from Manual Labeling
  107. 107. Evaluation | Example Labels from Manual Labeling
  108. 108. Evaluation | Example Labels from Manual Labeling
  109. 109. Evaluation | Example Labels from Manual Labeling
  110. 110. Evaluation | Example Labels from Manual Labeling
  111. 111. This is a driveway. Not a curb ramp. Evaluation | Example Labels from Manual Labeling
  112. 112. Evaluation | Example Labels from Manual Labeling
  113. 113. Evaluation | Example Labels from Manual Labeling
  114. 114. Examples Labels from CV + Verification
  115. 115. Raw Street View Image Evaluation | Example Labels from CV + Verification
  116. 116. False detection Automatic Detection Evaluation | Example Labels from CV + Verification
  117. 117. Automatic Detection + Human Verification Evaluation | Example Labels from CV + Verification
  118. 118. 8,209Intersections in DC
  119. 119. 8,209Intersections in DC BACK OF THE ENVELOPE CALCULATIONS Manually labeling GSV with our custom interfaces would take 214 hours With Tohme, this drops to 184 hours We think we can do better 
  120. 120. makeability lab Smart task management can improve efficiency of semi-automatic crowd-powered system Takeaway We can combine crowdsourcing and automated methods to collect accessibility data from Street View
  121. 121. FUTURE WORK: COMPUTER VISION Context integration & scene understanding 3D-data integration Improve training & sample size Mensuration
  122. 122. FUTURE WORK: DEPLOYMENT OF VOLUNTEER WEB SITE
  123. 123. This work is supported by Faculty Research Award makeability lab
  124. 124. THE CROWD-POWERED STREETVIEW ACCESSIBILITY TEAM! Kotaro Hara Jin Sun Victoria Le Robert Moore Sean Pannella Jonah Chazan David Jacobs Jon Froehlich Zachary Lawrence Graduate Student Undergraduate High School Professor Thanks! @kotarohara_en | kotaro@cs.umd.edu

×