SlideShare a Scribd company logo
1 of 133
makeability lab
クラウドソーシング・コンピュータビジョン・
ストリートビューを用いた歩道の
アクセシビリティデータの収集手法
原航太郎 | Project Sidewalk (PI: Jon E. Froehlich)
A
B
C
D
A
B
C
Human-Computer Interaction Lab
Characterizing Sidewalk
Accessibility at Scale
using Google Street View, Crowdsourcing, and
Automated Methods
Kotaro Hara | Project Sidewalk (PI: Prof. Jon Froehlich)
makeability lab
I want to start with a story…
You Your Friend
30.6million U.S. adults with mobility impairment
15.2million use an assistive aid
Incomplete Sidewalks Physical Obstacles Surface Problems No Curb Ramps Stairs/Businesses
The lack of street-level
accessibility information can
have a significant impact on
the independence and
mobility of citizens
cf. Nuernberger, 2008; Thapar et al., 2004
Accessibility-aware Navigation
Visualizing Accessibility of a City
Our goal is to collect and deliver data for
the accessibility of every city in the world
Physical Street Audits
Time-consuming and expensive
Mobile Crowdsourcing
SeeClickFix.com
These mobile tools require people to be on-site
Mobile Crowdsourcing
SeeClickFix.com
Use Google Street View (GSV) as a massive data source for
scalably finding and characterizing street-level accessibility
AutomationCrowdsourcing
How can we efficiently collect accurate accessibility data with…
Amazon Mechanical Turk is an online labor market
where you can hire workers to complete small tasks
Task: Find the company name from an email domain
$0.02 per task
Task interface
Timer: 00:07:00 of 3 hours
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
Kotaro Hara 10 3 hours
Crowdsourcing Data Collection
Hara K., Le V., and Froehlich J.E [ASSETS2012, CHI2013]
Crowdsourcing | Image Labeling
Manual labeling is accurate,
but labor intensive
Manual labeling is accurate,
but labor intensive
Computer Vision
Computer vision
automatically finds
curb ramps
Automatic Curb Ramp Detection
Automatic Curb Ramp Detection
Curb Ramp Labels Detected with Computer Vision
Automatic Curb Ramp Detection
Curb Ramp Labels Detected with Computer Vision
Some curb ramps
never get detected
False detections
Automatic Curb Ramp Detection
2x
Manual Label Verification
Computer vision + verification is cheaper
but less accurate compared to manual labeling
Automatic Task Allocation
Research Question
How can we combine manual labeling and
computer vision to achieve high accuracy and low cost?
Tohme遠目 Remote Eye・
Computer vision + verification is
cheaper but less accurate
Manual labeling is accurate,
but labor intensive
Design Principles
Computer vision + verification is
cheaper but less accurate
(not true for easy tasks)
Manual labeling is accurate,
but labor intensive
Design Principles
Dataset
svDetect
Automatic Curb
Ramp Detection
svCrawl
Web Scraper
Tohme
遠目 Remote Eye・
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
Tohme
遠目 Remote Eye・
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
Tohme
遠目 Remote Eye・
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
Tohme
遠目 Remote Eye・
.
Tohme
遠目 Remote Eye・
Tohme
遠目 Remote Eye・
Complexity:
Cardinality:
Depth:
CV:
0.14
0.33
0.21
0.22
Tohme
遠目 Remote Eye・
Complexity:
Cardinality:
Depth:
CV:
0.14
0.33
0.21
0.22
Predict computer vision
performance
Tohme
遠目 Remote Eye・
Complexity:
Cardinality:
Depth:
CV:
0.14
0.33
0.21
0.22
The easy task is passed to the
cheaper verification workflow.
Tohme
遠目 Remote Eye・
.
Tohme
遠目 Remote Eye・
Tohme
遠目 Remote Eye・
Complexity:
Cardinality:
Depth:
CV:
0.82
0.25
0.96
0.54
Tohme
遠目 Remote Eye・
Complexity:
Cardinality:
Depth:
CV:
0.82
0.25
0.96
0.54
Tohme
遠目 Remote Eye・ Complexity:
Cardinality:
Depth:
CV:
0.82
0.25
0.96
0.54The difficult task is passed to the
more accurate labeling workflow.
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
Google Street View Panoramas and Metadata
3D Point-cloud Data
Top-down Google Maps Imagery
Scraper
Saskatoon
Los Angeles
Baltimore
Washington D.C.
Washington D.C.
Baltimore
Los Angeles
Saskatoon
D.C. | Downtown D.C. | Residential
Scraper | Areas of Study
Washington D.C.
Dense urban area
Semi-urban residential areas
Scraper
Washington D.C. Baltimore Los Angeles Saskatoon
Total Area:11.3 km2
Intersections: 1,086
Curb Ramps: 2,877
Missing Curb Ramps:647
Avg. GSV Data Age:2.2 yr*
* At the time of downloading data in summer 2013
Scraper
How well does GSV data reflect
the current state of the physical
world?
Vs.Vs.
Washington
D.C.
Baltimore
Physical Audit Areas
GSV and Physical World
> 97.7% agreement
273 Intersections
Dataset | Validating Dataset
Small disagreement due to
construction.
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
Dataset
Ground Truth Curb Ramp Dataset
2 researchers labeled curb ramps in our dataset
2,877 curb ramp labels (M=2.6 per intersection)
Dataset
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
Deformable Part Models
Felzenszwalb et al. 2008
Automatic Curb Ramp Detection
http://www.cs.berkeley.edu/~rbg/latent/
Deformable Part Models
Felzenszwalb et al. 2008
Automatic Curb Ramp Detection
http://www.cs.berkeley.edu/~rbg/latent/
Root filter Parts filter Displacement cost
Automatic Curb Ramp Detection
Multiple redundant
detection boxes
Detected Labels
Stage 1: Deformable Part Model
Correct 1
False Positive 12
Miss 0
Automatic Curb Ramp Detection
Curb ramps shouldn’t be
in the sky or on roofs
Correct 1
False Positive 12
Miss 0
Detected Labels
Stage 1: Deformable Part Model
Automatic Curb Ramp Detection
Detected Labels
Stage 2: Post-processing
Automatic Curb Ramp Detection
Detected Labels
Stage 3: SVM-based Refinement
Filter out labels based on
their size, color, and position.
Correct 1
False Positive 5
Miss 0
Automatic Curb Ramp Detection
Correct 1
False Positive 3
Miss 0
Detected Labels
Stage 3: SVM-based Refinement
Google Street View Panoramic Image
Curb Ramp Labels Detected by Computer Vision
Automatic Curb Ramp Detection
Good example!
Bad Example :(
Used two-fold cross validation to evaluate CV sub-system
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Precision(%)
Recall (%)
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
Precision
Higher, less false positives
Recall
Higher, less false negatives
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Precision(%)
Recall (%)
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
Goal:
maximize area
under curve
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Precision(%)
Recall (%)
Stage 1: DPM
Stage 2: Post-Processing
Stage 3: SVM
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
More than 20% of
curb ramps were
missed
0%
20%
40%
60%
80%
100%
0% 20% 40% 60% 80% 100%
Precision(%)
Recall (%)
Stage 1: DPM
Stage 2: Post-Processing
Stage 3: SVM
Automatic Curb Ramp Detection
COMPUTER VISION SUB-SYSTEM RESULTS
Confidence
threshold of -
0.99, which
results in 26%
precision and
67% recall
Occlusion Illumination
Scale Viewpoint Variation
Structures Similar to Curb Ramps Curb Ramp Design Variation
Automatic Curb Ramp Detection
CURB RAMP DETECTION IS A HARD PROBLEM
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
Automatic Task Allocation | Features to Assess Scene Difficulty for CV
A number of streets connected in an intersection
Depth information for a road width and variance in distance
Top-down images to assess complexity of an intersection
A number of detections and confidence values
Automatic Task Allocation | Features to Assess Scene Difficulty for CV
A number of street from metadata
Depth information to assess a road width and variance in distance
Top-down images to assess complexity of an intersection
A number of detections and confidence values
Depth information for a road width and variance in distance
Automatic Task Allocation | Features to Assess Scene Difficulty for CV
Automatic Task Allocation | Features to Assess Scene Difficulty for CV
A number of streets from metadata
Depth information for a road width and variance in distance
Top-down images to assess complexity of an intersection
A number of detections and confidence values
Google Maps Styled Maps
Top-down images to assess complexity of an intersection
Automatic Task Allocation | Features to Assess Scene Difficulty for CV
Automatic Task Allocation | Features to Assess Scene Difficulty for CV
A number of streets from metadata
Depth information for a road width and variance in distance
Top-down images to assess complexity of an intersection
CV Output: A number of detections and confidence values
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
3x
Manual Labeling | Labeling Interface
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb
Ramp Detection
svControl
Automatic
Task Allocation
svVerify
Manual Label
Verification
svLabel
Manual Labeling
Tohme
遠目 Remote Eye・
2x
Manual Label Verification
Automatic Task Allocation
Can we combine manual labeling and
computer vision to achieve high accuracy and low cost?
STUDY METHOD: CONDITIONS
Manual labeling without
smart task allocation
&vs.
CV + Verification without
smart task allocation
Tohme遠目 Remote Eye・
vs.
Evaluation
Accuracy Task Completion Time
Evaluation
STUDY METHOD: MEASURES
Recruited workers from Mturk
Used 1,046 GSV images (40 used for golden insertion)
Evaluation
STUDY METHOD: APPROACH
RESULTS
Labeling Tasks Verification Tasks
# of distinct turkers: 242 161
1,270 582# of HITs completed:
# of tasks completed: 6,350 4,820
# of tasks allocated: 769 277
Evaluation
We used Monte Carlo simulations for evaluation
84%
68%
83%
88%
58%
86%86%
63%
84%
0%
20%
40%
60%
80%
100%
AccuracyMeasures(%)
Precision Recall F-measure 94
42
81
0
20
40
60
80
100
TaskCompletionTime/Scene(s)
Accuracy
measures
Task
completion
time per scene
Manual
Labeling
CV and Manual
Verification
& Tohme
遠目 Remote Eye・ Manual
Labeling
CV and Manual
Verification
& Tohme
遠目 Remote Eye・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
ACCURACY COST (TIME)
84%
68%
83%
88%
58%
86%86%
63%
84%
0%
20%
40%
60%
80%
100%
AccuracyMeasures(%)
Precision Recall F-measure
Error bars are standard deviations.
Manual
Labeling
CV and Manual
Verification
&
94
42
81
0
20
40
60
80
100
TaskCompletionTime/Scene(s)
Manual
Labeling
CV and Manual
Verification
&
Accuracy
measures
Task
completion
time per scene
Tohme
遠目 Remote Eye・
Tohme
遠目 Remote Eye・
Evaluation | Labeling Accuracy and Time Cost
13% reduction
in cost
ACCURACY COST (TIME)
svControl
Automatic
Task Allocation svVerify
Manual Label
Verification
svLabel
Manual Labeling
Evaluation | Smart Task Allocator
~80% of svVerify tasks were correctly routed
~50% of svLabel tasks were correctly routed
svControl
Automatic
Task Allocation svVerify
Manual Label
Verification
svLabel
Manual Labeling
Evaluation | Smart Task Allocator
If svControl worked perfectly,
Tohme’s cost would drop to 28% of
a manually labelling approach
alone.
Example Labels from Manual Labeling
Evaluation | Example Labels from Manual Labeling
Evaluation | Example Labels from Manual Labeling
Evaluation | Example Labels from Manual Labeling
Evaluation | Example Labels from Manual Labeling
Evaluation | Example Labels from Manual Labeling
This is a driveway.
Not a curb ramp.
Evaluation | Example Labels from Manual Labeling
Evaluation | Example Labels from Manual Labeling
Evaluation | Example Labels from Manual Labeling
Examples Labels from CV + Verification
Raw Street View Image
Evaluation | Example Labels from CV + Verification
False detection
Automatic Detection
Evaluation | Example Labels from CV + Verification
Automatic Detection + Human Verification
Evaluation | Example Labels from CV + Verification
8,209Intersections in DC
8,209Intersections in DC
BACK OF THE ENVELOPE CALCULATIONS
Manually labeling GSV with our custom interfaces
would take 214 hours
With Tohme, this drops to 184 hours
We think we can do better 
makeability lab
Smart task management can improve efficiency of
semi-automatic crowd-powered system
Takeaway
We can combine crowdsourcing and automated
methods to collect accessibility data from Street View
FUTURE WORK: COMPUTER VISION
Context integration & scene understanding
3D-data integration
Improve training & sample size
Mensuration
FUTURE WORK: DEPLOYMENT OF VOLUNTEER WEB SITE
This work is supported by
Faculty Research Award
makeability lab
THE CROWD-POWERED STREETVIEW ACCESSIBILITY TEAM!
Kotaro Hara Jin Sun Victoria Le Robert Moore Sean Pannella
Jonah Chazan David Jacobs Jon Froehlich
Zachary Lawrence
Graduate Student
Undergraduate
High School
Professor
Thanks!
@kotarohara_en | kotaro@cs.umd.edu

More Related Content

Viewers also liked

The Drone Alternative: Multispectral Kite Aerial Photography & Photo Processi...
The Drone Alternative: Multispectral Kite Aerial Photography & Photo Processi...The Drone Alternative: Multispectral Kite Aerial Photography & Photo Processi...
The Drone Alternative: Multispectral Kite Aerial Photography & Photo Processi...Michele Tobias
 
Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Social Street View: Blending Immersive Street Views with Geo-tagged Social MediaSocial Street View: Blending Immersive Street Views with Geo-tagged Social Media
Social Street View: Blending Immersive Street Views with Geo-tagged Social MediaRuofei Du
 
Using a GoPro or 360 Camera for On-the-Job Training
Using a GoPro or 360 Camera for On-the-Job TrainingUsing a GoPro or 360 Camera for On-the-Job Training
Using a GoPro or 360 Camera for On-the-Job Trainingtlslearning
 
A Local Business's Guide to Creating Winning AdWords Campaigns
A Local Business's Guide to Creating Winning AdWords CampaignsA Local Business's Guide to Creating Winning AdWords Campaigns
A Local Business's Guide to Creating Winning AdWords CampaignsSurefire Local
 
Google Street View Presentation_Final
Google Street View Presentation_FinalGoogle Street View Presentation_Final
Google Street View Presentation_FinalRathesh Pillai
 
Virtuele Realiteit in de openbare sector
Virtuele Realiteit in de openbare sectorVirtuele Realiteit in de openbare sector
Virtuele Realiteit in de openbare sectorFredo De Smet
 
Google Street View in Your Apps
Google Street View in Your AppsGoogle Street View in Your Apps
Google Street View in Your AppsCodemotion
 
Aerial Photography - Geoforum 2016 - Richard Evans, Getmapping
Aerial Photography - Geoforum 2016 - Richard Evans, GetmappingAerial Photography - Geoforum 2016 - Richard Evans, Getmapping
Aerial Photography - Geoforum 2016 - Richard Evans, GetmappingEDINA, University of Edinburgh
 
Make Google Work for You: A Local Business Guide to AdWords
Make Google Work for You: A Local Business Guide to AdWordsMake Google Work for You: A Local Business Guide to AdWords
Make Google Work for You: A Local Business Guide to AdWordsSurefire Local
 
ゴジラっぽい画面をつくる
ゴジラっぽい画面をつくるゴジラっぽい画面をつくる
ゴジラっぽい画面をつくるsasamijp
 
How to create 360 images with google street view app
How to create 360 images with google street view app How to create 360 images with google street view app
How to create 360 images with google street view app proyectoste
 

Viewers also liked (14)

Googleguide
GoogleguideGoogleguide
Googleguide
 
The Drone Alternative: Multispectral Kite Aerial Photography & Photo Processi...
The Drone Alternative: Multispectral Kite Aerial Photography & Photo Processi...The Drone Alternative: Multispectral Kite Aerial Photography & Photo Processi...
The Drone Alternative: Multispectral Kite Aerial Photography & Photo Processi...
 
Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
Social Street View: Blending Immersive Street Views with Geo-tagged Social MediaSocial Street View: Blending Immersive Street Views with Geo-tagged Social Media
Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
 
Using a GoPro or 360 Camera for On-the-Job Training
Using a GoPro or 360 Camera for On-the-Job TrainingUsing a GoPro or 360 Camera for On-the-Job Training
Using a GoPro or 360 Camera for On-the-Job Training
 
A Local Business's Guide to Creating Winning AdWords Campaigns
A Local Business's Guide to Creating Winning AdWords CampaignsA Local Business's Guide to Creating Winning AdWords Campaigns
A Local Business's Guide to Creating Winning AdWords Campaigns
 
Google Street View Presentation_Final
Google Street View Presentation_FinalGoogle Street View Presentation_Final
Google Street View Presentation_Final
 
Virtuele Realiteit in de openbare sector
Virtuele Realiteit in de openbare sectorVirtuele Realiteit in de openbare sector
Virtuele Realiteit in de openbare sector
 
Google Street View in Your Apps
Google Street View in Your AppsGoogle Street View in Your Apps
Google Street View in Your Apps
 
360 camera
360 camera360 camera
360 camera
 
Aerial Photography - Geoforum 2016 - Richard Evans, Getmapping
Aerial Photography - Geoforum 2016 - Richard Evans, GetmappingAerial Photography - Geoforum 2016 - Richard Evans, Getmapping
Aerial Photography - Geoforum 2016 - Richard Evans, Getmapping
 
Google Street View
Google Street ViewGoogle Street View
Google Street View
 
Make Google Work for You: A Local Business Guide to AdWords
Make Google Work for You: A Local Business Guide to AdWordsMake Google Work for You: A Local Business Guide to AdWords
Make Google Work for You: A Local Business Guide to AdWords
 
ゴジラっぽい画面をつくる
ゴジラっぽい画面をつくるゴジラっぽい画面をつくる
ゴジラっぽい画面をつくる
 
How to create 360 images with google street view app
How to create 360 images with google street view app How to create 360 images with google street view app
How to create 360 images with google street view app
 

Similar to Using Crowdsourcing, Automated Methods and Google Street View to Collect Sidewalk Accessibility Data

Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Co...
Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Co...Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Co...
Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Co...Jon Froehlich
 
ABD322_Implementing a Flight Simulator Interface Using AI, Virtual Reality, a...
ABD322_Implementing a Flight Simulator Interface Using AI, Virtual Reality, a...ABD322_Implementing a Flight Simulator Interface Using AI, Virtual Reality, a...
ABD322_Implementing a Flight Simulator Interface Using AI, Virtual Reality, a...Amazon Web Services
 
2018.07 Self-driving cars meetup
2018.07 Self-driving cars meetup2018.07 Self-driving cars meetup
2018.07 Self-driving cars meetupPunnu Phairatt
 
AWS re:Invent 2016: How Mapbox Uses the AWS Edge to Deliver Fast Maps for Mob...
AWS re:Invent 2016: How Mapbox Uses the AWS Edge to Deliver Fast Maps for Mob...AWS re:Invent 2016: How Mapbox Uses the AWS Edge to Deliver Fast Maps for Mob...
AWS re:Invent 2016: How Mapbox Uses the AWS Edge to Deliver Fast Maps for Mob...Amazon Web Services
 
Deep Reinforcement Learning | Amazon Robotics Challenge, Image Processing Lec...
Deep Reinforcement Learning | Amazon Robotics Challenge, Image Processing Lec...Deep Reinforcement Learning | Amazon Robotics Challenge, Image Processing Lec...
Deep Reinforcement Learning | Amazon Robotics Challenge, Image Processing Lec...Juxi Leitner
 
Webinar1 darpa07
Webinar1 darpa07Webinar1 darpa07
Webinar1 darpa07MKGJUICE
 
IRJET- Robust and Fast Detection of Moving Vechiles in Aerial Videos usin...
IRJET-  	  Robust and Fast Detection of Moving Vechiles in Aerial Videos usin...IRJET-  	  Robust and Fast Detection of Moving Vechiles in Aerial Videos usin...
IRJET- Robust and Fast Detection of Moving Vechiles in Aerial Videos usin...IRJET Journal
 
Cloudy Open Source and DevOps
Cloudy Open Source and DevOpsCloudy Open Source and DevOps
Cloudy Open Source and DevOpsMatt O'Keefe
 
Online Camer Calibration
Online Camer CalibrationOnline Camer Calibration
Online Camer CalibrationFei-Fei Zheng
 
TRIP GENERATION Survey and ORIGIN / DESTINATION STUDY
TRIP GENERATION Survey and ORIGIN / DESTINATION STUDYTRIP GENERATION Survey and ORIGIN / DESTINATION STUDY
TRIP GENERATION Survey and ORIGIN / DESTINATION STUDYGrant Johnson, PE
 
Academic research on graph processing: connecting recent findings to industri...
Academic research on graph processing: connecting recent findings to industri...Academic research on graph processing: connecting recent findings to industri...
Academic research on graph processing: connecting recent findings to industri...openCypher
 
LANE DETECTION USING IMAGE PROCESSING IN PYTHON
LANE DETECTION USING IMAGE PROCESSING IN PYTHONLANE DETECTION USING IMAGE PROCESSING IN PYTHON
LANE DETECTION USING IMAGE PROCESSING IN PYTHONIRJET Journal
 
tatacara kalibrasi Kendaraan.ppt
tatacara kalibrasi Kendaraan.ppttatacara kalibrasi Kendaraan.ppt
tatacara kalibrasi Kendaraan.pptdarmadi ir,mm
 
vehicle calibration.ppt
vehicle calibration.pptvehicle calibration.ppt
vehicle calibration.pptdarmadi ir,mm
 
Innovative trends in robotics
Innovative trends in roboticsInnovative trends in robotics
Innovative trends in roboticsDesign World
 
IRJET- Front View Identification of Vehicles by using Machine Learning Te...
IRJET-  	  Front View Identification of Vehicles by using Machine Learning Te...IRJET-  	  Front View Identification of Vehicles by using Machine Learning Te...
IRJET- Front View Identification of Vehicles by using Machine Learning Te...IRJET Journal
 
2009 APWA International Public Works Congress
2009 APWA International Public Works Congress2009 APWA International Public Works Congress
2009 APWA International Public Works CongressFranz Loewenherz
 
Automatically Discovering, Reporting and Reproducing Android Application Crashes
Automatically Discovering, Reporting and Reproducing Android Application CrashesAutomatically Discovering, Reporting and Reproducing Android Application Crashes
Automatically Discovering, Reporting and Reproducing Android Application CrashesKevin Moran
 

Similar to Using Crowdsourcing, Automated Methods and Google Street View to Collect Sidewalk Accessibility Data (20)

Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Co...
Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Co...Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Co...
Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Co...
 
ABD322_Implementing a Flight Simulator Interface Using AI, Virtual Reality, a...
ABD322_Implementing a Flight Simulator Interface Using AI, Virtual Reality, a...ABD322_Implementing a Flight Simulator Interface Using AI, Virtual Reality, a...
ABD322_Implementing a Flight Simulator Interface Using AI, Virtual Reality, a...
 
2018.07 Self-driving cars meetup
2018.07 Self-driving cars meetup2018.07 Self-driving cars meetup
2018.07 Self-driving cars meetup
 
AWS re:Invent 2016: How Mapbox Uses the AWS Edge to Deliver Fast Maps for Mob...
AWS re:Invent 2016: How Mapbox Uses the AWS Edge to Deliver Fast Maps for Mob...AWS re:Invent 2016: How Mapbox Uses the AWS Edge to Deliver Fast Maps for Mob...
AWS re:Invent 2016: How Mapbox Uses the AWS Edge to Deliver Fast Maps for Mob...
 
Deep Reinforcement Learning | Amazon Robotics Challenge, Image Processing Lec...
Deep Reinforcement Learning | Amazon Robotics Challenge, Image Processing Lec...Deep Reinforcement Learning | Amazon Robotics Challenge, Image Processing Lec...
Deep Reinforcement Learning | Amazon Robotics Challenge, Image Processing Lec...
 
Webinar1 darpa07
Webinar1 darpa07Webinar1 darpa07
Webinar1 darpa07
 
LiDAR feature extraction
LiDAR feature extractionLiDAR feature extraction
LiDAR feature extraction
 
IRJET- Robust and Fast Detection of Moving Vechiles in Aerial Videos usin...
IRJET-  	  Robust and Fast Detection of Moving Vechiles in Aerial Videos usin...IRJET-  	  Robust and Fast Detection of Moving Vechiles in Aerial Videos usin...
IRJET- Robust and Fast Detection of Moving Vechiles in Aerial Videos usin...
 
PhD Dissertation Defense (April 2015)
PhD Dissertation Defense (April 2015)PhD Dissertation Defense (April 2015)
PhD Dissertation Defense (April 2015)
 
Cloudy Open Source and DevOps
Cloudy Open Source and DevOpsCloudy Open Source and DevOps
Cloudy Open Source and DevOps
 
Online Camer Calibration
Online Camer CalibrationOnline Camer Calibration
Online Camer Calibration
 
TRIP GENERATION Survey and ORIGIN / DESTINATION STUDY
TRIP GENERATION Survey and ORIGIN / DESTINATION STUDYTRIP GENERATION Survey and ORIGIN / DESTINATION STUDY
TRIP GENERATION Survey and ORIGIN / DESTINATION STUDY
 
Academic research on graph processing: connecting recent findings to industri...
Academic research on graph processing: connecting recent findings to industri...Academic research on graph processing: connecting recent findings to industri...
Academic research on graph processing: connecting recent findings to industri...
 
LANE DETECTION USING IMAGE PROCESSING IN PYTHON
LANE DETECTION USING IMAGE PROCESSING IN PYTHONLANE DETECTION USING IMAGE PROCESSING IN PYTHON
LANE DETECTION USING IMAGE PROCESSING IN PYTHON
 
tatacara kalibrasi Kendaraan.ppt
tatacara kalibrasi Kendaraan.ppttatacara kalibrasi Kendaraan.ppt
tatacara kalibrasi Kendaraan.ppt
 
vehicle calibration.ppt
vehicle calibration.pptvehicle calibration.ppt
vehicle calibration.ppt
 
Innovative trends in robotics
Innovative trends in roboticsInnovative trends in robotics
Innovative trends in robotics
 
IRJET- Front View Identification of Vehicles by using Machine Learning Te...
IRJET-  	  Front View Identification of Vehicles by using Machine Learning Te...IRJET-  	  Front View Identification of Vehicles by using Machine Learning Te...
IRJET- Front View Identification of Vehicles by using Machine Learning Te...
 
2009 APWA International Public Works Congress
2009 APWA International Public Works Congress2009 APWA International Public Works Congress
2009 APWA International Public Works Congress
 
Automatically Discovering, Reporting and Reproducing Android Application Crashes
Automatically Discovering, Reporting and Reproducing Android Application CrashesAutomatically Discovering, Reporting and Reproducing Android Application Crashes
Automatically Discovering, Reporting and Reproducing Android Application Crashes
 

Recently uploaded

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????blackmambaettijean
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 

Using Crowdsourcing, Automated Methods and Google Street View to Collect Sidewalk Accessibility Data

Editor's Notes

  1. My name is Kotaro Hara. Today, I will talk about how we can use automated methods and crowdsourcing to collect accessibility information about cities
  2. My name is Kotaro Hara. Today, I will talk about how we can use automated methods and crowdsourcing to collect accessibility information about cities
  3. I want to tell you a story…
  4. Imagine that you and a friend are on a walk. You’re both somewhat unfamiliar with the area. Suddenly, in the middle of the sidewalk, you encounter a fire hydrant -- Image Reference http://www.iconsdb.com/black-icons/fire-hydrant-icon.html
  5. In this case, you manage to go around because there is a driveway, but they are temporarily forced onto the street which is dangerous.
  6. Now, you get to the end of the block and discover that there is no curb cut. You are forced to turn around and find another way. The problem is not only the sidewalks remain inaccessible, but there are currently few mechanisms to find out about the accessibility of a route in advance
  7. Now, you get to the end of the block and discover that there is no curb cut. You are forced to turn around and find another way. The problem is not only the sidewalks remain inaccessible, but there are currently few mechanisms to find out about the accessibility of a route in advance -- Quote from paper The problem is not just that sidewalk accessibility fundamentally affects where and how people travel in cities but also that there are few, if any, mechanisms to determine accessible areas of a city a priori -- What Jon wrote The problem is not just that there are inaccessible areas of cities but that there are currently few methods for us to determine them a priori
  8. Now, you get to the end of the block and discover that there is no curb cut. You are forced to turn around and find another way. The problem is not only the sidewalks remain inaccessible, but there are currently few mechanisms to find out about the accessibility of a route in advance -- Quote from paper The problem is not just that sidewalk accessibility fundamentally affects where and how people travel in cities but also that there are few, if any, mechanisms to determine accessible areas of a city a priori -- What Jon wrote The problem is not just that there are inaccessible areas of cities but that there are currently few methods for us to determine them a priori
  9. According to the most recent US Census (2010), roughly 30.6 million adults have physical disabilities that affect their ambulatory activities [128]. ----- Flickr: 3627562740_c74f7bfb82_o.jpg
  10. Of these, nearly half report using an assistive aid such as a wheelchair (3.6 million) or a cane, crutches, or walker (11.6 million) 内閣府のデータでは日本では総数366.3万人。 ---- Flickr: 14816521847_5c3c7af348_o.jpg
  11. Despite comprehensive civil rights legislation for Americans with disabilities (e.g., [9,75]), many city streets, sidewalks, and businesses in the US remain inaccessible [90,96,120].
  12. The lack of street-level accessibility information can have a significant negative impact on the independence and mobility of citizens [99,120]. 99: Nuernberger, A. (2008). Presenting accessibility to mobility-impaired travelers. (Doctoral dissertation, University of California, Berkeley). 120: Thapar, N., Warner, G., Drainoni, M., Williams, S., Ditchfield, H., Wierbicky, J., & Nesathurai, S. (2004). A pilot of functional access to public buildings and facilities for persons with impairments. Disability and Rehabilitation, 26(5), 280-9.
  13. So we would like to develop technologies such as an accessibility aware navigation system. It shows an accessible path instead of a shortest path based on your mobility level.
  14. We also want to build an application that allows you to visualize the accessibility of a city. You can quickly compare which area of a city is more accessible. We need geo-data to make these.
  15. To do this, we need a lot of data about accessibility. Our group’s goal is to collect and deliver street-level accessibility data for every city in the world. -- Image http://www.flickr.com/photos/rgb12/6225459696/lightbox/
  16. Traditionally, information about a neighborhood have been gathered by volunteers or government organizations through physical audit.
  17. However, this is time-consuming and expensive.
  18. Mobile crowdsourcing such as SeeClickFix.com
  19. Mobile crowdsourcing such as SeeClickFix.com
  20. And NYC 311 allows citizens to report neighborhood sidewalk accessibility issues.
  21. But this requires people to be on-site
  22. Our approach is different though complementary. Use Google Street View as a massive data source…
  23. Today, I am going to talk about how we can use crowdsourcing and automated methods to collect accessibility data Google Street View.
  24. Amazon Mechanical Turk is an online labor market where you can hire workers to complete small tasks.
  25. For example, if you are a worker, you can go to Amazon’s website to browse through available tasks
  26. Choose one of the tasks. For example, this task is about finding the company name from an email domain. You can get 2 cents for completing a task through this web interface.
  27. We recruit crowd worker from Amazon Mechanical Turk. For those of you who don’t know Mechanical Turk, it is an online labor market where you can work or recruit workers to perform small tasks over the Internet.
  28. Using this platform, we recruit workers to work on our task. We developed this interface where you can see Google Street View imagey, and label, in this case, an obstacle in path.
  29. We showed that this is an effective method, but it is labor intensive.
  30. We showed that this is an effective method, but it is labor intensive.
  31. To more efficiently find accessibility attributes, we turned to computer vision, which is used for applications like face detection.
  32. Different attributes affect sidewalk accessibility for people with mobility impairment. For example, presence of curb ramps, surface conditions, obstacles, steep gradients, and more.
  33. And removed even more errors
  34. And removed even more errors
  35. Computer vision is not perfect. And there are false positives, which can be fixed by verification. It misses curb ramps, and humans need to label these.
  36. Here you see detected curb ramps as green boxes on top of the Street View image (to the next slide to play).
  37. The question is, can we achieve same or better accuracy as a system with a lower time cost compared to manual labeling. 5 min
  38. To do this, we developed a system called Tohme. It combines the two approach.
  39. This is the overview of the system. A custom web scraper that collects dataset including Street View images. A computer vision based detector finds curb ramps.
  40. So we designed a smart task allocator.
  41. It routes detection results to a cheap manual verification workflow to remove false positive errors. However, since our verification task disallow workers to fix the false negatives, curb ramps that are missed never get detected.
  42. So if the allocator predicts false negative, it then passes tasks to manual labeling workflow.
  43. We get a Street View image.
  44. We run a detector
  45. Then extract features.
  46. Our task allocator predicts presence of false negatives. If it predicts no false negative, then it allocates a task to a verification workflow.
  47. Our task allocator predicts presence of false negatives. If it predicts no false negative, then it allocates a task to a verification workflow.
  48. Another example.
  49. Run a detector
  50. Extract features.
  51. If the allocator predicts false negative, then it passes the task to the labeling workflow.
  52. If the allocator predicts false negative, then it passes the task to the labeling workflow.
  53. Let’s first talk about our web scraper
  54. Let’s first talk about our web scraper
  55. We scraped GSV panoramas and metadata from the intersections. We also scraped their accompanying 3-d point cloud data. As well as top-down Google Maps imagery. These datasets are used to train automatic task allocator. _AUz5cV_ofocoDbesxY3Kw -dlUzxwCI_-k5RbGw6IlEg 0C6PG3Zpuwz11kZKfG_vUg D-2VNbhqOqYAKTU0hFneIw
  56. Because sidewalk infrastructure can vary in design and appearance across cities and countries, we included 4 regions including Washington DC, Baltimore, Los Angeles, and Saskatoon.
  57. We also looked at different types of city areas.
  58. Blue regions represent dense urban areas, and red regions represent residential area.
  59. In all, we had 11.3 square kilometers. There were 1,086 intersections. We found 2,877 curb ramps and 647 missing curb ramps based on the ground truth data. Average Street View image age was 2.2 years old.
  60. (pause) But how well does Street View data reflect the current state of curb ramp infrastructure.
  61. To answer this question, we compared Street View intersections with physical intersections
  62. To answer this question, we compared Street View intersections with physical intersections
  63. First, we physically visited intersections and took multiple pictures. The areas included four subset regions, and it consisted of 273 intersections. We then counted the numbers of curb ramps and missing curb ramps in both dataset, and evaluate their concordance. As a result, we observed over 97% agreement between Google Street View and the real world. A small disagreement due to construction.
  64. Moving on to our dataset
  65. Moving on to our dataset
  66. Moving on to our dataset
  67. To train and evaluate our computer vision program, 2 members of our research team manually labeled curb ramps in Street View images. In total, we collected 2,877 curb ramp labels.
  68. To train and evaluate our computer vision program, 2 members of our research team manually labeled curb ramps in Street View images. In total, we collected 2,877 curb ramp labels.
  69. Our computer vision component has three parts.
  70. Our computer vision component has three parts.
  71. We experimented with various object detection. We chose to build it on top of a framework called DPM, one of the most successful approaches in object detection.
  72. DPM models a target object and its parts with histogram of gradient features. It also models the spatial relationship between the parts.
  73. DPM sweeps through an entire image, and detects areas that look like a curb ramp. Detections are shown in red boxes. Numbers of correct detections and errors are shown in this table. There are some redundant labels such as overlapping boxes. h7ZW0_VasRt3vhevz1mjeg
  74. And there shouldn’t be curb ramps in the sky. h7ZW0_VasRt3vhevz1mjeg
  75. We use non-maxima suppression to remove overlapping labels, and 3D point cloud data to remove curb ramps that are not on ground level. Note, that this 3D data is coarse we cannot identify detailed structure of curb ramps. h7ZW0_VasRt3vhevz1mjeg
  76. We get a cleaner result, but we still have some errors. We try to remove them by utilizing other information such as size of a bounding box and RGB information. h7ZW0_VasRt3vhevz1mjeg
  77. This is the final result with computer vision alone. h7ZW0_VasRt3vhevz1mjeg
  78. I will talk about how we can combine crowdsourcing and automated methods to collect curb ramp data from Google Street View efficiently. Today, how algorithmic work management plays a role in this process.
  79. And removed even more errors
  80. Our curve is less ideal
  81. For our system, we set the confidence threshold to emphasize higher recall than higher precision because false positives are easier to correct
  82. We observed various image properties that could cause computer vision to make errors. Including occlusion, illumination, scale, view point variation, structures similar to curb ramps, and variation in design of curb ramps.
  83. That’s what we do with the task allocator.
  84. That’s what we do with the task allocator.
  85. We used following features. To assess complexity of intersections, we used street cardinality in the meta data.
  86. Depth data
  87. It allows us to estimate a size of a street, which is useful because further the curb ramp, harder to detect.
  88. We also assessed the complexity of each intersection with top-down imagery.
  89. Because looks of curb ramps vary more in irregular intersections, computer vision tend to fail finding curb ramps. For example, the intersection on the right is arguably more complex than the one on the left.
  90. We also used the number of detection boxes, their positions, and confidence to see how confused the computer vision program was.
  91. Our manual labeling tool allows people to control a viewing angle. You select the curb ramp button at the top, and label the target. We collect outline labels of curb ramps to collect rich data to train computer vision.
  92. Let’s talk about the verification task
  93. Let’s talk about the verification task
  94. Here you see detected curb ramps as green boxes on top of the Street View image (to the next slide to play).
  95. The question is, can we achieve same or better accuracy as a system with a lower time cost compared to manual labeling.
  96. We compare the performance of manual labeling without smart task allocation, computer vision plus verification without smart task allocation, and finally Tohme.
  97. We measured accuracy and average task completion time of each workflow.
  98. Turkers completed over 6,300 labeling tasks and 4,800 verification tasks and we used monte carlo simulations for evaluation
  99. On the left, I show accuracy. On the right, I show cost. We want accuracy to be high, and cost to be low.
  100. On the left, I show accuracy. On the right, I show cost. We want accuracy to be high, and cost to be low. For manual labeling approach alone, our accuracy measures are 84 – 86%. 94 seconds per intersection For CV + manual verification, our results dropped substantially but so did the time cost by more than half So, now, for Tohme, here we saw similar accuracies to the manual baseline approach
  101. 217 of 277 tasks correctly routed to svVerify
  102. We compare the performance of manual labeling without smart task allocation, computer vision plus verification without smart task allocation, and finally Tohme.
  103. We compare the performance of manual labeling without smart task allocation, computer vision plus verification without smart task allocation, and finally Tohme.
  104. We measured accuracy and average task completion time of each workflow.
  105. We recruited multiple workers to work on labeling tasks and verification tasks. We evaluated the result with Monte Carlo simulation.
  106. Let’s see how turkers labeled.
  107. In general, their labels were high quality
  108. In general, their labels were high quality
  109. Even with a difficult scene with shadows, they labeled correctly most of the times.
  110. Even with a difficult scene with shadows, they labeled correctly most of the times.
  111. But some times there were errors.
  112. For example this person labeled a drive way as a curb ramp.
  113. And some was a little lazy.
  114. And labeled two curb ramps with a single label.
  115. Here are some examples.
  116. Here are some examples.
  117. With only computer vision, there are false positive detections.
  118. With human verification, errors get corrected.
  119. Based on the shapefile downloaded from data.dc.gov, there are 8,209 intersections in DC Manual labeling: 94s per intersection * 8,209 intersections = Tohme: 81 s per intersection ---- Source: http://data.dc.gov/Metadata.aspx?id=2106
  120. Based on the shapefile downloaded from data.dc.gov, there are 8,209 intersections in DC Manual labeling: 94s per intersection * 8,209 intersections = Tohme: 81 s per intersection ---- Source: http://data.dc.gov/Metadata.aspx?id=2106
  121. (i) Context integration. While we use some context information in Tohme (e.g., 3D-depth data, intersection complexity inference), we are exploring methods to include broader contextual cues about buildings, traffic signal poles, crosswalks, and pedestrians as well as the precise location of corners from top-down map imagery. (ii) 3D-data integration. Due to low-resolution and noise, we currently use 3D-point cloud data as a ground plane mask rather than as a feature to our CV algorithms. We plan to explore approaches that combine the 3D and 2D imagery to increase scene structure understanding (e.g., [28]). If higher resolution depth data becomes available, this may be useful to directly detect the presence of a curb or corner, which would likely improve our results. (iii) Training. Our CV algorithms are currently trained using GSV scenes from all eight city regions in our dataset. Given the variation in curb ramp appearance across geographic areas, we expect that performance could be improved if we trained and tested per city.