This talk was given as part of the Human-Computer Interaction Institute seminar series at Carnegie Mellon University. My host was Professor Jeffrey Bigham. More info here: https://www.hcii.cmu.edu/news/seminar/event/2014/10/characterizing-physical-world-accessibility-scale-using-crowdsourcing-computer-vision-machine-learning
You can download the original PowerPoint deck with videos here:
http://www.cs.umd.edu/~jonf/talks.html
Abstract: Roughly 30.6 million individuals in the US have physical disabilities that affect their ambulatory activities; nearly half of those individuals report using an assistive aid such as a wheelchair, cane, crutches, or walker. Despite comprehensive civil rights legislation, many city streets, sidewalks, and businesses remain inaccessible. The problem is not just that street-level accessibility affects where and how people travel in cities but also that there are few, if any, mechanisms to determine accessible areas of a city a priori.
In this talk, I will describe our research developing novel, scalable data-collection methods for acquiring accessibility information about the built environment using a combination of crowdsourcing, computer vision, and online map imagery (e.g., Google Street View). Our overarching goal is to transform the ways in which accessibility information is collected and visualized for every sidewalk, street, and building façade in the world. This work is in collaboration with University of Maryland Professor David Jacobs and graduate students Kotaro Hara and Jin Sun along with a number of undergraduate students and high school interns.
Characterizing Physical World Accessibility at Scale Using Crowdsourcing, Computer Vision, & Machine Learning
1. Human Computer
Interaction
Laboratory
makeability lab
CHARACTERIZING PHYSICAL WORLD ACCESSIBILITY AT SCALE
USING CROWDSOURCING, COMPUTER VISION, & MACHINE LEARNING
25. Human Computer
Interaction
Laboratory
makeability lab
CHARACTERIZING PHYSICAL WORLD ACCESSIBILITY AT SCALE
USING CROWDSOURCING, COMPUTER VISION, & MACHINE LEARNING
30. The National Council on Disability noted that there is no comprehensive information on “the degree to which sidewalks are accessible” in cities.
National Council on Disability, 2007
The impact of the Americans with Disabilities Act: Assessing the progress toward achieving the goals of the ADA
31. The lack of street-level accessibility information can have a significant impact on the independence and mobility of citizens
cf. Nuernberger, 2008; Thapar et al., 2004
32. I usually don’t go where I don’t know [about accessible routes]
-P3, congenital polyneuropathy
33. “Man in Wheelchair Hit By Vehicle Has Died From Injuries”
-The Aurora, May 9, 2013
39. How might a tool like AccessScore:
Change the way people think about and understand their neighborhoods
Influence property values
Impact where people choose to live
Change how governments/citizens make decisions about infrastructural investments
40. AccessScore would not change how people navigate the city, for this we need a different tool…
42. Routing for: Manual Wheelchair
1st of 3 Suggested Routes 16 minutes, 0.7 miles, 1 obstacle
!
!
!
!
A
B
Route 1
Route 2
Surface Problem
Avg Severity: 3.6 (Hard to Pass)
Recent Comments:
“Obstacle is passable in a manual chair but not in a motorized chair”
Routing for: Manual Wheelchair
A
1st of 3 Suggested Routes 16 minutes, 0.7 miles, 1 obstacle
!
ACCESSIBILITY AWARE NAVIGATION SYSTEMS
45. Safe Routes to School Walkability Audit
Rock Hill, South Carolina
Walkability Audit Wake County, North Carolina
Walkability Audit Wake County, North Carolina
TRADITIONAL WALKABILITY AUDITS
46. Safe Routes to School Walkability Audit
Rock Hill, South Carolina
Walkability Audit
Wake County, North Carolina
Walkability Audit
Wake County, North Carolina
TRADITIONAL WALKABILITY AUDITS
50. Similar to physical audits, these tools are built for in situ reporting and do not support remote, virtual inquiry—which limits scalability
Not designed for accessibility data collection
51. MARK & FIND ACCESSIBLE BUSINESSES
wheelmap.org
axsmap.com
52. MARK & FIND ACCESSIBLE BUSINESSES
wheelmap.org
axsmap.com
Focuses on businesses rather than streets & sidewalks
Model is still to report on places you’ve visited
53. Our Approach: Use Google Street View (GSV) as a massive data source for scalably finding and characterizing street-level accessibility
54. HIGH-LEVEL RESEARCH QUESTIONS
1.Can we use Google Street View (GSV) to find street- level accessibility problems?
2.Can we create interactive systems to allow minimally trained crowdworkers to quickly and accurately perform remote audit tasks?
3.Can we use computer vision and machine learning to scale our approach?
55. ASSETS’12 Poster
Feasibility study + labeling interface evaluation
HCIC’13 Workshop
Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
1st investigation of CV + crowdsourced verification
CHI’13 Large-scale turk study + label validation with wheelchair users
ASSETS’13
Applied to new domain: bus stop accessibility for visually impaired
UIST’14
Crowdsourcing + CV + “smart” work allocation
TOWARDS SCALABLE ACCESSIBILITY DATA COLLECTION
56. ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop
Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
1st investigation of CV + crowdsourced verification
CHI’13
Large-scale turk study + label validation with wheelchair users
ASSETS’13 Applied to new domain: bus stop accessibility for visually impaired
UIST’14
Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
57. ASSETS’12 Poster
Feasibility study + labeling interface evaluation
HCIC’13 Workshop Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
1st investigation of CV + crowdsourced verification
CHI’13
Large-scale turk study + label validation with wheelchair users
ASSETS’13
Applied to new domain: bus stop accessibility for visually impaired
UIST’14 Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
58. ASSETS’12 GOALS:
1.Investigate viability of reapproprating online map imagery to determine sidewalk accessibility via crowd workers
2.Examine the effect of three different interactive labeling interfaces on task accuracy and duration
60. WEB-BASED LABELING INTERFACE
FOUR STEP PROCESS
1. Find and mark accessibility problem
2. Select problem category
3. Rate problem severity
4. Submit completed image
61. WEB-BASED LABELING INTERFACE VIDEO
Video shown to crowd workers before they labeled their first image
http://youtu.be/aD1bx_SikGo
65. DATASET BREAKDOWN
34
29
27
11
19
0
10
20
30
40
No Curb Ramp
Surface Problem
Object in Path
Sidewalk Ending
No Sidewalk Accessibility Issues
Manually curated 100 images from urban neighborhoods in LA, Baltimore, Washington DC, and NYC
66. DATASET BREAKDOWN
34
29
27
11
19
0
10
20
30
40
No Curb Ramp
Surface Problem
Object in Path
Sidewalk Ending
No Sidewalk Accessibility Issues
Manually curated 100 images from urban neighborhoods in LA, Baltimore, Washington DC, and NYC
Used to evaluate false positive labeling activity
67. DATASET BREAKDOWN
34
29
27
11
19
0
10
20
30
40
No Curb Ramp
Surface Problem
Object in Path
Sidewalk Ending
No Sidewalk Accessibility Issues
Manually curated 100 images from urban neighborhoods in LA, Baltimore, Washington DC, and NYC
Used to evaluate false positive labeling activity
This breakdown based on majority vote data from 3 independent researcher labels
77. Object in Path
Curb Ramp Missing
R1
R2
R3
Researcher Label Table
Image Level Analysis
This table tells us what accessibility problems exist in the image
78. Pixel Level Analysis
Labeled pixels tell us where the accessibility problems exist in the image.
79. Why do we care about image level vs. pixel level?
82. Coarse
Precise
Point Location Level
Sub-block Level
Block Level
(Pixel Level)
(Image Level)
Pixel level labels could be used for training machine learning algorithms for detection and recognition tasks
83. Coarse
Precise
Localization Spectrum
Point Location Level
Sub-block Level
Block Level
Class Spectrum
Multiclass
Object in Path
Curb Ramp Missing
Prematurely Ending Sidewalk
Surface Problem
Binary
Problem
No Problem
(Pixel Level)
(Image Level)
TWO ACCESSIBILITY PROBLEM SPECTRUMS
Different ways of thinking about accessibility problem labels in GSV
Coarse
Precise
84. Object in Path
Curb Ramp Missing
R1
R2
R3
Researcher Label Table
Problem
Multiclass label
Binary Label
Sidewalk Ending
Surface Problem
Other
85. To produce a single ground truth dataset, we used majority vote.
86. R1
R2
R3
Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
87. R1
R2
R3
Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
88. R1
R2
R3
Maj. Vote
Researcher Label Table
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
89. ASSETS’12 MTURK STUDY METHOD
Independently posted 3 labeling interfaces to MTurk. Crowdworkers could work with only one interface.
For training, turkers required to watch first 1.5 mins of 3-min instructional video.
Hired ~7 workers per image to explore avg accuracy
Turkers paid ~3-5 cents per HIT. We varied number of images/HIT from 1-10.
90. ASSETS’12 MTURK DESCRIPTIVE RESULTS
Hired 132 unique workers
Worked on 2,325 assignments
Provided a total of 4,309 labels (AVG=1.9/image)
91. MAIN FINDINGS: IMAGE-LEVEL ANALYSIS
0%
20%
40%
60%
80%
100%
Point-and-click
Outline
Rectangle
AVERAGE ACCURACY
Higher is better
0
10
20
30
40
50
Point-and-click
Outline
Rectangle
MEDIAN TASK TIME (SECS)
Lower is better
92. MAIN FINDINGS: IMAGE-LEVEL ANALYSIS
83.0%
82.6%
79.2%
0%
20%
40%
60%
80%
100%
Point-and-click
Outline
Rectangle
AVERAGE ACCURACY
All three interfaces performed similarly. This is without quality control.
0
10
20
30
40
50
Point-and-click
Outline
Rectangle
MEDIAN TASK TIME (SECS)
Higher is better
Lower is better
93. MAIN FINDINGS: IMAGE-LEVEL ANALYSIS
83.0%
82.6%
79.2%
0%
20%
40%
60%
80%
100%
Point-and-click
Outline
Rectangle
32.9
41.5
43.3
0
10
20
30
40
50
Point-and-click
Outline
Rectangle
AVERAGE ACCURACY
MEDIAN TASK TIME (SECS)
All three interfaces performed similarly. This is without quality control.
Point-and-click is the fastest; 26% faster than Outline & 32% faster than Rectangle
Higher is better
Lower is better
94. ASSETS’12 CONTRIBUTIONS:
1.Demonstrated that minimally trained crowd workers could locate and categorize sidewalk accessibility problems in GSV images with > 80% accuracy
2.Showed that point-and-click fastest labeling interface but that outline faster than rectangle
95. ASSETS’12 Poster
Feasibility study + labeling interface evaluation
HCIC’13 Workshop
Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
1st investigation of CV + crowdsourced verification
CHI’13
Large-scale turk study + label validation with wheelchair users
ASSETS’13
Applied to new domain: bus stop accessibility for visually impaired
UIST’14
Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
96. ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop
Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
1st investigation of CV + crowdsourced verification
CHI’13
Large-scale turk study + label validation with wheelchair users
ASSETS’13
Applied to new domain: bus stop accessibility for visually impaired
UIST’14
Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
97. CHI’13 GOALS:
1.Expand ASSETS’12 study with larger sample.
•Examine accuracy as function of turkers/image
•Evaluate quality control mechanisms
•Gain qualitative understanding of failures/successes
2.Validate researcher ground truth with labels from three wheelchair users
99. CHI’13 GOALS:
1.Expand ASSETS’12 study with larger sample.
•Examine accuracy as function of turkers/image
•Evaluate quality control mechanisms
•Gain qualitative understanding of failures/successes
2.Validate researcher ground truth with labels from three wheelchair users
102. IN-LAB STUDY METHOD
Three wheelchair participants
Independently labeled 75 of 229 GSV images
Used think-aloud protocol. Sessions were video recorded
30-min post-study interview
We used Fleiss’ kappa to measure agreement between wheelchair users and researchers
103. Here is an example recording from the study session
104.
105. IN-LAB STUDY RESULTS
Strong agreement (κmulticlass=0.74) between wheelchair participants and researcher labels (ground truth)
In interviews, one participant mentioned using GSV to explore areas prior to travel
106. CHI’13 GOALS:
1.Expand ASSETS’12 study with larger sample.
•Examine accuracy as function of turkers/image
•Evaluate quality control mechanisms
•Gain qualitative understanding of failures/successes
2.Validate researcher ground truth with labels from three wheelchair users
107. CHI’13 GOALS:
1.Expand ASSETS’12 study with larger sample.
•Examine accuracy as function of turkers/image
•Evaluate quality control mechanisms
•Gain qualitative understanding of failures/successes
2.Validate researcher ground truth with labels from three wheelchair users
108. CHI’13 MTURK STUDY METHOD
Similar to ASSETS’12 but more images (229 vs. 100) and more turkers (185 vs. 132)
Added crowd verification quality control
Recruited 28+ turkers per image to investigate accuracy as function of workers
109. University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
Kotaro Hara
Timer: 00:07:00 of 3 hours
10
3 hours
Labeling Interface
110. Kotaro Hara
Timer: 00:07:00 of 3 hours
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
3 hours
10
Verification Interface
111. Kotaro Hara
Timer: 00:07:00 of 3 hours
University of Maryland: Help make our sidewalks more accessible for wheelchair users with Google Maps
3 hours
10
Verification Interface
112. CHI’13 MTURK LABELING STATS
Hired 185 unique workers Worked on 7,517 labeling tasks (AVG=40.6/turker) Provided a total of 13,379 labels (AVG=1.8/image)
Hired 273 unique workers
Provided a total of 19,189 verifications
CHI’13 MTURK VERIFICATION STATS
113. CHI’13 MTURK LABELING STATS
Hired 185 unique workers
Worked on 7,517 labeling tasks (AVG=40.6/turker)
Provided a total of 13,379 labels (AVG=1.8/image)
CHI’13 MTURK VERIFICATION STATS
Hired 273 unique workers
Provided a total of 19,189 verifications
Median image labeling time vs. verification time: 35.2s vs. 10.5s
114. CHI’13 MTURK KEY FINDINGS
81% accuracy without quality control
93% accuracy with quality control
127. TURKER LABELING ISSUES
Overlabeling
Some Turkers Prone to High False Positives
No Curb Ramp
128. No Curb Ramp
TURKER LABELING ISSUES
Overlabeling
Some Turkers Prone to High False Positives
Incorrect Object in Path label. Stop sign is in grass.
129. TURKER LABELING ISSUES
Overlabeling
Some Turkers Prone to High False Positives
Surface Problems
130. TURKER LABELING ISSUES
Overlabeling
Some Turkers Prone to High False Positives
Surface Problems
Tree not actually an obstacle
131. TURKER LABELING ISSUES
Overlabeling
Some Turkers Prone to High False Positives
No problems in this image
133. T1
T2
T3
Maj. Vote
3 Turker Majority Vote Label
Object in Path
Curb Ramp Missing
Sidewalk Ending
Surface Problem
Other
T3 provides a label of low quality
134. To look into the effect of turker majority vote on accuracy, we had 28 turkers label each image
135. 28 groups of 1:
We had 28 turkers
label each image:
136. 28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
137. 28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
138. 28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
139. 28 groups of 1:
We had 28 turkers
label each image:
9 groups of 3:
5 groups of 5:
4 groups of 7:
3 groups of 9:
149. CHI’13 CONTRIBUTIONS:
1.Extended and reaffirmed findings from ASSETS’12 about viability of GSV and crowd work for locating and categorizing accessibility problems
2.Validated our ground truth labeling approach
3.Assessed simple quality control approaches
150. ASSETS’12 Poster
Feasibility study + labeling interface evaluation
HCIC’13 Workshop
Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
1st investigation of CV + crowdsourced verification
CHI’13
Large-scale turk study + label validation with wheelchair users
ASSETS’13
Applied to new domain: bus stop accessibility for visually impaired
UIST’14
Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
151. ASSETS’12 Poster Feasibility study + labeling interface evaluation
HCIC’13 Workshop
Exploring early solutions to computer vision (CV)
HCOMP’13 Poster
1st investigation of CV + crowdsourced verification
CHI’13
Large-scale turk study + label validation with wheelchair users
ASSETS’13
Applied to new domain: bus stop accessibility for visually impaired
UIST’14
Crowdsourcing + CV + “smart” work allocation
TODAY’S TALK
152. All of the approaches so far relied purely on manual labor, which limits scalability
157. svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb Ramp Detection
svControl
Automatic Task Allocation
svVerify
Manual Label Verification
svLabel
Manual Labeling
Tohme
遠目
Remote Eye
・
Design Principles
1.Computer vision is cheap (zero cost)
2.Manual verification is far cheaper than manual labeling
3.Automatic curb ramp detection is hard and error prone
4.Fixing a false positive is easy, fixing a false negative is hard (requires manual labeling).
158.
159. The “lack of curb cuts is a primary obstacle to the smooth integration of those with disabilities into the commerce of daily life.”
Kinney et al. vs. Yerusalim & Hoskins, 1993
3rd Circuit Court of Appeals
160. “Without curb cuts, people with ambulatory disabilities simply cannot navigate the city”
Kinney et al. vs. Yerusalim & Hoskins, 1993
3rd Circuit Court of Appeals
165. svCrawl
Web Scraper
Dataset
svDetect
Automatic Curb Ramp Detection
svControl Automatic Task Allocation
svVerify
Manual Label Verification
Tohme
遠目
Remote Eye
・
svVerify can only fix false positives, not false negatives! That is, there is no way for a worker to add new labels at this stage!
177. Washington D.C.
Baltimore
Los Angeles
Saskatoon
* At the time of downloading data in summer 2013
Scraper & Dataset
Total Area: 11.3 km2
Intersections: 1,086
Curb Ramps: 2,877
Missing Curb Ramps: 647
Avg. GSV Data Age: 2.2 yrs
178. How well does GSV data reflect the current state of the physical world?
181. Washington D.C.
Baltimore
Physical Audit Areas
GSV and Physical World > 97.7% agreement
273 Intersections
Dataset | Validating Dataset
Small disagreement due to construction.
182. Washington D.C.
Baltimore
Physical Audit Areas
273 Intersections
> 97.7% agreement
Dataset
Key Takeaway
Google Street View is a viable source of curb ramp data
186. Deformable Part Models
Felzenszwalb et al. 2008
Automatic Curb Ramp Detection
http://www.cs.berkeley.edu/~rbg/latent/
187. Deformable Part Models
Felzenszwalb et al. 2008
Automatic Curb Ramp Detection
http://www.cs.berkeley.edu/~rbg/latent/
Root filter
Parts filter
Displacement cost
188. Automatic Curb Ramp Detection
Multiple redundant detection boxes
Detected Labels Stage 1: Deformable Part Model
Correct
1
False Positive
12
Miss
0
189. Automatic Curb Ramp Detection
Curb ramps shouldn’t be in the sky or on roofs
Correct
1
False Positive
12
Miss
0
Detected Labels Stage 1: Deformable Part Model
191. Automatic Curb Ramp Detection
Detected Labels Stage 3: SVM-based Refinement
Filter out labels based on their size, color, and position.
Correct
1
False Positive
5
Miss
0
196. Some curb ramps never get detected
False positive detections
Automatic Curb Ramp Detection
Correct
6
False Positive
4
Miss
1
197. Some curb ramps never get detected
False positive detections
Automatic Curb Ramp Detection
Correct
6
False Positive
4
Miss
1
These false negatives are expensive to correct!
205. Occlusion
Illumination
Scale
Viewpoint Variation
Structures Similar to Curb Ramps
Curb Ramp Design Variation
Automatic Curb Ramp Detection
CURB RAMP DETECTION IS A HARD PROBLEM
206. Can we predict difficult intersections & CV performance?
209. Automatic Task Allocation | Features to Assess Scene Difficulty for CV
Number of connected streets from metadata
Depth information for intersection complexity analysis
Top-down images to assess complexity of an intersection
Number of detections and confidence values
217. Automatic Detection and Manual Verification
Automatic Task Allocation
Can Tohme achieve equivalent or better accuracy at a lower time cost compared to a completely manual approach?
218. STUDY METHOD: CONDITIONS
Manual labeling without smart task allocation
&
vs.
CV + Verification without smart task allocation
Tohme
遠目
Remote Eye
・
vs.
Evaluation
220. Recruited workers from Mturk
Used 1,046 GSV images (40 used for golden insertion)
Evaluation
STUDY METHOD: APPROACH
221. RESULTS
Labeling Tasks
Verification Tasks
# of distinct turkers:
242
161
1,270
582
# of HITs completed:
# of tasks completed:
6,350
4,820
# of tasks allocated:
769
277
Evaluation
We used Monte Carlo simulations for evaluation
222. 84%
88%
86%
0%
20%
40%
60%
80%
100%
Accuracy Measures (%)
Precision
Recall
F-measure
Manual Labeling
CV and Manual Verification
&
94
0
20
40
60
80
100
Task Completion Time / Scene (s)
Manual Labeling
CV and Manual Verification
&
Tohme
遠目
Remote Eye
・
Tohme
遠目
Remote Eye
・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
ACCURACY
COST (TIME)
223. 84%
68%
88%
58%
86%
63%
0%
20%
40%
60%
80%
100%
Accuracy Measures (%)
Precision
Recall
F-measure
Manual Labeling
CV and Manual Verification
&
94
42
0
20
40
60
80
100
Task Completion Time / Scene (s)
Manual Labeling
CV and Manual Verification
&
Tohme
遠目
Remote Eye
・
Tohme
遠目
Remote Eye
・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
ACCURACY
COST (TIME)
224. 84%
68%
83%
88%
58%
86%
86%
63%
84%
0%
20%
40%
60%
80%
100%
Accuracy Measures (%)
Precision
Recall
F-measure
Manual Labeling
CV and Manual Verification
&
94
42
81
0
20
40
60
80
100
Task Completion Time / Scene (s)
Manual Labeling
CV and Manual Verification
&
Tohme
遠目
Remote Eye
・
Tohme
遠目
Remote Eye
・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
ACCURACY
COST (TIME)
225. 84%
68%
83%
88%
58%
86%
86%
63%
84%
0%
20%
40%
60%
80%
100%
Accuracy Measures (%)
Precision
Recall
F-measure
Manual Labeling
CV and Manual Verification
&
94
42
81
0
20
40
60
80
100
Task Completion Time / Scene (s)
Manual Labeling
CV and Manual Verification
&
Tohme
遠目
Remote Eye
・
Tohme
遠目
Remote Eye
・
Evaluation | Labeling Accuracy and Time Cost
Error bars are standard deviations.
13% reduction in cost
ACCURACY
COST (TIME)
226. svControl
Automatic Task Allocation
svVerify
Manual Label Verification
svLabel
Manual Labeling
Evaluation | Smart Task Allocator
~80% of svVerify tasks were correctly routed
~50% of svLabel tasks were correctly routed
227. svControl
Automatic Task Allocation
svVerify
Manual Label Verification
svLabel
Manual Labeling
Evaluation | Smart Task Allocator
If svControl worked perfectly, Tohme’s cost would drop to 28% of a manually labelling approach alone.
244. UIST’14 CONTRIBUTIONS:
1.First CV system for automatically detecting curb ramps in images
2.Showed that automated methods could be used to improve labeling efficiency for curb ramps
3.Validated GSV as a viable curb ramp dataset
245. TOWARDS SCALABLE ACCESSIBILITY DATA COLLECTION
ASSETS’12 Poster
Feasibility study + labeling interface evaluation
HCIC’13 Workshop
Exploring early solutions to computer vision (CV)
HCOMP’13 Poster 1st investigation of CV + crowdsourced verification
CHI’13
Large-scale turk study + label validation with wheelchair users
ASSETS’13
Applied to new domain: bus stop accessibility for visually impaired
UIST’14
Crowdsourcing + CV + “smart” work allocation
The Future
248. 8,209
Intersections in DC
BACK OF THE ENVELOPE CALCULATIONS
Manually labeling GSV with our custom interfaces would take 214 hours With Tohme, this drops to 184 hours We think we can do better Unclear how long a physical audit would take
249. FUTURE WORK: COMPUTER VISION
Context integration & scene understanding 3D-data integration Improve training & sample size Mensuration
254. FUTURE WORK: ADDITIONAL SURVEYING TECHNIQUES
Transmits real-time imagery of physical space along with measurements
255. THE CROWD-POWERED STREETVIEW ACCESSIBILITY TEAM!
Kotaro Hara
Jin Sun
Victoria Le
Robert Moore
Sean Pannella
Jonah Chazan
David Jacobs
Jon Froehlich
Zachary Lawrence
Graduate Student
Undergraduate
High School
Professor
256. Flickr User: Pedro Rocha
https://www.flickr.com/photos/pedrorocha/3627562740/
Flickr User: Brooke Hoyer
https://www.flickr.com/photos/brookehoyer/14816521847/
Flickr User: Jen Rossey
https://www.flickr.com/photos/jenrossey/3185264564/
Flickr User: Steven Vance https://www.flickr.com/photos/jamesbondsv/8642938765
Flickr User: Jorge Gonzalez
https://www.flickr.com/photos/macabrephotographer/6225178809/
Flickr User: Mike Fraser https://www.flickr.com/photos/67588280@N00/10800029263//
PHOTO CREDITS
Flickr User: Susan Sermoneta
https://www.flickr.com/photos/en321/344387583/
257. This work is supported by:
Faculty Research Award
Human Computer
Interaction
Laboratory
makeability lab
258. Human Computer
Interaction
Laboratory
makeability lab
CHARACTERIZING PHYSICAL WORLD ACCESSIBILITY AT SCALE
USING CROWDSOURCING, COMPUTER VISION, & MACHINE LEARNING