A Technology Demonstration for Assessment of Unpaved Road Condition with High-Resolution Aerial Remote Sensing

1. www.mtri.org
A Technology Demonstration for Assessment
of Unpaved Road Condition with
High-Resolution Aerial Remote Sensing

2. www.mtri.org
Colin N. Brooks, Michigan Tech Research Institute
cnbrooks@mtu.edu 734-604-4196 (cell/texting)
Demonstration: Rapid City, South Dakota
October 20, 2015
A Technology Demonstration for
Assessment of Unpaved Road
Condition with
High-Resolution Aerial Remote
Sensing

3. Introductions & Welcome
Integrated Global Dimensions (Valerie Lefler)
Michigan Tech Research Institute (MTRI team – Colin
Brooks, Rick Dobson, David Dean)
Michigan Tech Center for Technology & Training (CTT
– Dr. Tim Colling)
3

4. Agenda
4

5. Your Time is Appreciated!
5

6. Demonstration Overview
Research Goal
– Key Objectives
– Road Characteristics
– Requirements
Implementation
– Equipment
– Analysis
– Software
Next Steps
– Cost
– Regulations
– Other Applications
6

7. www.mtri.org
RESEARCH GOAL

8. Project Goal
Phase 1: Extend available Commercial Remote Sensing and Spatial
Information (CRS&SI) tools to enhance and develop an unpaved road
assessment system by developing a sensor for, and demonstrating the utility
of remote sensing platform(s) for unpaved road assessment.
Phase 2: Take our working prototype technologies from a useful,
successfully demonstrated research-level tool to a commercially-available,
implemented system available to transportation agencies for objective
unpaved road assessment on a day-to-day, as needed basis.
8

9. Original Project Objectives
1. Commercially viable in that it can measure inventory and distress
data at a rate and cost competitive with traditional methods
2. Rapid ID & characterization of unpaved roads
3. Inventory level with meaningful metrics
4. Develop a sensor for, & demonstrate the utility of remote sensing
platform(s) for unpaved road assessment
5. Platform could be a typical manned fixed-wing aircraft, UAV, or
both; depends on relative strengths & weaknesses in meeting
user community requirements
6. Simplify mission planning, control of sensor system, & data
processing fitting for a commercial entity or large transportation
agency
7. Demonstrate prototype system(s) to stakeholders for potential
implementation developed through best engineering practices
8. Develop a decision support system to aid the user in asset
management and planning
9

10. New Outreach & Implementation Phase:
Additional Project Activities
Update and maintain www.mtri.org/unpaved project website with focus on
professional outreach materials.
Organize and meet with the Technical Advisory Committee for input on
moving our Aerial Unpaved Road Assessment (AURA) system to full
commercial implementation and providing advice on related project direction.
Provide a state of the practice for available UAVs and systems that could be
used for AURA
Prepare the AURA remote sensing system (RSPS) project to be moved
towards an operational commercial-ready product.
– Investigate additional assessment functionality for transportation agencies such as road
geometry evaluations.
– Includes new task of assessing capabilities of fixed-wing UAVs
Develop and provide professional outreach for implementation to a
nationwide audience.
Work closely with the private sector to commercialize AURA for day-to-
day operation in light of new FAA regulations to likely to take effect in
2016/2017.
10

11. Road Characteristic Analysis Detail
11
– Surface Width
• Collected every 10', with a precision of
+/- 4”
– Cross Section (Loss of Crown)
• Facilitates drainage, typically 2% - 4%
(up to 6%) vertical change, sloping away
from the centerline to the edge
• Measure the profile every 10' along the
road direction, able to detect a 1%
change across a 9'-wide lane
– Potholes
• <1', 1'-2', 2'-3', >3‘ width bins
• <2”, 2”-4”, >4” depth bins
– Ruts
• Detect features >5”, >10' in length,
precision +/-2”

12. Road Characteristic Analysis Detail Cont.
– Corrugations (Washboarding)
• Classify by depth to a precision of +/-1”
– <1”, 1”-3”, >3”
• Report total area of the reporting
segment affected
– Roadside Drainage
• System should be able to measure ditch
bottom relative to road surface within
+/-2”, if >6”
• Detect the presence of water, elevation
+/-2”, width +/-4”
– Float Aggregate (Berms)
12

13. Combined Methods: Dept. Army
Unsurfaced Road Condition Index (URCI)
Representative Sample Segments
2 Part Rating System
– Density
• Percentage of the Sample Area
– Severity
• Low, Medium, High
Clear Set of Measurement Requirements
Realistic Possibility of Collecting Most of
the Condition Indicator Parameters
Potential Applicability to a Wide Variety of
U.S. Unpaved Roads
Endorsed by TAC as Effective Rating
System
13

14. Combined Methods: Dept. Army
Unsurfaced Road Condition Index
Decision Matrix from Distress Criteria (Eaton 1987a)
14

15. Summary of Requirements
15
Number Name Type Definition
1 Data Collection Rate Sensor The systems must collect data at a rate that is competitive
with current practice (to be determined, TBD)
2 Data Output Rate System Processed outputs from the system will be available no
later than 5 days after collection
3 Sensor Operation Sensor “easy”, little training required
4 Platform Operation Platform Training needed TBD, based on platform choice
5 Reporting Segment System <100ft x 70ft, with location precision of 10ft. Map position
accuracy +/- 40ft
6 Sample locations System Specified by the user a map waypoints
7 Inventory System A classified inventory of road types is required prior to
system operation. This will consist of 3 classes: Paved,
Gravel, Unimproved Earth
8 Surface Width System This is part of the inventory, and may also be estimated by
the system measured every 10ft, precision of +/- 4”
9 Cross Section Distress Estimate every 10ft, able to detect 1” elevation change in
9’, from center to edge.
10 Potholes Distress Detect hole width >6”, precision +/-4”, hole depth >4”,
precision +/-2”. Report in 4 classes: <1’, 1’-2’, 2’-3’, >3’

16. Summary of Requirements Cont.
16
precision +/-2”. Report in 4 classes: <1’, 1’-2’, 2’-3’, >3’
11 Ruts Distress Detect >5” wide x 10’ long, precision +/-2”
12 Corrugations Distress Detect spacing perpendicular to direction of travel >8” -
<40”, amplitude >1”. Report 3 classes: <1”, 1”-3”, >3”.
Report total surface area of the reporting segment
exhibiting these features
13 Roadside Drainage Distress Detect depth >6” from pavement bottom, precision +/-2”,
every 10ft. Sense presence of standing water, elevation
precision +/-2”, width precision +/-4”
14 Loose Aggregate Distress Detect berms in less-traveled part of lane, elevation
precision +/-2”, width +/-4”
15 Dust Distress Optional – measure opacity and settling time of plume
generated by pilot vehicle

17. Demonstration Overview
Research Goal
– 8 Key Objectives
– Road Characteristics
– Requirements
Implementation
– Equipment
– Analysis
– Software
Next Steps
– Cost
– Regulations
– Other Applications
17

18. www.mtri.org
IMPLEMENTATION

19. Equipment Platforms
19
Bergen Helicopter (1st year platform)
– Total flight time: 16 minutes (not including 2 minute reserve); flight time for a
200 meter (650’) section ~ 4 minutes.
– Flown at 2 m/s at 25 and 30 meters (4.5 mph @ 80-100 feet)
– 50mm prime lens

20. Equipment Platforms
20
Cessna 172 and 152 Aircraft
– Average air speed: 65 knots (~ 75 mph)
– Flown at altitudes of 500 and 1000 feet
– 105 mm prime lens (2012), 70-200mm zoom (2013)
– Not able to provide sub-inch 3D reconstruction (angular diversity
issue)

21. Equipment Platforms
21
Bergen Hexacopter
– Total flight time: up to 30 minutes with small payloads
– Weight: 4kg unloaded
– Maximum Payload: 5kg
– Includes autopilot system, stabilized mount that is independent of platform
movement, and first person viewer system (altitude, speed, battery life,
etc.)
Success!

22. Selected Sensor: Nikon D800
Nikon D800 – Full-Sized
(FX) Sensor
36.3 Megapixels
4 Frames per Second
Cost: $3,000
Evaluating Sony A7R
Mirrorless camera
Same cost/resolution, half
the weight.
22

23. Manned Aerial Sensor Performance
Algorithm performance and the ability to
meet the stringent requirements on
resolution, depends on the ability to
collect data that has enough angular
diversity to be able to reconstruct three
dimensions from two dimensions.
– As distance from the ground increases,
the solid angle that an object subtends
decreases, and at some point, becomes
too small for high-res reconstruction.
– Data taken from an altitude of 500 feet do
not meet the system requirements in
resolution. This is due to the lack of
sufficient angular diversity.
23

24. Airframe Recommendation - updated
24
 Airframe recommendations
• Multirotor systems recommended
– Bergen Hexacopter still meets
requirements
– Other airframes in selected 6 and 8
rotor configurations also meet
requirements (“quad-8” below –
heavier lift, more rotors – more
redundancy)
– details in Project Deliverable 5-B
(Updated Requirements, Feb. 2015)

25. Fixed-wing UAV options –
ongoing evaluation
Can fly for longer, further, but carries a lighter payload (lower resolution 18mp point & shoot
camera vs. 36mp DSLR) – different systems can be right for different needs
– Partnering with Dr. Jarlath O’Neil-Dunne, Univ. Vermont, also funded by USDOT
Currently evaluating the tradeoffs of flight time vs. resolution
25
Sensefly eBee system – RTK GPS version, 40 min flight time - $51k
Orthoimage from Sensefly eBee system
MTRI fixed wing tests, Oct. 2014

26. Performance – Collected Imagery
26
Taken from 25m

27. Ground Truth
27

28. 3D Reconstruction – Initial & Densified
Point Cloud
28

29. 3D Reconstruction – Surface
29

30. 3D Model of Rutting
30

31. 3D of Piotter Rd (Hexacopter, 27 images)
31

32. 3D of an Iowa Road (Hexacopter, 18
images)
32

33. 3D Data Examples
Important to categorizing distresses by severity
Obtaining 0.9 cm ground sample distance
33

34. Distress Detection – Potholes
Canny Edge Detection Used to Locate Edges
Hough Circle Transform is Used to Locate Potholes
34
Note: circles near edges ignored.
Edge Detection Identified Circles

35. Distress Detection – Washboarding
35

36. Distress Detection – Washboarding
36
Ground Truth Corrugation Area:
19.6 sq. m
Computed Corrugation Area:
17.2 sq. m
Missing
due to
area
threshold

37. Crown estimation – better than 1%
37

38. June, 2014 technical demo
in Sioux Falls, SD
Successful demonstration to South Dakota DOT &
local transportation agencies in Sioux Falls, June
26, 2014
– 36 attendees, 15 groups (state & local agencies) – SDDOT,
SDLTAP, 3 local counties, Nebraska LTAP, ND LTAP
– Overview, live field demo, quick data processing, round table
discussion
– Feedback:
• Definite improvement over “windshield surveys”
• A practical system – “see tremendous use”
• Strong interest in accessing as a third-party service
– Other interests expressed: Haul road inventories, road
geometry evaluations, encroachment issues, natural disaster
documentation
Partnered with outreach specialist (Valerie Lefler,
Integrated Global Dimensions - IGD) – organized
June demonstration workshop
– www.mtriaura.com (updated)

39. July, 2015 technical demo in Salina, KS
39

40. Algorithm Performance Summary
Data Collection Parameters Meet System Performance
Requirements:
1. 24mp to 36mp Sensor
2. 50mm, f/1.4 Lens Set at f/2.8
3. 1/250s (max) Shutter Speed (shorter is better)
4. ISO Set as Needed for Proper Exposure Given
Ambient Lighting
5. Distance of 20m-30m from Surface
6. 2m/s (max) Forward Speed
7. 2fps (min) Image Capture Rate
8. 64GB High-Speed Storage Medium
40

41. Pothole Detection: 96% accuracy
Crown Damage: within 0.1%
Rut Detection: (being updated)
Corrugation Detection: (being updated)
Algorithm Performance Summary
41

42. Road Analysis Process Flow – RoadSoft DSS
integration
42
3
Identify
Unsurfaced
Road Network
Maintenance
Plan & Budget
Surface Identification
Data From Images
Surface Identification
Manual Inspection
4
Identify Sample
Locations In
Flight System
5
Fly Data
Collection
Sorties with
Platform
Distress Data
From Platform
7
Compile
Distress and
Inventory Data
For Samples
Distress Data From
Manual Inspection
1
Collect Aerial
Imagery
2
Aerial Imagery
Analysis
8
Assign Samples
to Represent
Network
9
DSS Analysis of
Data
10
Selection of
Candidates &
Scheduling
11
Record
Competed Work
Field Report
Completed
Project History
Network
Condition
Report
Functions in
RoadSoft
12
Determine Data
Needs and Repeat
Cycle
Functions in
eCognition
Functions in
Platform System
6
Data Processing

43. RoadSoft GIS
43

44. Analyzed Data Integrated into RoadSoft
GIS Decision Support System
44

45. DSS Ranking System
45

46. All of these together – components of the
AURA system!
Aerial Unpaved Road Assessment (AURA) system
www.mtri.org/unpaved (project details site)
www.auramtri.com (public outreach site)
46

47. Commercialization of AURA
for Day-to-Day Usage
Outputs
– Inclusion of private sector inputs on the best practices for third-
party commercialization of AURA
– Working with Woolpert, Inc. of Dayton, Ohio
• Provide expertise on commercialization potential form a business
perspective
• Expertise in commercial UAV deployment
Woolpert to write “Commercialization Report on AURA
for Day-to-Day Operations”
47

48. Demonstration Overview
Research Goal
– 8 Key Objectives
– Road Characteristics
– Requirements
Implementation
– Equipment
– Analysis
– Software
Next Steps
– Cost
– Regulations
– Other Applications
48

49. www.mtri.org
NEXT STEPS

50. Costs – Manual Characterization
50

51. Costs – Remote Sensing
Manned Fixed Wing:
• Cost per mile rated $54.47 per mile assessed for up to five sites
per mile
• $10.26 per mile (generous assumption of continuous data
collection)
• $16,340 for same type of analysis as listed above
51

52. Costs – Remote Sensing
UAV, high-resolution camera, and good-quality lens:
• Cost per mile rated $30,590/yr/1575 mi/yr = $19.42/mi rated.
• HOWEVER…two 100-foot measured segments represent one
mile of road, so 5,280 ft/200ft is 26.4. Therefore each mile of
measured road represents a road network 26 times larger.
• Therefore cost is $0.74 per mile, in addition to the cost of
vehicle use ($0.55/mi)
– 8 hours/day, 3 days/week, 21 week season to collect 300
road-miles of data segments
Caution must be made for cost comparisons between remote sensing and
manual characterization of road conditions due to the resolutions of the
outputs; centimeter-by-centimeter analysis of entire road segments is
essentially impossible via manual inspection.
52

53. FAA Regulation Situation
Current FAA regulations do not adequately address UAS operations
for private entities.
– The FAA document (14 CFR Part 91) specifically excludes individuals or
companies flying model aircraft for business (commercial) purposes. Commercial
exemptions (Sec. 333) are available now
– As of June 9, 548 commercial exemptions approved, more every week
– For public entities (such as the USDOT), the process of operating a UAS had
involved obtaining a Certificate of Authorization (COA) for a particular mission.
• Some agencies with COAs were able to get them reapproved within
relatively short time periods (< 1 month).
– Dec. 2013 5-year FAA UAV integration RoadMap
– New “sUAS” (small UAV) draft rules released Feb 2015
Likely approval of sUAS rules in 2016/2017
– Line of sight operation (for now)
– 55 lbs max gross weight
– UAS operator permit (no pilot certificate) - ~$300 cost
– No airworthiness certificate required
53

54. Next steps in light of changing FAA
regulations
FAA Section 333 program has enabled over 1,700 commercial
exemptions for use of small UAVs since Dec. 2014 (3-4 months for initial
approval) – up from 548 in July, 2015!
New “Small UAS” (sUAS) rules proposed by FAA Feb. 2015, comment
period closed Apr. 2015, implementation in 2016/2017 – $300 operators
permit instead of pilot’s license, line of sight, rapid deployment – more
practical, 55 lbs max gross weight, no manned aircraft certificate
Beyond line of sight testing efforts under FAA Pathfinder program
(BNSF – railroads, CNN – newsgathering, PrecisionHawk - agriculture)
Continued need for R&D efforts – integrating multiple sensors, testing
capabilities of new platforms (fixed wing), automated feature detection,
converting data into useful information (algorithm development)
– Applied research services – MTRI interested in partnering
– Commercializing MTRI efforts – looking for partners in all parts of the country
(road condition assessment, bridge condition evaluation)
54

55. Inventory: Surface Type
 How many miles of unpaved road are there? Not all counties have this.
 Need to able to determine inventory
 c. 43,000 (1984 estimate) – but no up-to-date, accurate state inventory exists
 c. 800 miles in Oakland County estimate
 We are extracting this from recent, high-resolution aerial imagery, focusing on
unincorporated areas – attribute existing state Framework roads layer
 Completed Oakland, Monroe, Livingston, St. Clair, Macomb, Washtenaw,
Counties; shared with SEMCOG, adding to RoadSoft GIS asset management
tool
 87%-94% accuracy
 Ex: Livingston Co.: 894 miles unpaved
 1289 miles unpaved
55

56. Unpaved Road Detection Results
Users Producers Overall
Unpaved 93.9% 77.5% 94.3%
Paved 94.3% 98.7%
Monroe County
Accuracy Assessment
at 30% coverage
Mileage
Paved 1390.0
Unpaved 351.9
Total
Mileage 1741.9

57. Integration of Unpaved Road Inventory
Results with RoadSoft GIS
57

58. Demonstration Overview
Research Goal
– 8 Key Objectives
– Road Characteristics
– Requirements
Implementation
– Equipment
– Analysis
– Software
Next Steps
– Cost
– Regulations
– Other Applications
58

59. Questions?
59

60. Travel to Test Collection Site
60

61. Rest of Day Overview
1:00: Welcome, Brief Overview of Research Development,
System Capabilities & Features (Colin Brooks, MTRI)
2:00: Travel to Demonstration Site
2:15: Live Demonstration of Data Collection (Rick
Dobson, MTRI)
3:30: Review of Data Collection & Asset Management
Decision Support Systems
– Dr. Tim Colling, Michigan Tech Transportation Institute, Michigan
Tech University
– Rick Dobson, MTRI
4:30: Panel Discussion of Costs & Implementation,
Wrapup
– Colin Brooks, Valerie Lefler, others 61

62. Questions?
62

63. Panel Discussion
Colin Brooks
– Michigan Tech Research Institute (Michigan Technological University) – Project
PI
Tim Colling, P.E.
– Center for Technology and Training (Michigan Technological University) –
Project Co-PI
Valerie Lefler, M.P.A.
– President, CEO – Integrated Global Dimensions, LLC
Local agency representative (TBD)
63

64. Thank you!
64

65. 65
Contact Info
Colin Brooks cnbrooks@mtu.edu
Desk: 734-913-6858, Mobile: 734-604-4196
Michigan Tech Research Institute, MTRI
3600 Green Court, Suite 100
Ann Arbor, MI 48105
www.mtri.org
Tim Colling, Ph.D., P.E. tkcollin@mtu.edu
Chris Roussi croussi@mtu.edu
Rick Dobson rjdobson@mtu.edu
Ben Hart behart@mtu.edu
Joe Garbarino jgarb@mtu.edu
David Dean dbdean@mtu.edu
Valerie Lefler, M.P.A. valerie.lefler@integratedglobaldimensions.com
www.mtri.org/unpaved
Project #: RITARS-11-H-MTU1

66. 66
DISCLAIMER: The views, opinions, findings and
conclusions reflected in this presentation are the
responsibility of the authors only and do not
represent the official policy or position of the
USDOT/OST-R, or any State or other entity.

67. www.mtri.org
Evaluating the Use of
Unmanned Aerial Vehicles
for Transportation Purposes
MDOT research project, contract no. 2013-
067, Auth. No. 1, OR13-008
Michigan Tech team members: Colin Brooks (cnbrooks@mtu.edu, 734-604-4196),
Thomas Oommen, Timothy C. Havens, Theresa M. Ahlborn, Richard J. Dobson, Dave
Dean, Ben Hart, Chris Roussi, Nate Jesse, Rudiger Escobar Wolf, Michelle Wienert,
Blaine Stormer, John Behrendt
MDOT program manager: Steve Cook; MDOT Research Manager: André Clover
http://www.mtri.org/mdot_uav.html

68. www.mtri.org
Evaluating the Use of
Unmanned Aerial Vehicles
for Transportation Purposes
Michigan Tech team members: Colin Brooks (cnbrooks@mtu.edu, 734-
604-4196), Thomas Oommen, Timothy C. Havens, Theresa M. Ahlborn,
Richard J. Dobson, Dave Dean, Ben Hart, Chris Roussi, Nate Jesse,
Rudiger Escobar Wolf, Michelle Wienert, Blaine Stormer, John Behrendt
MDOT program manager: Steve Cook; MDOT Research Manager:
André Clover
http://www.mtri.org/mdot_uav.html

69. Confined space inspection
Initial flights - understand
capability to fly in confined
spaces; later flights -
smaller UAVs
– MDOT Pump Station
– 4’ culvert (1.2m)
Is it safe to send a person
into the pump station?
– Eventually: unlit, retrieve
through opening
DJI Phantom 1, Walkera QR
W100S, Helimax 1Si;
Blackout Mini H Quad ready
to fly
69

70. Tethered Blimps for Traffic Monitoring
Aerostats/Blimps
• Long loitering time on station – up to
several days
• Can be sized to payload
requirements
• Tethered, lower FAA requirements for
flight operations, can operate at night
• Area needed for launch and recovery
• Some designs can operate in windy
weather
• Less need for permanent equipment

71. Support for emergency response
71
Post-spill response; post-flooding evaluation, crash scene reconstruction, landslide mapping, thermal mapping

72. Bridge asset management & condition
assessment imagery: collecting data
72

73. Bridge asset management & condition
assessment imagery: examples
73

74. Automated spall detection
Automated spall
detection algorithm
(developed by Brooks,
Dobson)
Applied to high-
resolution 3D elevation
model (DEM) for
Merriman East
(pictured), Stark Road
bridges.
Merriman East: 4.4%
spalled (150.0 square
feet)
74

75. Combined thermal data for 2 bridges
Merrim
an
Stark

76. Automated delamination detection
Delamination should be evident in thermal but not in visible!
Criteria can be
added: eliminate
small areas (e. g.
single pixels, pixels
with low number of
neighbors, etc.),
look at individual
bands, etc.
Only pixels with
more than 6
neighbors.
Area = 0.18 m2

77. UAV-Based LiDAR
LiDAR sensor pod developed
– Hokuyo UTM-30LX LIDAR
– VectorNAV MEMS IMU
– Beaglebone Black onboard
computer
– WIFI bridge
– LiPo battery power
Three-dimensional Simultaneous
Localization and Mapping (SLAM)
algorithms developed
77
sensing
sensing
Comm. with cloud
Comm. with ground sta on
Intra-vehicle comm.
sensing
NavigationGlobal map
Cloud Computing
Cloud Simultaneous Localization and Mapping (SLAM) Onboard UAV
• Motor control
• GPS
• IMU
• Local SLAM
• Local map
• Local naviga on
• Basic sensor processing
• Telemetry and
communica on
Sensor
Fusion
Machine Intelligence Operations
Chemical spill
Injured person
Suspicious
activity
Damaged
infrastructure
Bridge with linear interpolation assumption

78. ITS World Congress 2014 demonstrations
Indoor flights at the indoor Test Track by the Demo
Launch area
Live video feed of Belle Isle from blimp displayed in
MDOT Traffic Operations Center at Cobo Hall
Outdoor demonstrations at Belle Isle – Technology
Showcase
Spotlight, technical session talks
Mock Incident participation – UAV, blimp demos
78

79. 79
Contact Info
Colin Brooks cnbrooks@mtu.edu
Desk: 734-913-6858, Mobile: 734-604-4196
Michigan Tech Research Institute, MTRI
3600 Green Court, Suite 100
Ann Arbor, MI 48105
www.mtri.org
www.mtri.org/mdot_uav.html