Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Geospatial machine learning for urban development

186 views

Published on

Ilke Demir, Postdoctoral Research Scientist, Facebook

Published in: Technology
  • Be the first to comment

Geospatial machine learning for urban development

  1. 1. Geospatial Machine Learning for Urban Development Ilke Demir Facebook MLConf – The Machine Learning Conference
  2. 2. Understanding the World
  3. 3. Exploring the World
  4. 4. Understanding the World urban safety socioeconomic data & voting patterns poverty mapping disaster mapping
  5. 5. Open Datasets o DeepGlobe: https://deepglobe.org o Road extraction o Building detection o Land cover classification o SpaceNet: https://spacenetchallenge.github.io/ o Road network extraction o Building detection o Earth observation challenge: http://eochallenge.bemyapp.com/ o Water resource extraction o Change detection o Data fusion contest: http://www.grss-ieee.org/data-fusion-contest/ o Land cover land use classification from various sensor data o Functional map of the world: https://www.iarpa.gov/challenges/fmow.html o Labeling the world into land use categories
  6. 6. Case Study: DeepGlobe o Public datasets and benchmarks for scalable and reliable approaches o Satellite imagery is powerful as it is more structured than everyday images DeepGlobe focuses on machine learning and computer vision approaches on satellite images and brings together researchers with different perspectives by; o Publishing public datasets and baselines o Creating public challenges to benchmark different approaches o Organizing a workshop to sparkle new collaborations and ideas
  7. 7. DeepGlobe Tracks Road Extraction Challenge: - Maps, accessibility, and connectivity - Economic and developmental inclusion - Crisis response
  8. 8. DeepGlobe Tracks Building Detection Challenge: - Population dynamics and demographics - Disaster recovery and damage coordination - Urban development
  9. 9. DeepGlobe Tracks Land Cover Classification Challenge: - Sustainable development - Automation in agriculture - Urban planning and growth
  10. 10. DeepGlobe Challenges 1. Road Extraction Challenge - DigitalGlobe Vivid+ - 50 cm/pixel - Pixel-wise manual annotation - 2 classes - Thailand, Indonesia, India - 8570 images of 2220km2 - 70%/15%/15% split - ~4% positive pixels - Diverse road networks - 345 participants - 2150 submissions - 84 results in the leaderboard
  11. 11. DeepGlobe Challenges 2. Building Detection Challenge - SpaceNet Buildings v2 - 31cm single-band panchromatic - 1.24m 8 band multi-spectral - Manual annotation of polygons - 2 classes - Las Vegas, Paris, Shanghai, Khartoum - 24586 images of 9623 km2 - 60%/20%/20% split - 302701 buildings - 296 participants - 576 submissions - 25 results in the leaderboard
  12. 12. DeepGlobe Challenges 3. Land Cover Classification Challenge - DigitalGlobe Vivid+ - 50 cm/pixel - Pixel-wise manual annotation - 7 classes - Thailand, India, Indonesia - 1146 images of 1717 km2 - 70%/15%/15% split - 20m minimum granularity area - 311 participants - 1155 submissions - 28 results in the leaderboard
  13. 13. DeepGlobe Results and Baselines Roads DeepLab variation Only data augmentation by rotation IoU score 0.545
  14. 14. DeepGlobe Results and Baselines Roads DeepLab variation Only data augmentation by rotation IoU score 0.545 Buildings Ensemble 3 U-Net models Boost by OpenStreetMap data F1 score 0.693
  15. 15. DeepGlobe Results and Baselines Roads DeepLab variation Only data augmentation by rotation IoU score 0.545 Buildings Ensemble 3 U-Net models Boost by OpenStreetMap data F1 score 0.693 Lands DeepLab variation Data augmentation by rotations and class weights IoU score of 0.433
  16. 16. DeepGlobe Winners Roads Buildings Land D-LinkNet: LinkNet with Pretrained Encoder and Dilated Convolution for High Resolution Satellite Imagery Road Extraction Lichen Zhou, BUPT Building Detection from Satellite Imagery using Ensemble of Size-specific Detectors Ryuhei Hamaguc hi, Pasco Dense Fusion Classmate Network for Land Cover Classification Chao Tian, Harbin Institute of Technology 1. Semantic Binary Segmentation using Convolutional Networks without Decoders Shubhra Aich*; William van der Kamp; Ian Stavness, University of Saskatchewan 2. Stacked U-Nets with Multi-Output for Road Extraction Tao Sun*; Zehui Chen; Wenxiang Yang; Yin Wang, Tongji University 3. D-LinkNet: LinkNet with Pretrained Encoder and Dilated Convolution for High Resolution Satellite Imagery Road Extraction Lichen Zhou*; Chuang Zhang; Ming Wu, Beijing University of Posts and Telecommunications 4. Fully Convolutional Network for Automatic Road Extraction from Satellite Imagery Alexander Buslaev*, Mapbox; Selim Seferbekov, Veeva Systems; Vladimir Iglovikov, Lyft Inc; Alexey Shvets Massachusetts Institute of Technology 5. Road Detection with EOSResUNet and Post Vectorizing Algorithm Oleksandr Filin*; Serhii Panchenko; Anton Zapara, EOS Data Analytics 6. Residual Inception Skip Network for Binary Segmentation Jigar Doshi*, CrowdAI 7. Roadmap Generation using a Multi-Stage Ensemble of Neural Networks with Smoothing-Based Optimization Dragos Costea*; Alina Marcu; Emil Slusanschi; Marius Leordeanu, University Politehnica of Bucharest 8. Rotated Rectangles for Symbolized Building Footprint Extraction Matthew Dickenson*; Lionel Gueguen, Uber 9. Building Detection from Satellite Imagery Using Composite Loss Function Sergey Golovanov*; Rauf Kurbanov; Aleksey Artamonov; Alex Davydow; Sergey Nikolenko, Neuromation 10. Building Detection from Satellite Imagery using Ensemble of Size- specific Detectors Ryuhei Hamaguchi*; Shuhei Hikosaka, Pasco Corporation 11. TernausNetV2: Fully Convolutional Network for Instance Segmentation Vladimir Iglovikov*, Lyft Inc; Selim Seferbekov, Veeva Systems; Alexander Buslaev, Mapbox; Alexey Shvets Massachusetts Institute of Technology 12. Semantic Segmentation based Building Extraction Method using Multi-source GIS Map Datasets and Satellite Imagery Weijia Li*; Conghui He; Jiarui Fang; Haohuan Fu, Tsinghua University 13. CNNs Fusion for Building Detection in Aerial Images for the Building Detection Challenge Remi Delassus*, Qucit - LaBRI; Romain Giot, Univ. Bordeaux 14. Building Extraction from Satellite Images Using Mask R-CNN with Building Boundary Regularization Kang Zhao*; Jungwon Kang; Jaewook Jung; Gunho Sohn, York University 15. Deep Aggregation Net for Land Cover Classification Tzu-Sheng Kuo*; Keng-Sen Tseng; Jia-Wei Yan; Yen-Cheng Liu; Yu- Chiang Frank Wang, National Taiwan University 16. Stacked U-Nets for Ground Material Segmentation in Remote Sensing Imagery Arthita Ghosh*; Max Ehrlich; Sohil Shah; Larry Davis; Rama Chellappa, University of Maryland 17. Land Cover Classification from Satellite Imagery With U-Net and Lovasz-Softmax Loss Alexander Rakhlin*; Alex Davydow; Sergey Nikolenko, Neuromation 18. Dense Fusion Classmate Network for Land Cover Classification Chao Tian*, Harbin Institute of Technology; Cong Li; Jianping Shi, Sensetime 19. NU-Net: Deep Residual Wide Field of View Convolutional Neural Network for Semantic Segmentation Mohamed Samy; Karim Amer*; Kareem Eissa; Mahmoud Shaker; Mohamed ElHelw, Nile University; 20. Feature Pyramid Network for Multi-Class Land Segmentation Selim Seferbekov*, Veeva Systems; Vladimir Iglovikov, Lyft Inc; Alexander Buslaev, Mapbox; Alexey Shvets Massachusetts Institute of Technology 21. Uncertainty Gated Network for Land Cover Segmentation Guillem Pascual*; Santi Seguí; Jordi Vitria, Universitat de Barcelona 22. Land Cover Classification With Superpixels and Jaccard Index Post-Optimization Alex Davydow*; Sergey Nikolenko, Neuromation
  17. 17. Changing the World urban planning simulations generative models
  18. 18. Proceduralization Generalized Proceduralization
  19. 19. Shape Understanding Point clouds: Arch Physical objects: Urban spaces: Satellite Images: Architectural models:
  20. 20. Point clouds: Satellite images: Urban spaces: Architecture: Generative Modeling
  21. 21. Case Study: Street Addresses • 75% of the world lives without adequate addressing. What3Words • 4 billion people are ‘invisible’. United Nations • Haiti earthquake: 48 hours reaction time, 6 months complete road vectors. OpenStreetMap
  22. 22. Geocoding Solutions “issuer.lollipop.ripe” “ItsADream” “37.482825, -122.145661” “75682SB3084” “parrot.casino.failed” “SweetPotato” “102.22556, -12.166981” “7098HGT3083” • f(lat, lon) = “address” • Hashes, random words, manual tags • No spatial relation cue • No city/state/country • No accessibility • No geometry A B Humans need streets!
  23. 23. Traditional Addressing Systems London postal code system: Radial regions based on orientation and distance South Korea streets: Meter markers Japan block system: Hard to decipher Dubai addressing: Uses districts Berlin numbering: Zigzag house pattern
  24. 24. Our Generative Scheme • 5 alphanumeric fields • Hierarchical and linear descriptors • To close the gap between physical addresses and automated geocoding Road naming scheme: - distance from the center - orientation in odd parity Region naming scheme: - orientation wrt downtown - distance from downtown House numbering scheme: - meter markers on the road - block letters from the road “I7 Hacker Way, Menlo Park, CA, US”
  25. 25. Pipeline: Satellite Images ç • Irregular urban structure • Illumination/weather/country • Different road types
  26. 26. Pipeline: Road Predictions • Binary road masks • 19K*19K, 0.5m/pixel • SegNet
  27. 27. Pipeline: Road Network • Orientation based median filtering • Road segments by orientation bucketing
  28. 28. NF NH NE Pipeline: Regions • Road graph: Node=intersection, edge=road, weight=length • Partition for max inter, min intra connectivity, using normalized min-cut.
  29. 29. Pipeline: Naming • Orientation bucketing into N, S, W, E • Trace regions based on distance to CA • Orientation bucketing into major axes • Trace roads based on order
  30. 30. Pipeline: Address Cells • 5 meter marker along the road • Odd/even based on RHR • Distance field of roads: block offset
  31. 31. Results: Unmapped Developing Country • Improve coverage up to 80% • Processed more than 200 districts (and increasing!) • Regions follow natural boundaries • Road network is being discovered in non-urban settings
  32. 32. Results: Unmapped Developing Country • Improve coverage up to 80% • Processed more than 200 districts (and increasing!) • Regions follow natural boundaries • Road network is being discovered in non-urban settings
  33. 33. Results • Improve coverage up to 80% • Processed more than 200 districts (and increasing!) • Regions follow natural boundaries • Road network is being discovered in non-urban settings • Changing the world!
  34. 34. News & Ads! o Geospatial Modeling and Visualization, Special Issue in Big Earth Data Journal http://bit.ly/BigEarthData o SUMO Challenge: Understanding indoor scenes from 360 RGBD data https://sumochallenge.org/ o Challenges and opportunities for deep learning in remote sensing, Special session in Living Planet Symposium 2019 https://lps19.esa.int/ o EarthVision 2019! (coming soon…) o DeepGlobe v2! (coming some day…)
  35. 35. Thanks… and your turn! Generative Street Addresses Code: https://github.com/facebookresearch/street-addresses Paper: https://research.fb.com/publications/robocodes DeepGlobe Benchmark Papers: http://bit.ly/deepglobe_papers Website: http://deepglobe.org Dataset: http://bit.ly/deepglobe Ilke Demir e-mail: idemir@fb.com Twitter: @ilkedemir
  36. 36. Additional Slides 36
  37. 37. Evaluation Metrics Roads Pixel-wise mean IoU Buildings Average F1 score Lands Pixel-wise mean IoU
  38. 38. Design Choices Linear: similar addresses stored in a linear fashion Hierarchical: top-down structure for spatial encapsulation Compressible: 5x4 max (chars x words) Universal: independent of local language Inquirable: useful for geometric, proximity-based, and type-ahead queries Extendible: dynamically modifiable for new places Robust: flexible for overestimation and noise StructuralDesignParameters forefficientcomputerimplementation Linear: closer addresses are given related names Hierarchical: top-down subdivision of the world Memorable: short and alphanumeric, easily convertible Intuitive: with a sense of direction and distance Topological: consistent with road topology Inclusive: with local names (city, state) Physical: consistent with natural boundaries SemanticDesignParameters foruserfriendliness Machine Needs Human Needs
  39. 39. 39Geometric Shape Processing: Satellite Images [*] I. Demir et al., 2018. “Generative Street Addresses from Satellite Imagery”. International Journal on Geo-Information (IJGI).
  40. 40. Output Maps and Tools • .osm maps with roads (meter marking and offsetting on the fly) • ID-tool of MapBox for on-demand inverse/forward geocoding • rtree extension for efficient spatial querying • Experimental mobile app for self navigation • 21.7% decrease in arrival time using Robocodes
  41. 41. Results: Evaluation with Ground Truth • System learns 90.51% of roads • Approximately 80% on average • Better in urban environments • Ground truth prepared as if training data
  42. 42. Results: Mapped US City • More than 95% of the roads are found (compared to OSM). • Traditional addresses are more established, however • Robocodes are contextually and spatially easier to remember.
  43. 43. Results: Comparison Automated geocoding: A: parrot.casino.failed B: issuer.lollipop.ripe - Have irrelevant words based on lat/lon. Robocodes: 715D.NE127.Dhule.MhIn 716C.NE127.Dhule.MhIn - Have hierarchical and linear addresses. Landmark based: Green Park Green Park - Have roads but no addresses or labels. OSM: lat/lon lat/lon - Have neither road geometry, nor labels.
  44. 44. Limitations & Future Work • Robotic meter marking and offsetting: • (i) use smart parcel subdivision, • (ii) adapt to population density. • Imperfect training data: sample more countries. • Metric to evaluate regions: supervised learning of land annotations.
  45. 45. Inaccessible Areas • To extend our format to cover areas that are not accessible by streets, we explored different implementations to cover such areas, which are 26*5 m away from any street. • Geocoding as a function (excluding the version field): f (info, lat, lon) = x.y.z.t • For places with roads, info={road network, city, country} f (R, C) = x.y.city.country • Extreme case: only reliable information is latitude/longitude! 45
  46. 46. f(C,lat,lon) = hash(round(lat,3)) + dir(lat) . hash(round(lon,3)) +dir(lon) . C L-A-T-dir.L-O-N-dir.name.area Inaccessible Areas: Blackholes! • Linear hashing: • 26 letters + 10 digits • 100m x 100 m granularity • Last letter is the hemisphere • Range: 359.999, longitude: 7PRZ W • Hierarchical hashing: • Enlarge the grid from to 1 km x 1 km • Using two floating points = three letters • Within each cell, re-hash it to a 36 x 36 grid = one letter • New resolution: 30m, represented by five letters 46 f(C,lat,lon) = hash(round(lat,2)) + hash(lat - round(lat,2)) + dir(lat) . hash(round(lon,2)) + hash(lon - round(lon,2)) + dir(lon) . C LlatLlatHlatDlat .LlonLlonHlonDlon . name . Ocean /Continent /etc
  47. 47. Completion & Reconstruction 47 • Voxelize building proxy from footprint • Find roofs with photo- consistency in aerial images • Apply graph-cuts: •Building •Building-ground •Ground [*] I. Garcia-Dorado I. Demir, D. Aliaga. 2013. “Automatic Urban Modeling Using Volumetric Reconstruction with Surface Graph-cuts”. Computers & Graphics.
  48. 48. Website: http://deepglobe.org Papers: http://bit.ly/deepglobe_papers Dataset: http://bit.ly/deepglobe Hashtags: #DeepGlobe

×