Geospatial Analysis in the Cloud
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Geospatial Analysis in the Cloud

on

  • 1,496 views

Presented at the Government Cloud Service Oriented Architecture Workshop

Presented at the Government Cloud Service Oriented Architecture Workshop

Statistics

Views

Total Views
1,496
Views on SlideShare
1,490
Embed Views
6

Actions

Likes
0
Downloads
14
Comments
0

2 Embeds 6

http://www.slideshare.net 5
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Geospatial Analysis in the Cloud Presentation Transcript

  • 1. Use of Cloud Computing for scalable geospatial data processing and access Andrew Turner CTO, FortiusOne andrew@fortiusone.com Partner: U.S. Federal Geographic Data Committee
  • 2. What is GeoCommons? A Brief History
  • 3. Vulnerability Identification Chicago Denver Atlanta Fiber Density Route 2 Los Angeles Route 1 Electric Transmission Line Density
  • 4. Columbus Circle Holland Tunnel Baseline connectivity of a fiber WTC network provider in NYC. This particular provider is a good proxy for the structure of the entire island of Manhattan since they have about 80% of the right of ways on the island and a large number of egress points off the island. The higher the peak in the map the more frequently used the path is as a possible routing path.
  • 5. Lastly a scenario is run where just 10,000 sq ft. of damage is done to the Holland Tunnel and the impact calculated. The result is a 8.6% loss of network connectivity, 134 times the impact of the WTC simulation. The dramatic impact is seen in the image from the loss as well as the stress put on the GW Bridge route out of the city.
  • 6. GeoCommons: Version 1
  • 7. Find interesting data
  • 8. Find Map a interesting data relevant area
  • 9. Find Map a Visualize to interesting data relevant area find meaning
  • 10. Find Map a Visualize to interesting data relevant area find meaning Layer, Modify, and Analyze
  • 11. Find Map a Visualize to interesting data relevant area find meaning Collaborate Layer, Modify, with others and Analyze
  • 12. Find Map a Visualize to interesting data relevant area find meaning Publish and Collaborate Layer, Modify, share results with others and Analyze
  • 13. Visualization
  • 14. Analysis
  • 15. Applying Lessons Learned
  • 16. Modularize Application Programming Interface Finder Maker RESTful Core Interfaces
  • 17. Relational Databases Don’t Scale Well
  • 18. Datasets as Databases KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Core
  • 19. Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Core
  • 20. Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Parse & Store Core
  • 21. Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Parse & Store Core
  • 22. Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Parse & Store Core
  • 23. Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Finder Maker Parse & Store Core
  • 24. Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Download Finder Maker Parse & Store Core
  • 25. Datasets as Databases Upload KML Shapefile CSV (Excel) GeoRSS Documents Download Finder Maker Parse & Store Analyze Core
  • 26. Datasets as Databases Upload KML Shapefile Visualize CSV (Excel) GeoRSS Documents Download Finder Maker Parse & Store Analyze Core
  • 27. Geospatial Catalog and Server
  • 28. Delivery Mechanisms
  • 29. Appliances • Sun 4150 • RAID Array
  • 30. Web Scaled Racks • 3 Appliances • Network File Storage • Load Balancer • Monitoring and Tunnels • Production & Staging racks • Racks in office for development
  • 31. Limits in Limits in Scaling Development
  • 32. Limits in Limits in Scaling Development People
  • 33. Limits in Limits in Scaling Development People Power
  • 34. Limits in Limits in Scaling Development People Power Size
  • 35. Limits in Limits in Scaling Development People Power Size Cost
  • 36. Limits in Limits in Scaling Development People Power Size Cost Time
  • 37. Limits in Limits in Scaling Development People Power Size Cost Time
  • 38. Limits in Limits in Scaling Development People Testing on “clean” machines Power Size Cost Time
  • 39. Limits in Limits in Scaling Development People Testing on “clean” machines Power Size Deployment testing of Cost upgrades Time
  • 40. Limits in Limits in Scaling Development People Testing on “clean” machines Power Size Deployment testing of Cost upgrades Time Controlled Environments
  • 41. Leveraging the Cloud http://www.flickr.com/photos/kky/704056791 url
  • 42. Amazon Web Services
  • 43. Management Consoles
  • 44. Processing via MapReduce
  • 45. Launching New Instances
  • 46. Elastic Computing Cluster - EC2 • Virtual Servers • Machine Images (AMI) • On-Demand CentOS AMI
  • 47. Elastic Computing Cluster - EC2 • Virtual Servers • Machine Images (AMI) • On-Demand CentOS AMI build
  • 48. Elastic Computing Cluster - EC2 • Virtual Servers • Machine Images (AMI) • On-Demand register CentOS AMI bundle build
  • 49. Elastic Computing Cluster - EC2 • Virtual Servers • Machine Images (AMI) • On-Demand register CentOS AMI bundle instantiate build
  • 50. Elastic Computing Cluster - EC2 • Virtual Servers • Machine Images (AMI) • On-Demand register CentOS AMI bundle instantiate build
  • 51. Elastic Computing Cluster - EC2 • Virtual Servers • Machine Images (AMI) • On-Demand register CentOS AMI bundle instantiate build
  • 52. Elastic Computing Cluster - EC2 • Virtual Servers • Machine Images (AMI) • On-Demand register CentOS AMI bundle instantiate build
  • 53. Elastic Computing Cluster - EC2 • Virtual Servers • Machine Images (AMI) • On-Demand register CentOS AMI bundle instantiate build
  • 54. Elastic Block Store - EBS Create EBS 100 GB
  • 55. Elastic Block Store - EBS Create EBS attach 100 GB
  • 56. Elastic Block Store - EBS Create EBS attach 100 GB snapshot
  • 57. Elastic Block Store - EBS Create EBS attach 100 GB snapshot S3 Diff v1
  • 58. Elastic Block Store - EBS Create EBS attach 100 GB snapshot S3 Diff v1 Diff v2
  • 59. Elastic Block Store - EBS Create EBS attach 100 GB snapshot Create & Attach S3 Diff v1 Diff v2
  • 60. Elastic Block Store - EBS Create EBS attach 100 GB snapshot Create & Attach S3 Diff v1 Diff v2
  • 61. Elastic Block Store - EBS Create EBS attach 100 GB snapshot Create & Attach S3 Diff v1 Diff v2
  • 62. Elastic Block Store - EBS Create EBS attach 100 GB snapshot Create & Attach S3 Diff v1 Diff v2
  • 63. Public Datasets
  • 64. Additional Benefits • Federation • Tile generation • Content-delivery System • Simple Queue System (SQS) tiles/openstreetmap/9/74/97.png tiles/openstreetmap/9/74/98.png tiles/bluemarble/9/74/97.png S3 Storage tiles/bluemarble/9/74/98.png
  • 65. Cloud Architecture • EC2 image of current system architecture • EBS image stored to S3 of default database • Current application release in S3 • Start an EC2, attach data, attach code, startup v1.4.3 Default Datasets
  • 66. Cloud Architecture • EC2 image of current system architecture • EBS image stored to S3 of default database • Current application release in S3 • Start an EC2, attach data, attach code, startup create instance v1.4.3 Default Datasets
  • 67. Cloud Architecture • EC2 image of current system architecture • EBS image stored to S3 of default database • Current application release in S3 • Start an EC2, attach data, attach code, startup create instance v1.4.3 Default Datasets
  • 68. Cloud Architecture • EC2 image of current system architecture • EBS image stored to S3 of default database • Current application release in S3 • Start an EC2, attach data, attach code, startup create instance v1.4.3 attach data Default Datasets
  • 69. Cloud Architecture • EC2 image of current system architecture • EBS image stored to S3 of default database • Current application release in S3 • Start an EC2, attach data, attach code, startup create instance v1.4.3 Snapshot attach data Default Datasets Backup Backup Backup
  • 70. Cloud Architecture • EC2 image of current system architecture • EBS image stored to S3 of default database • Current application release in S3 • Start an EC2, attach data, attach code, startup create instance Cache S3 Downloads v1.4.3 Snapshot attach data Default Datasets Backup Backup Backup
  • 71. Scaling • RESTful architecture • Caching for speed, and CDN support • Amazon Web Services • CloudWatch • Elastic Scaling • Load Balancer
  • 72. Private Instances
  • 73. First Users: Meedan, Media
  • 74. Repeatable
  • 75. Repeatable
  • 76. Data Federation community
  • 77. Geospatial Federated Search Search
  • 78. Geocoding
  • 79. Geocoding - Scale as Required Upload CSV Cache Geocode Results API Geocoding Engine TIGER/Line SQLite
  • 80. Geocoding - Scale as Required Upload CSV Cache Geocode Results API Geocoding Engine TIGER/Line SQLite
  • 81. Best Practices Applied to the Government • Built using open, established tools • Full choice - Linux, Windows • Full Control • Repeatable processes • Continual backup • Scaling dynamic and large datasets • Synchronous and Asynchronous analysis
  • 82. Level of Maturity • Widely adopted • Broad support and ecosystem • Full stack support
  • 83. Perceived Impediments to Adoption • Single Vendor (open-source alternatives arising) • Maintenance and Location • Data Security
  • 84. Thank you Andrew Turner andrew@fortiusone.com http://highearthorbit.com/presentations