GPUs, Cloud and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living
Upcoming SlideShare
Loading in...5
×
 

GPUs, Cloud and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living

on

  • 2,072 views

An overview of Azavea's recent work to increase geoprocessing performance through distributed computing, cloud computing, GPUs and other techniques.

An overview of Azavea's recent work to increase geoprocessing performance through distributed computing, cloud computing, GPUs and other techniques.

Statistics

Views

Total Views
2,072
Views on SlideShare
2,072
Embed Views
0

Actions

Likes
1
Downloads
13
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • So this story starts with my wife and I looking for a house and being frustrated by the type of information we had access to. We didn't know where to start. Each of the real estate agents we met knew a particular part of the city really well, so they tended to steer us toward those houses.
  • We didn't own a car, so we wanted to find a place that was * Close to Center City (somewhat important) * Walking distance to a grocery store (vital) * Near some restaurants (very important) * Walk to a library (nice to have) * Near a Park (yes!) * Close to fencing * Biking / walking distance from our work
  • The factors you consider important are probably not the same as ours. Maybe yours are related to child care or rankings for local schools or being near a farmer's market. Or maybe you want to be close to PhillyCarShare or public transit, but don't want to be too close to downtown.
  • We selected a variety of factors that contribute to sustainability, ranging from location in a state or federal tax incentive zone to environmental amenities like tree canopy to transit considerations like access to bus and regional rail lines. Retail businesses targeting markets may be interested in demographic factors like age and per capita income and proxies for environmental engagement like recycling participation.
  • Many of the ideas here are not new. Actually, they were developed here in Philadelphia by Ian McHarg, who was chair of the Landscape Architecture and Regional Planing department at the University of Pennsylvania
  • He wrote a book in 1969 called Design with Nature, and focused on sustainable and ecological design. Among other concepts, he described how a series of inputs drawn on transparent acetate sheets. Could be combined as a set of map overlays to identify the best site for a particular facility, road or whatever.
  • Now fast forward to the 1990’s. We’re interested in digital maps. This approach to compositing several digital maps was developed further by Dana Tomlin, who is now also a professor at University of Pennsylvania. Tomlin developed the computational vocabulary to perform this type of work – he called it Map Algebra. He’s also a really great teacher, so if you ever get a chance to take a class with him. Do it. He’s great.
  • So, these days people do this kind of work using desktop GIS systems. You are looking at the ArcMap application from ESRI
  • Export to KML
  • scenarios
  • Heat map
  • Export to KML
  • Export to KML
  • Export to KML

GPUs, Cloud and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living GPUs, Cloud and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living Presentation Transcript

  • GPUs, Clouds and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living Robert Cheetham 17 February 2011 NC GIS 2011
  • About Azavea
    • Founded in 2000
    • 27 people
      • software engineers
      • spatial analysts
      • project managers
    • Web & Mobile apps
    • Spatial Analysis
    • R&D
    • High Performance Computing
    • User Experience
  • B Corporation
    • 10% Research Program
    • Pro Bono Program
    • Time-to-Give-Back Program
    • Employee-focused Culture
    • Projects with Social Value
  • High Performance Geoprocessing
  • Classic GIS Use Case ...
  • Close to Center City  Walk to Grocery Store  Nearby Restaurants  Library  Near a Park  Biking / walking distance from our work  Biking distance to fencing club  somewhat important vital very important nice to have somewhat important very important somewhat important Robert’s Rules of Housing
    • Child Care
    • Local School Rankings
    • Farmer's Market
    • PhillyCarShare
    • Public Transit
    Your Factors might include…
    • Tax Incentives
    • Commercial Corridor Health
    • Public Transit
    • Car Share
    • Open Space
    • Farmers’ Markets
    • Street Network Density
    • Recycling Participation
    • Walkability
    Sustainability Factors
  • Not a new idea… Ian McHarg
  • Not a new idea … Design with Nature
  • Not a new idea … Map Algebra
  • Desktop GIS
  • x 5 x 2 x 3 x 1 + + + = Generate Output Heat Map
  • The Web is different from the Desktop
    • Lots of simultaneous users
    • Stateless environment
    • HTML+JS+CSS
    • Users are less skilled
    • Users are less patient
  • ArcGIS Server
    • Flex, Silverlight and JS API’s
    • Publish tasks and models
    • Caching
    • Optimized MSD files
  • But wait … there’s a problem
    • 10 – 60 second calculation time
    • Multiple simultaneous users …
    • … that are impatient
  • User Interface version 1
  •  
  •  
  •  
  •  
  •  
  •  
  • Reports
  • Reports
  •  
  • Sustainable Business Network
  •  
  •  
  •  
  •  
  •  
  •  
  • Walkability: Walkshed.org
  • NYC Big Apps Submission
  • Specific Optimization Goals
    • New Raster File format
    • Distributed processing
    • Binary messaging protocol
  • Optimization: File Format
    • Simple - strip out metadata
    • Limit data type and range
    • 1D arrays are fast to read/write
    • Assume
      • Same extent
      • Same cell size
      • Same pixel data type
      • Same cell alignment
      • Same projection
    • Azavea Raster Grid (ARG)
  • Optimization: Distributed Processing
    • Parallelizable - Local Ops and Focal Ops
    • Support multiple
      • Threads
      • Cores
      • CPU’s
      • Machines
    • Considered
      • Hadoop
      • Amazon Map Reduce
      • Beowolf
  • Distributed Processing
  • Binary Messaging Protocol
    • Started with XML
    • Binary Protocol Buffer is better
      • simpler
      • 3 to 10 times smaller
      • 20 to 100 times faster
      • less ambiguous
      • a bit easier to use programmatically
    • Considered
      • AMF
      • Google Protocol Buffer
  • Success!!
    • Reduced from 10-60 seconds to
    • <500 milliseconds
  • Additional [Experimental] Measures
    • Tiling
    • Pyramids
    • EC2 for planned peaks – NYC Big Apps
    • HTTP file caching - Varnish
  • Optimizing one process sub-optimizes others
    • Complex to configure and maintain
    • One type of operation
    • No interpolation
    • No mixing cell sizes
    • No mixing extents
    • No mixing projections
    • No Map Algebra
    • No ModelBuilder
    • etc.
  • High Performance Geoprocessing 2.0
    • More generic
    • Cache data – memory is cheaper
    • New programming technology
  • OMB Watch: Federal Spending Equity
  • High Performance Geoprocessing 2.0
    • Reduced calculation time to
    • ~40ms
    • GPU geoprocessing research
    But wait, there’s more…
  • GPU geoprocessing research
  • GPU geoprocessing research
      • New languages
        • CUDA
        • OpenCL
        • DirectCompute
      • Re-write every algorithm
      • Hardware Diversity
    Challenges
      • We re-wrote a few Map Algebra operations:
        • Local
        • Neighborhood
        • Zonal
        • Viewshed
        • etc.
      • 15 – 120x speed improvement
      • Large grids
      • Large neighborhoods
    Results
  • Not for the faint of heart
  • Sea Level Rise
  • Crime Analysis, Early Warning and Forecasting
  • Hunch Helper
  • Risk Forecasting
  • Animation
  •  
  •  
  •  
  • Food, Culture and Sustainability
  • Quick Demo http://commonspace.us
  • The Future
    • Clouds of Processors - Google App Engine
    • Faster is different
  • Summary
    • Challenges
      • Geographic data growth rates are exploding
      • Size of data sets is growing
        • Lidar
        • Raster
        • Social Media
      • New form factors that are less powerful
      • Distributed data sets
      • Larger numbers of less technical users
    • New Options
      • Clouds of processors
      • Clouds of virtual machines
      • GPUs
  • Many Thanks! © Photo used with permission from Alphafish , via Flickr.com
  • GPUs, Clouds and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living Robert Cheetham 17 February 2011 NC GIS 2011