• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Geoprocessing in Web Time (Robert Cheetham)

Geoprocessing in Web Time (Robert Cheetham)






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • So this story starts with my wife and I looking for a house and being frustrated by the type of information we had access to. We didn't know where to start. Each of the real estate agents we met knew a particular part of the city really well, so they tended to steer us toward those houses.
  • We didn't own a car, so we wanted to find a place that was * Close to Center City (somewhat important) * Walking distance to a grocery store (vital) * Near some restaurants (very important) * Walk to a library (nice to have) * Near a Park (yes!) * Close to fencing * Biking / walking distance from our work
  • The factors you consider important are probably not the same as ours. Maybe yours are related to child care or rankings for local schools or being near a farmer's market. Or maybe you want to be close to PhillyCarShare or public transit, but don't want to be too close to downtown.
  • We selected a variety of factors that contribute to sustainability, ranging from location in a state or federal tax incentive zone to environmental amenities like tree canopy to transit considerations like access to bus and regional rail lines. Retail businesses targeting markets may be interested in demographic factors like age and per capita income and proxies for environmental engagement like recycling participation.
  • He wrote a book in 1969 called Design with Nature, and focused on sustainable and ecological design. Among other concepts, he described how a series of inputs drawn on transparent acetate sheets. Could be combined as a set of map overlays to identify the best site for a particular facility, road or whatever.
  • Now fast forward to the 1990’s. We’re interested in digital maps. This approach to compositing several digital maps was developed further by Dana Tomlin, who is now also a professor at University of Pennsylvania. Tomlin developed the computational vocabulary to perform this type of work – he called it Map Algebra. He’s also a really great teacher, so if you ever get a chance to take a class with him. Do it. He’s great.
  • So, these days people do this kind of work using desktop GIS systems. You are looking at the ArcMap application from ESRI
  • Export to KML
  • Export to KML
  • Export to KML

Geoprocessing in Web Time (Robert Cheetham) Geoprocessing in Web Time (Robert Cheetham) Presentation Transcript

  • Geoprocessing in Web Time: Distributed Computing for High Performance Geoprocessing Robert Cheetham [email_address] @rcheetham
  • Site Selection Tools
  • Buying a Home
  • Close to Center City  Walk to Grocery Store  Nearby Restaurants  Library  Near a Park  Biking / walking distance from our work  Biking distance to fencing  somewhat important vital very important nice to have somewhat important very important somewhat important Robert’s Rules of Housing
    • Child Care
    • Local School Rankings
    • Farmer's Market
    • PhillyCarShare
    • Public Transit
    Your Factors might include…
    • Tax Incentives
    • Commercial Corridor Health
    • Public Transit
    • Car Share
    • Open Space
    • Farmers’ Markets
    • Street Network Density
    • Recycling Participation
    • Walkability
    Siting Decision Factors
  • Not a new idea … Design with Nature
  • Not a new Idea … Dana Tomlin
  • Desktop GIS
  • x 5 x 2 x 3 x 1 + + + = Generate Output Heat Map
  • Web is different from the Desktop
    • Lots of simultaneous users
    • Stateless environment
    • Users are less skilled
    • Users are less patient
  • ArcGIS Server
    • Flex, Silverlight and JS API’s
    • Publish tasks and models
    • Caching
    • Optimized MSD files
  • But wait … there’s a problem
    • 10 – 60 second calculation time
    • Multiple simultaneous users …
    • … that are impatient
  • Specific Optimization Goals
    • New Raster File format
    • Distributed processing
    • Binary messaging protocol
  • Optimization: File Format
    • Simple - strip out metadata
    • Limit data type and range
    • 1D arrays are fast to read/write
    • Assume
      • Same extent
      • Same cell size
      • Same pixel data type
      • Same cell alignment
      • Same projection
    • Azavea Raster Grid (ARG)
  • Optimization: Distributed Processing
    • Parallelizable - Local Ops and Focal Ops
    • Support multiple
      • Threads
      • Cores
      • CPU’s
      • Machines
    • Considered
      • Hadoop
      • Amazon Map Reduce
      • Beowolf
  • Distributed Processing
  • Binary Messaging Protocol
    • Started with XML
    • Binary Protocol Buffer is better
      • simpler
      • 3 to 10 times smaller
      • 20 to 100 times faster
      • less ambiguous
      • a bit easier to use programmatically
    • Considered
      • AMF
      • Google Protocol Buffer
  • Success!!
    • Reduced from 10-60 seconds to
    • <500 milliseconds
  • Additional [Experimental] Measures
    • Tiling
    • Pyramids
    • EC2 for planned peaks – NYC Big Apps
    • HTTP file caching - Varnish
  • Optimizing one process sub-optimizes others
    • Complex to configure and maintain
    • One type of operation
    • No interpolation
    • No mixing cell sizes
    • No mixing extents
    • No mixing projections
    • No Map Algebra
    • No ModelBuilder
    • etc.
  • High Performance Geoprocessing 2.0
    • More generic
    • Cache data – memory is cheaper
    • New programming technology
  • High Performance Geoprocessing 2.0
    • Reduced calculation time to
    • ~40ms
  • GPU Processing Research
  • GPUs
  • GPU geoprocessing research
      • We re-wrote a few Map Algebra operations:
        • Local
        • Neighborhood
        • Zonal
        • Viewshed
        • etc.
      • 15 – 120x speed improvement
      • Large grids
      • Large neighborhoods
  • Walkshed.org
  • Walkshed.org
  • Food, Culture and Sustainability
  • OMB Watch: Federal Spending Equity
  • Sea Level Rise
  • GPU Processing Research
  • GPUs
  • GPU geoprocessing research
  • Stormwater Modeling
  • Stormwater Modeling Game
  • Stormwater Modeling Game
  • Stormwater Modeling Game
  • Summary
    • New technologies are changing what can be done
    • Faster geoprocessing is not just faster, it’s different
    • Opportunity and responsibility to re-think the user GIS user experience
      • Tablets
      • GPUs
      • Cloud Computing
      • Crowd-sourcing
      • Increased sampling and tracking
      • More corporate and gov transparency
      • Many, many more sensors
  • Many Thanks! © Photo used with permission from Alphafish , via Flickr.com
  • Geoprocessing in Web Time: Distributed Computing for High Performance Geoprocessing Robert Cheetham [email_address] @rcheetham