This document discusses using Google AppEngine and Fusion Tables with ArcGIS Server for cloud geoprocessing. It describes how Fusion Tables can be used for storage, visualization and analysis. It then introduces Arc2Cloud, which adds additional geospatial functionality to Fusion Tables like a native API and services. The document demonstrates using Google AppEngine's task queue to perform geoprocessing across multiple Fusion Tables in parallel by dividing data into ranges, grids or other strategies. Spatial operations are performed using JTS or ArcGIS Server services. Results are written back to Fusion Tables.

1. Using Google AppEngine and Fusion Tables with ArcGIS ServerBrian FloodArc2Earth

2. Interoperability with ArcGIS Server REST APIUsing ArcGIS Server for Cloud geoprocessing

3. Google AppEngineGoogle Fusion TablesArcGIS Server – Geometry ServiceArc2Cloud

4. Google Fusion TablesBasically, a Cloud databaseStorage/QueryA NoSQLDB that acts like a SQL oneVisualizationMaps, Maps API support (really fast)Charts, timelines, etcMerge Tables – Collaborative Analysis HTTP APISpatial and Attribute QueryRead/WriteOthersSimpleGeoArcGIS FeatureLayerAppEngineDatastoreFusion Tables

5. Fusion Tables - StorageSQL Syntax not quite but close enoughSELECT, GROUP BY, FiltersBBOX and Radius Search need to filter in userland for polylines/polygons“contains ignoring case” – poor man’s full text search Read/Write Inserts, Bulk Inserts, Updates, Deletes rowidas primary key, a little wonkyBigTable/Megastore backendHuge ScaleHilbert curve for spatial indexSpatial and attribute portions of query execute in parallel, results merged (“all data near NYC with four stars”)

6. Fusion Tables - VisualizationMapsDynamic Map Tiles – really fastGoogle Maps API supportCould be used from any client api that supports tilesRenderer StylesInfo Window templatesSimple HeatmapsTimelinesChartsTable View

7. Fusion Tables – Why?Instant ScalabilityZero ConfigurationZero MaintenancePremium Access from GoogleGoogle Infrastructure

8. GeoServices REST SpecificationPublished by ESRIOpen SourceSpec, not implementationMake your map services look like ArcGIS ServerInteroperable ArcGIS.comArcGIS ExplorerAGS Mobile Apps

9. Can we make Fusion Tables look like an ArcGIS Server?

10. Arc2CloudAdditional geospatial functionality for Fusion Tables Google AppEngineDatasourcesMapsFeatureCacheNative APIFusion TablesTilesetsGeospatial REST SpecOAuthServicesTask Queue WFS & WFS/TAnalysisMulti Tenant ConfigSites

11. Arc2CloudEach feature has an endpoint and multiple formats. Feature attachments (photos etc) are stored in the AppEngineblobstoreinstance idfusion tablearc2cloud.com/brian8/data/datasources/Counties/101?f=htmlCounties?f=json?f=gjson?f=kml?f=gml?f=wkt?f=qr/brian9/NJState/census/a2eProjects/whateveryouwantFusion TablesStates.../brian8/ArcGIS/rest/services/Census/FeatureServer/0/101/attachments/addAttachment../query../addFeaturesetcESRI REST

12. Arc2Cloud Admin

13. Demo – Import Map

14. Google AppEnginePaaS – Platform as a ServiceZero ConfigurationAuto ScalingUtility Based Billing Only pay for what you usePython or Java Lots of secondary languages/frameworks can be compiled to JavaTask Queue – Parallel Execution

15. Google AppEngine400 requests/second

16. How can we leverage AppEngine’s Task Queue with ArcGIS Server’s geoprocessing?

17. Task Queue GeoprocessingExperimental!Why?Basic full layer geoprocessing without desktop softwareMerging Fusion Tables based on location is powerfulFun projectMapReduce-like processingWork on many feature collections in parallelCommon overlay/proximity operations between FT TablesBuffer, Merge, Join, Clip, Snap etcDivide and conquerUse ArcGIS Server to perform spatial operations Write results back to FT in parallel

18. Prepare yourself for programmer artwork

19. Task Queue GeoprocessingTask QueueRequest“spatialJoin”(parallel execution)Ranges DS: CountiesRange: 0-9Datasource IteratorFusion Tables DS: CountiesRange: 10-19Sequential

20. Ranges

21. GridsGrids DS: CountiesRange: 20-29 DS: StatesGrid: 5_20_35Status and Log(sharded)Collect, Analyze, Write DS: CountiesRange: 20-29ArcGISServerJTS - GeometryStatusRequestsBuffer

22. Merge

23. Union DS: StatesRange: 10-19

24. Phase 1 - Datasource IteratorMaps any FusionTable into smaller pieces for processingThe “Map” in MapReduceDispatch to Task QueueTasks are bulk loaded, 100 at a timeTask Queue is set for 100 tasks per second (max)Multiple queues can be used Different strategies for different spatial operationsSequentialRangeGridsRangesDatasource IteratorGridsSequential

25. Ranges

26. GridsPhase 2 – Retrieve DataTask QueueRunning in parallel on Task QueuePrimary and Secondary dataUnlike MapReduceEx: target and join tablesSequential, Rangeoffset, limit sql parametersGridWell known spatial grid for querying dataCan use Arc2Cloud FeatureCache (big perf win)Example:Grid strategy good for comparing two layers spatially (Union etc)Ranges are more efficient, use built-in Fusion Table cursors(parallel execution)DS: CountiesRange: 0-9DS: CountiesRange: 10-19Fusion Tables

27. Feature CacheVector TilesKML RegionsGridsFusion TablesSearchAnalysis GridPost Edit TasksInvalidate Cache