Advanced Map Caching TopicsTecnicas avanzadas de almacenamiento en cache de mapas                      Esri European User ...
Prerequisite knowledge  •   What is map caching?  •   How to author a map for caching  •   How to build map cache tiles
Topics in this session  •   What’s new in caching in ArcGIS 10  •   Approaching a large caching job      -   Building a te...
What’s new in caching in       ArcGIS 10
Mixed mode image format•   Creates JPEG tiles unless    transparent    pixels detected•   If transparent pixels detected, ...
How the mixed format creates tiles
Compact cache storage format  •   Stores tiles in compact, continuous file streams      (“bundles”)      -   Maximum ~16,0...
Move spatial subsets of tiles between caches  •   New export and import tools allow collaborative and      “best available...
Cache export tool •   Export Map Server Cache     -   Based on extent or polygon features     -   Convert storage format  ...
Cache import tool •   Import Map Server Cache     -    Based on extent or polygon features     -    Must have same storage...
Cache import for imagery •   Merge high resolution imagery into a base cache •   Seamless integration import feature bound...
Collaborative caching  •   Use export and import tools      -   Import the “best available” cached tiles      -   Esri Com...
Caches treated as a raster dataset in ArcGIS  •   Add Data and browse to cache directory.      -   Looks like any other ra...
Approaching a large caching job
What should you cache?  •   Base maps  •   Operational layers that satisfy one of the following:      -   High volumes of ...
Choosing scales                                          Sample 10 level cache•   Build just the scales you need•   Determ...
Choosing an image format  •   Image format affects      -   Tile storage space requirements      -   Web application perfo...
Basemap: Image format guidelines  •   Fewer colors (~256)      -   PNG 8  •   Many colors      -   JPEG           -   Smal...
Choosing JPEG quality •   Which one looks better?          JPEG 90 – 21KB         JPEG 55 – 15KB        JPEG 55 – 10KB    ...
Operational layers: image format guidelines  •   Vector overlays (roads, boundaries)      -   PNG 32 because of good antia...
Example: Tiles are too large     Aerial photo and    vector blend using         PNG 32
Antialiasing: Often worth the cost
Apply antialiasing in the MSD for best performance       1.                               2.Gives you this…               ...
Reducing duplicate labels•   Caches drawn using supertiles•   Duplicates can occur across supertile         Supertile     ...
B u i l d i n g   a   t e s t   c a c h e
How to build a test cache  1.   Select test area with       varying geography  2.   Create a simple feature class       co...
Note the following from the test cache  •   Appearance of tiles  •   Performance of tiles in client  •   Cache creation ti...
Choosing the tiles to cache
Not all tiles are created equal                  http://hotmap.msresearch.us
Use GIS modeling to predict frequently visited areas  This feature class covers 25% of California’s area          but incl...
Tiles you don’t want to create and store   •   Cache on demand               OR   •   Insert a “Data not available” tileMa...
Caching based on a feature class  •   Features should be:      -   Big      -   Generalized
Why do the features need to be big?•   How do I find out the bundle    size?    -   Map Server Cache Tiling        Scheme ...
Monitoring the tile creation
Setting the number of instances  •   Keep your CPU below 100%      •   Start with N+1 instances where N = the number of co...
Generate using multiple LAN hosts •   Configure instances for CPU’s on all SOC hosts •   Enable “Use local cache directory...
Use status tracking with your featuresHelpful script:Create gridded feature class for tracking ArcGIS Server map caching j...
Recovering from errors  •   Generate rebuild features from error log      -   Convert map cache-build log failures to foot...
Case study: State of Indiana    imagery basemap
Requirements  •   Single Indiana state basemap with most recent data      -   Maintain 6” resolution where available  •   ...
Approach6" scale                                  576.00   20   Individual 6" counties                                    ...
Stats on the final cache  •   Image format: JPEG 55  •   62,245,522 tiles  •   Total cache size 452.72 GB  •   Average til...
C a c h e u p d a te s tr a te g ie s
Can you update the cache?  •   How current does the data need to be?  •   How fast can the update take place?  •   Can the...
Strategic cache updates  1.   Detect where changes have taken place  2.   Merge these features  3.   Use them to define th...
Updating a cache using a staging server Staging ArcGIS                          Production ArcGIS Server Instance         ...
Updating a cache and data using a staging server Staging ArcGIS                             Production ArcGIS Server Insta...
Q   u e s t i o n s
Upcoming SlideShare
Loading in...5
×

Advanced Map Caching Topics

2,871

Published on

Sterling Quinn´s great presentation about map caching on Esri European User Conference 2011.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,871
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
39
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Advanced Map Caching Topics

  1. 1. Advanced Map Caching TopicsTecnicas avanzadas de almacenamiento en cache de mapas Esri European User Conference 2011
  2. 2. Prerequisite knowledge • What is map caching? • How to author a map for caching • How to build map cache tiles
  3. 3. Topics in this session • What’s new in caching in ArcGIS 10 • Approaching a large caching job - Building a test caches - Choosing the tiles to cache - Monitoring the tile creation - Case study: State of Indiana imagery basemap • Cache update strategies
  4. 4. What’s new in caching in ArcGIS 10
  5. 5. Mixed mode image format• Creates JPEG tiles unless transparent pixels detected• If transparent pixels detected, creates PNG32• Enormous space savings for large caches Mixed format helps with this JPEG on JPEG cache overlay problem
  6. 6. How the mixed format creates tiles
  7. 7. Compact cache storage format • Stores tiles in compact, continuous file streams (“bundles”) - Maximum ~16,000 tiles per bundle • Faster copying • Smaller size on disk • Generally faster to create • Less fragmentation
  8. 8. Move spatial subsets of tiles between caches • New export and import tools allow collaborative and “best available” caching Export Import
  9. 9. Cache export tool • Export Map Server Cache - Based on extent or polygon features - Convert storage format - Use for cache import or as a disconnected cache Export Exported using Nevada and Utah state boundary features.
  10. 10. Cache import tool • Import Map Server Cache - Based on extent or polygon features - Must have same storage format Import Import from a previously exported map cache.
  11. 11. Cache import for imagery • Merge high resolution imagery into a base cache • Seamless integration import feature boundary
  12. 12. Collaborative caching • Use export and import tools - Import the “best available” cached tiles - Esri Community Base Maps program Pasadena City College Contribution
  13. 13. Caches treated as a raster dataset in ArcGIS • Add Data and browse to cache directory. - Looks like any other raster • Export caches for disconnected field work - Export to compact format recommended
  14. 14. Approaching a large caching job
  15. 15. What should you cache? • Base maps • Operational layers that satisfy one of the following: - High volumes of traffic - Don’t change often - Cover small scales only
  16. 16. Choosing scales Sample 10 level cache• Build just the scales you need• Determine closest scale (Raster resolution)• Divide scale by 2 for each subsequent scale• Adjust smallest scale to full extent Final level is ~75% of the total• Consider Web Mercator scales
  17. 17. Choosing an image format • Image format affects - Tile storage space requirements - Web application performance (speed and supported browsers) - Tile image quality - Tile transparency
  18. 18. Basemap: Image format guidelines • Fewer colors (~256) - PNG 8 • Many colors - JPEG - Small file size - Quality settings range from 55 (orthos) - 90 (vector maps)
  19. 19. Choosing JPEG quality • Which one looks better? JPEG 90 – 21KB JPEG 55 – 15KB JPEG 55 – 10KB JPEG 90 – 33KB
  20. 20. Operational layers: image format guidelines • Vector overlays (roads, boundaries) - PNG 32 because of good antialiasing • Other overlays (weather, etc) - MIXED or PNG (bit depth depends on number of colors)
  21. 21. Example: Tiles are too large Aerial photo and vector blend using PNG 32
  22. 22. Antialiasing: Often worth the cost
  23. 23. Apply antialiasing in the MSD for best performance 1. 2.Gives you this… Box already checked shows that antialiasing is happening through MSD ☺
  24. 24. Reducing duplicate labels• Caches drawn using supertiles• Duplicates can occur across supertile Supertile No antialiasing boundaries 4096x4096 Supertile• Solution is annotation antialiasing 2048x2048 512x512 tile size 256x256 tile size ArcGIS Online / Bing / Google
  25. 25. B u i l d i n g a t e s t c a c h e
  26. 26. How to build a test cache 1. Select test area with varying geography 2. Create a simple feature class covering the test area 3. Create tiles at all scale levels based on this feature class
  27. 27. Note the following from the test cache • Appearance of tiles • Performance of tiles in client • Cache creation time • Cache size on disk
  28. 28. Choosing the tiles to cache
  29. 29. Not all tiles are created equal http://hotmap.msresearch.us
  30. 30. Use GIS modeling to predict frequently visited areas This feature class covers 25% of California’s area but includes 97% of its population
  31. 31. Tiles you don’t want to create and store • Cache on demand OR • Insert a “Data not available” tileMake your own heatmap of tile usage: ArcGIS Server descriptive tile analysis
  32. 32. Caching based on a feature class • Features should be: - Big - Generalized
  33. 33. Why do the features need to be big?• How do I find out the bundle size? - Map Server Cache Tiling Scheme to Polygons tool - See Esri Dev Summit 2011 presentation online for full demo
  34. 34. Monitoring the tile creation
  35. 35. Setting the number of instances • Keep your CPU below 100% • Start with N+1 instances where N = the number of cores on server • Adjust number of instances based on test cache Task Manager Manage Map Server Cache Tiles 2 CPU’s 3 Instances
  36. 36. Generate using multiple LAN hosts • Configure instances for CPU’s on all SOC hosts • Enable “Use local cache directory…” Temp cache Shared SOC Temp Cache cache hosts SOM Temp cache
  37. 37. Use status tracking with your featuresHelpful script:Create gridded feature class for tracking ArcGIS Server map caching jobs
  38. 38. Recovering from errors • Generate rebuild features from error log - Convert map cache-build log failures to footprint polygons tool • Cache Validation tool - Looks for missing tiles - Looks for tiles below a certain size (white or gray tiles) - False positives with building roofs and water - Useful for validating source imagery
  39. 39. Case study: State of Indiana imagery basemap
  40. 40. Requirements • Single Indiana state basemap with most recent data - Maintain 6” resolution where available • ArcGIS Online/Bing/Google tiling scheme • Integrate neighboring data for small scales • Record metadata for imagery in basemap
  41. 41. Approach6" scale 576.00 20 Individual 6" counties 1,128.50 19 2,256.99 18 Use gridded tile 4,513.99 17 Web Mercator Tiling Scheme 9,027.98 16 18,055.96 15 36,111.91 14 Clip to state boundary 72,223.82 13 144,447.64 12 288,895.29 11 577,790.58 10 1,155,581.15 9 IN Full extent 2,311,162.31 8 4,622,324.61 7 9,244,649.23 6 18,489,298.45 5 36,978,596.91 4 73,957,193.82 3 Not Built 147,914,387.60 2 295,828,775.30 1 591,657,550.50 0
  42. 42. Stats on the final cache • Image format: JPEG 55 • 62,245,522 tiles • Total cache size 452.72 GB • Average tile size 7.63 KB • 3 weeks to build
  43. 43. C a c h e u p d a te s tr a te g ie s
  44. 44. Can you update the cache? • How current does the data need to be? • How fast can the update take place? • Can the update be scripted? • Esri Spain elections example
  45. 45. Strategic cache updates 1. Detect where changes have taken place 2. Merge these features 3. Use them to define the update area Helpful scripts •Show edits since reconcile •Compare two feature classes in a file geodatabase
  46. 46. Updating a cache using a staging server Staging ArcGIS Production ArcGIS Server Instance Server InstanceMap service Layers for TOC All layers for and Query cartography of map service Map serviceCache folder Copy tiles Cache folder (On-demand caching needs the full map to build the cache)
  47. 47. Updating a cache and data using a staging server Staging ArcGIS Production ArcGIS Server Instance Server Instance All layers for cartography of map service Layers for TOC Copy tiles and QueryMap service Map service Cache folder Cache folder Geodatabase ReplicationMulti-user versioned Geodatabase Geodatabase (File or Multi-user)
  48. 48. Q u e s t i o n s
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×