WMS Performance Shootout 2009

21,645 views
21,432 views

Published on

Presented at FOSS4G 2009 in Sydney, Australia

Published in: Technology
2 Comments
12 Likes
Statistics
Notes
No Downloads
Views
Total views
21,645
On SlideShare
0
From Embeds
0
Number of Embeds
6,192
Actions
Shares
0
Downloads
432
Comments
2
Likes
12
Embeds 0
No embeds

No notes for slide

WMS Performance Shootout 2009

  1. 1. WMS Performance Shootout Andrea Aime (GeoServer) Jeff McKenna (MapServer)
  2. 2. Executive summary <ul><li>Compare the performance of WMS servers </li><ul><li>GeoServer
  3. 3. MapServer </li></ul><li>In a number of different workloads: </li><ul><li>Vector: plain polygon, point with graphical icons, road network with cased lines, road network with labels
  4. 4. Raster: global imagery </li></ul><li>Against different data backends: </li><ul><li>Vector: shapefiles, PostGIS, Oracle spatial, SDE over Oracle spatial
  5. 5. Raster: ECW and GeoTiff mosaics </li></ul></ul>
  6. 6. Benchmarking History <ul><li>3 rd FOSS4G benchmarking exercise. Past exercises included: </li><ul><li>FOSS4G 2007: Refractions Research run and published the first comparison with the help of GeoServer and MapServer developers. Focus on big shapefiles, postgis, minimal styling
  7. 7. FOSS4G 2008: OpenGeo run and published the second comparison with some review from the MapServer developers. Focus on simple thematic mapping, raster data access, WFS and tile caching </li></ul><li>Friendly competition: goal is to improve all software </li></ul>
  8. 8. Past MapServer improvements <ul><li>improvements in large shapefile indexing
  9. 9. raster read optimization (single pass for multiple bands)
  10. 10. enhancing polygon labelling placement
  11. 11. EPSG codes caching
  12. 12. PostGIS improvements
  13. 13. Oracle improvements </li></ul>
  14. 14. Past GeoServer improvements <ul><li>Overall rendering pipeline improvements
  15. 15. GML writing speed-ups
  16. 16. Raster data access improvements
  17. 17. PostGIS data store optimizations </li></ul>
  18. 18. Rules of engagement <ul><li>Each server is tested in its latest version
  19. 19. Each server performs exactly the same workload </li><ul><li>Same set of WMS requests
  20. 20. Same data backends
  21. 21. Same image output format </li></ul><li>All modifications made to improve performance are to be included in a future release of the software
  22. 22. Data used cannot be modified for display, other than indexing
  23. 23. All testing to be done on the same benchmarking machines </li></ul>
  24. 24. Hardware Configuration Bench WMS Database JMeter GeoServer MapServer Shapefiles ECW GeoTIFF Oracle PostGIS SDE
  25. 25. Software Specs <ul><li>MapServer: </li><ul><li>5.6.0-beta3, GDAL SVN
  26. 26. Apache 2.2
  27. 27. FastCGI - 8 static processes </li></ul><li>GeoServer: </li><ul><li>2.0-RC2, Java 6 update18, GDAL 1.4.5
  28. 28. Tomcat 6.0.20 - 8 threads </li></ul><li>Database: </li><ul><li>Oracle 11.1.0.7
  29. 29. SDE 9.3.1
  30. 30. PostGIS 1.4.0 </li></ul></ul>
  31. 31. Hardware specs <ul><li>Bench (2003): </li><ul><li>Dell PowerEdge 1750
  32. 32. 1 Xeon Nocona 3Ghz, 1 GB RAM, OS: RedHat Enterprise 5.3 </li></ul><li>WMS (2004): </li><ul><li>Dell PowerwEdge 2850
  33. 33. 4 Xeon Nocona 3.4Ghz, 8 GB RAM
  34. 34. 6 73 GB, 15K RPM hard drives, OS: RedHat Enterprise 5.3 </li></ul><li>Database (2007): </li><ul><li>Dell Optiplex 755 tower
  35. 35. 1 x Intel Core2 Duo CPU E6750 @ 2.66GHz / Dual Core, 4Gb RAM
  36. 36. 100Gb SATA 3.0Gb/s 7200 rpm HD, OS: RedHat Enterprise 5.3 </li></ul></ul>
  37. 37. Hot vs Cold Benchmarks <ul><li>Hot benchmark </li><ul><li>The file system caches are fully loaded
  38. 38. The software has had the occasion to perform whatever pre-processing is necessary (e.g., open connections to the data source) </li></ul><li>Cold benchmark </li><ul><li>The system has been just started
  39. 39. There is no existing cache </li></ul><li>Both are unrealistic, production is usually a mix of both
  40. 40. Hot benchmarks are more practical to run </li></ul>
  41. 41. Methodology <ul><li>Each test run performs requests with 1, 10, 20 and 40 parallel clients (for a total of 1500 requests)
  42. 42. Each test uses a random set of requests stored in a CSV file: no two requests in the same run are equal, but all servers perform the same workload
  43. 43. For each request the random factors are: </li><ul><li>The image size (between 640x480 and 1024x768)
  44. 44. The geographic envelope (extent) </li></ul><li>Each test is run three times in a row, the results of the third run are used for the comparison: this benchmark assumes full file system caches (“hot” benchmark)
  45. 45. The other GIS server is shut down while the tests are run </li></ul>
  46. 46. Datasets Used <ul><li>Polygon layer: areawater_merge : the TIGER 2008 set of polygons describing water surfaces for the state of Texas. 380000 polygons, 195MB shapefile, EPSG:4269
  47. 47. Point layer: gnis_names09 : all location and points of interest names for the state of Texas in the GNIS database. 103000 points, 2.8 MB shapefile, EPSG:4269
  48. 48. Line layer, edges_merge : all line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. over 5M lines, 1.2GB shapefile, 1.4GB dbf, EPSG:4269
  49. 49. Raster layer : Bluemarble TNG, 86400 x 43200 pixels, 7 overview layers (TIF, ECW, tiled 512&8, GeoRaster) </li></ul>
  50. 50. Equivalent traffic table <ul><li>The presentation charts results as throughput in requests per second.
  51. 51. This table translates hits per second at sustained peak load in terms of requests per hour and per day </li></ul>
  52. 52. T1: plain polygon rendering <ul><li>TIGER 2008 “areawater” layer, merged over Texas
  53. 53. Simply fill the polygon in blue, no outline
  54. 54. Image scales roughly between 1:10.000 and 1:100.000 </li></ul>
  55. 55. T1: Shapefiles
  56. 56. T1: PostGIS
  57. 57. T1: Oracle
  58. 58. T1: SDE
  59. 59. T1: observations <ul><li>GeoServer is between 25% and 50% faster reading shapefiles out of the box, MapServer with FastCGI is slightly faster
  60. 60. PostGIS performs roughly 5% better than Oracle on higher loads
  61. 61. SDE seems to introduce no overhead compared to direct Oracle access for GeoServer, and a slight overhead for MapServer
  62. 62. GeoServer and MapServer have exactly the same performance when the backend is PostGIS and Oracle </li></ul>
  63. 63. T2: point rendering <ul><li>GNIS names dataset over Texas
  64. 64. Each point is rendered using a little GIF/PNG graphics </li></ul>
  65. 65. T2: shapefiles
  66. 66. T2: PostGIS
  67. 67. T2: Oracle
  68. 68. T2: SDE
  69. 69. T3: roads map <ul><li>TIGER 2008 edges, just roads (rivers filtered out), three different line styles based on the road class </li></ul>
  70. 70. T3: shapefiles
  71. 71. T3: PostGIS
  72. 72. T3: Oracle
  73. 73. T3: SDE
  74. 74. T4: labelled roads <ul><li>Same as T3, but higher scale (between 1:10k and 1:3k) and with labels following lines </li></ul>
  75. 75. T4: shapefiles
  76. 76. T4: PostGIS
  77. 77. T4: Oracle
  78. 78. T4: SDE
  79. 79. T5: raster <ul><li>Bluemarble TNG, as single ECW file, 512 mosaic TIFF, and BigTIFF (MapServer) vs 8 tiles mosaic TIFF (GeoServer)
  80. 80. ECW file is 260MB
  81. 81. TIFFs are uncompressed, tiled, with overviews. Around 16GB for each mosaic </li></ul>
  82. 82. T5: ECW
  83. 83. T5: 512 tiles mosaic
  84. 84. T5: BigTiff vs 8 tiles tiff GeoServer cannot do BigTiff, so a 8 tiles mosaic was used (almost 2G B per tile)
  85. 85. FOSS4G 2009 improvements <ul><li>Several code enhancements were made during this year's exercise: </li><ul><li>MapServer </li><ul><li>Fixed problem with labels following features
  86. 86. Fixed problem with 5.6.0-beta for SDE layers
  87. 87. Found problem with 5.6.0-beta for Oracle fastcgi and memory leak </li></ul><li>GeoServer </li><ul><li>Fixed problem with Oracle connection pooling and index scanning
  88. 88. Fixed problem with ECW crashes under high load </li></ul></ul></ul>
  89. 89. Hopeful Changes for Next Shootout <ul><li>Involvement of more WMS servers
  90. 90. Put the servers under a faster internet connection for quicker data transfers and tests setups
  91. 91. Modify the number of parallel client requests performed so that the 2-4-8 parallel clients case gets exposed
  92. 92. Phases/cycles for benchmarking – to allow for teams to interpret results and make software enhancements
  93. 93. Reprojection in both vector and raster data
  94. 94. Audience suggestions? </li></ul>
  95. 95. More Information <ul><li>Scripts, configuration files, results stored in OSGeo SVN: http://svn.osgeo.org/osgeo/foss4g/benchmarking/
  96. 96. Wiki home: http://wiki.osgeo.org/wiki/Benchmarking_2009
  97. 97. Mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking </li></ul>
  98. 98. Additional MapServer Results (not presented)
  99. 99. MapServer Roads/Lines Test Results
  100. 100. MapServer Labelled Roads/Lines Test Results
  101. 101. MapServer Polygon Test Results
  102. 102. MapServer Point Test Results
  103. 103. MapServer Raster Test Results
  104. 104. FastCGI Configuration Used for MapServer Tests <ul><li>FastCgiServer /var/www/cgi-bin/mapserv.fcgi -processes 8 -initial_env LD_LIBRARY_PATH=/usr/local/lib:/opt/build/instant_client_11_1/lib:/opt/build/arcsde_c_sdk/lib </li></ul>

×