WMS Performance Shootout




         Andrea Aime (GeoServer)
         Jeff McKenna (MapServer)




Open Source Geospatial...
Executive summary
• Compare the performance of WMS servers
   – GeoServer
   – MapServer
• In a number of different worklo...
Benchmarking History
• 3rd FOSS4G benchmarking exercise. Past exercises
  included:
   – FOSS4G 2007: Refractions Research...
Past MapServer improvements
•   improvements in large shapefile indexing
•   raster read optimization (single pass for mul...
Past GeoServer improvements
•   Overall rendering pipeline improvements
•   GML writing speed-ups
•   Raster data access i...
Rules of engagement
• Each server is tested in its latest version
• Each server performs exactly the same workload
   – Sa...
Hardware Configuration

                   WMS
                                               SDE
              GeoServer
...
Software Specs
• MapServer:
  – 5.6.0-beta3, GDAL SVN
  – Apache 2.2
  – FastCGI - 8 static processes
• GeoServer:
  – 2.0...
Hardware specs

• Bench (2003):
     Dell PowerEdge 1750
   – 1 Xeon Nocona 3Ghz, 1 GB RAM, OS: RedHat Enterprise 5.3
• WM...
Hot vs Cold Benchmarks
• Hot benchmark
   – The file system caches are fully loaded
   – The software has had the occasion...
Methodology
• Each test run performs requests with 1, 10, 20 and 40 parallel
  clients (for a total of 1500 requests)
• Ea...
Datasets Used
• Polygon layer: areawater_merge: the TIGER 2008 set of
  polygons describing water surfaces for the state o...
Equivalent traffic table
• The presentation charts results as throughput in requests per
  second.
• This table translates...
T1: plain polygon rendering
• TIGER 2008 “areawater” layer, merged over Texas
• Simply fill the polygon in blue, no outlin...
T1: Shapefiles
                        GeoServer      MapServer CGI     MapServer FCGI
 30


 25


 20


 15


 10


  5

...
T1: PostGIS
                             GeoServer   MapServer FCGI

 30


 25


 20


 15


 10


  5


  0
      1      ...
T1: Oracle
                                      GeoServer   MapServer FCGI

 25



 20



 15



 10



  5



  0
      ...
T1: SDE
                                  GeoServer    MapServer FCGI
 25


 20


 15


 10


  5


  0
      1           ...
T1: observations
• GeoServer is between 25% and 50% faster reading
  shapefiles out of the box, MapServer with FastCGI is ...
T2: point rendering
• GNIS names dataset over Texas
• Each point is rendered using a little GIF/PNG graphics




         ...
T2: shapefiles
                      GeoServer    MapServer CGI   MapServer FCGI
 30


 25


 20


 15


 10


  5


  0
 ...
T2: PostGIS
                           GeoServer   MapServer FCGI
 30


 25


 20


 15


 10


  5


  0
      1         ...
T2: Oracle
                            GeoServer   MapServer FCGI

 30

 25

 20

 15

 10

  5

  0
      1              ...
T2: SDE
                          GeoServer   MapServer FCGI

 30

 25

 20

 15

 10

  5

  0
      1                   ...
T3: roads map
• TIGER 2008 edges, just roads (rivers filtered out), three
  different line styles based on the road class
...
T3: shapefiles
                       GeoServer    MapServer CGI   MapServer FCGI

 20
 18
 16
 14
 12
 10
  8
  6
  4
  2...
T3: PostGIS
                            GeoServer   MapServer FCGI
 18

 16

 14

 12

 10

  8

  6

  4

  2

  0
      ...
T3: Oracle
                             GeoServer   MapServer FCGI

 14

 12

 10

  8

  6

  4

  2

  0
      1        ...
T3: SDE
                             GeoServer   MapServer FCGI
 14

 12

 10

  8

  6

  4

  2

  0
      1            ...
T4: labelled roads
• Same as T3, but higher scale (between 1:10k and 1:3k) and
  with labels following lines




         ...
T4: shapefiles
                       GeoServer    MapServer CGI   MapServer FCGI
 18

 16

 14

 12

 10

  8

  6

  4

...
T4: PostGIS
                              GeoServer   MapServer FCGI

 25


 20


 15


 10


  5


  0
      1           ...
T4: Oracle
                             GeoServer    MapServer FCGI
 20
 18
 16
 14
 12
 10
  8
  6
  4
  2
  0
      1   ...
T4: SDE
                             GeoServer   MapServer FCGI
 20
 18
 16
 14
 12
 10
  8
  6
  4
  2
  0
      1       ...
T5: raster
• Bluemarble TNG, as single ECW file, 512 mosaic TIFF, and
  BigTIFF (MapServer) vs 8 tiles mosaic TIFF (GeoSer...
T5: ECW
                      GeoServer         MapServer CGI     MapServer FCGI
 25


 20


 15


 10


  5


  0
      1...
T5: 512 tiles mosaic
                      GeoServer    MapServer CGI   MapServer FCGI
 18

 16

 14

 12

 10

  8

  6

...
T5: BigTiff vs 8 tiles tiff
                     GeoServer    MapServer CGI   MapServer FCGI
 30


 25


 20


 15


 10

...
FOSS4G 2009 improvements
• Several code enhancements were made during this year's
  exercise:
   – MapServer
      • Fixed...
Hopeful Changes for Next Shootout
• Involvement of more WMS servers
• Put the servers under a faster internet connection f...
More Information

• Scripts, configuration files, results stored in OSGeo SVN:
  http://svn.osgeo.org/osgeo/foss4g/benchma...
Additional MapServer Results (not
presented)




        Open Source Geospatial Foundation   42
MapServer Roads/Lines Test Results

                                                            Line Drawing
           20...
MapServer Labelled Roads/Lines Test
Results
                                                        Lines with Labels
    ...
MapServer Polygon Test Results

                                                           Polygon Drawing
            30
...
MapServer Point Test Results

                                                            Point Drawing
            30

  ...
MapServer Raster Test Results

                                                                                     Raster...
FastCGI Configuration Used for
      MapServer Tests


• FastCgiServer /var/www/cgi-bin/mapserv.fcgi -processes 8
  -initi...
Upcoming SlideShare
Loading in …5
×

Teste de performance de WMS entre GeoServer e Mapserver

5,168 views

Published on

Web conferência ministrada por Jeff Mckenna no Geolivre Conference 2009, comparando o teste de performance de WMS entre Geoserver e MapServer.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,168
On SlideShare
0
From Embeds
0
Number of Embeds
45
Actions
Shares
0
Downloads
146
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Teste de performance de WMS entre GeoServer e Mapserver

  1. 1. WMS Performance Shootout Andrea Aime (GeoServer) Jeff McKenna (MapServer) Open Source Geospatial Foundation 1
  2. 2. Executive summary • Compare the performance of WMS servers – GeoServer – MapServer • In a number of different workloads: – Vector: plain polygon, point with graphical icons, road network with cased lines, road network with labels – Raster: global imagery • Against different data backends: – Vector: shapefiles, PostGIS, Oracle spatial, SDE over Oracle spatial – Raster: ECW and GeoTiff mosaics Open Source Geospatial Foundation 2
  3. 3. Benchmarking History • 3rd FOSS4G benchmarking exercise. Past exercises included: – FOSS4G 2007: Refractions Research run and published the first comparison with the help of GeoServer and MapServer developers. Focus on big shapefiles, postgis, minimal styling – FOSS4G 2008: OpenGeo run and published the second comparison with some review from the MapServer developers. Focus on simple thematic mapping, raster data access, WFS and tile caching • Friendly competition: goal is to improve all software Open Source Geospatial Foundation 3
  4. 4. Past MapServer improvements • improvements in large shapefile indexing • raster read optimization (single pass for multiple bands) • enhancing polygon labelling placement • EPSG codes caching • PostGIS improvements • Oracle improvements Open Source Geospatial Foundation 4
  5. 5. Past GeoServer improvements • Overall rendering pipeline improvements • GML writing speed-ups • Raster data access improvements • PostGIS data store optimizations Open Source Geospatial Foundation 5
  6. 6. Rules of engagement • Each server is tested in its latest version • Each server performs exactly the same workload – Same set of WMS requests – Same data backends – Same image output format • All modifications made to improve performance are to be included in a future release of the software • Data used cannot be modified for display, other than indexing • All testing to be done on the same benchmarking machines Open Source Geospatial Foundation 6
  7. 7. Hardware Configuration WMS SDE GeoServer JMeter Oracle MapServer PostGIS Shapefiles Database Bench ECW GeoTIFF Open Source Geospatial Foundation 7
  8. 8. Software Specs • MapServer: – 5.6.0-beta3, GDAL SVN – Apache 2.2 – FastCGI - 8 static processes • GeoServer: – 2.0-RC2, Java 6 update18, GDAL 1.4.5 – Tomcat 6.0.20 - 8 threads • Database: – Oracle 11.1.0.7 – SDE 9.3.1 – PostGIS 1.4.0 Open Source Geospatial Foundation 8
  9. 9. Hardware specs • Bench (2003): Dell PowerEdge 1750 – 1 Xeon Nocona 3Ghz, 1 GB RAM, OS: RedHat Enterprise 5.3 • WMS (2004): – Dell PowerwEdge 2850 – 4 Xeon Nocona 3.4Ghz, 8 GB RAM – 6 73 GB, 15K RPM hard drives, OS: RedHat Enterprise 5.3 • Database (2007): – Dell Optiplex 755 tower – 1 x Intel Core2 Duo CPU E6750 @ 2.66GHz / Dual Core, 4Gb RAM – 100Gb SATA 3.0Gb/s 7200 rpm HD, OS: RedHat Enterprise 5.3 Open Source Geospatial Foundation 9
  10. 10. Hot vs Cold Benchmarks • Hot benchmark – The file system caches are fully loaded – The software has had the occasion to perform whatever pre-processing is necessary (e.g., open connections to the data source) • Cold benchmark – The system has been just started – There is no existing cache • Both are unrealistic, production is usually a mix of both • Hot benchmarks are more practical to run Open Source Geospatial Foundation 10
  11. 11. Methodology • Each test run performs requests with 1, 10, 20 and 40 parallel clients (for a total of 1500 requests) • Each test uses a random set of requests stored in a CSV file: no two requests in the same run are equal, but all servers perform the same workload • For each request the random factors are: – The image size (between 640x480 and 1024x768) – The geographic envelope (extent) • Each test is run three times in a row, the results of the third run are used for the comparison: this benchmark assumes full file system caches (“hot” benchmark) • The other GIS server is shut down while the tests are run Open Source Geospatial Foundation 11
  12. 12. Datasets Used • Polygon layer: areawater_merge: the TIGER 2008 set of polygons describing water surfaces for the state of Texas. 380000 polygons, 195MB shapefile, EPSG:4269 • Point layer: gnis_names09: all location and points of interest names for the state of Texas in the GNIS database. 103000 points, 2.8 MB shapefile, EPSG:4269 • Line layer, edges_merge: all line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. over 5M lines, 1.2GB shapefile, 1.4GB dbf, EPSG:4269 • Raster layer: Bluemarble TNG, 86400 x 43200 pixels, 7 overview layers (TIF, ECW, tiled 512&8, GeoRaster) Open Source Geospatial Foundation 12
  13. 13. Equivalent traffic table • The presentation charts results as throughput in requests per second. • This table translates hits per second at sustained peak load in terms of requests per hour and per day Hits/s Hits/hour Hits/day 5 18,000 432,000 10 36,000 864,000 15 54,000 1,296,000 20 72,000 1,728,000 25 90,000 2,160,000 Open Source Geospatial Foundation 13
  14. 14. T1: plain polygon rendering • TIGER 2008 “areawater” layer, merged over Texas • Simply fill the polygon in blue, no outline • Image scales roughly between 1:10.000 and 1:100.000 Open Source Geospatial Foundation 14
  15. 15. T1: Shapefiles GeoServer MapServer CGI MapServer FCGI 30 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer CGI MapServer FCGI 1 11 6.5 12 10 25 14.9 25.8 20 25.2 15.3 26.2 40 25.1 15.5 26.8 Open Source Geospatial Foundation 15
  16. 16. T1: PostGIS GeoServer MapServer FCGI 30 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer FCGI 1 10.7 10.1 10 24.4 23.7 20 25.2 24.2 40 25.7 24.5 Open Source Geospatial Foundation 16
  17. 17. T1: Oracle GeoServer MapServer FCGI 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer FCGI 1 10.2 9.2 10 23.1 21.1 20 23.8 21.7 40 23 22 Open Source Geospatial Foundation 17
  18. 18. T1: SDE GeoServer MapServer FCGI 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer FCGI 1 9.5 8.9 10 23.3 22.3 20 23.6 23.2 40 23.3 23.6 Open Source Geospatial Foundation 18
  19. 19. T1: observations • GeoServer is between 25% and 50% faster reading shapefiles out of the box, MapServer with FastCGI is slightly faster • PostGIS performs roughly 5% better than Oracle on higher loads • SDE seems to introduce no overhead compared to direct Oracle access for GeoServer, and a slight overhead for MapServer • GeoServer and MapServer have exactly the same performance when the backend is PostGIS and Oracle Open Source Geospatial Foundation 19
  20. 20. T2: point rendering • GNIS names dataset over Texas • Each point is rendered using a little GIF/PNG graphics Open Source Geospatial Foundation 20
  21. 21. T2: shapefiles GeoServer MapServer CGI MapServer FCGI 30 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer CGI MapServer FCGI 1 9.3 6.6 12.2 10 24.3 14.9 26.5 20 25 15.3 26.6 40 25.2 15.7 27.1 Open Source Geospatial Foundation 21
  22. 22. T2: PostGIS GeoServer MapServer FCGI 30 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer FCGI 1 11 10.8 10 25.1 24.3 20 25.3 24.6 40 25.8 25 Open Source Geospatial Foundation 22
  23. 23. T2: Oracle GeoServer MapServer FCGI 30 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer FCGI 1 10.5 9.8 10 25 23.3 20 25.2 23.4 40 24.6 23.9 Open Source Geospatial Foundation 23
  24. 24. T2: SDE GeoServer MapServer FCGI 30 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer FCGI 1 10.1 9.6 10 24.5 23.6 20 24.6 23.9 40 23.9 24.5 Open Source Geospatial Foundation 24
  25. 25. T3: roads map • TIGER 2008 edges, just roads (rivers filtered out), three different line styles based on the road class Open Source Geospatial Foundation 25
  26. 26. T3: shapefiles GeoServer MapServer CGI MapServer FCGI 20 18 16 14 12 10 8 6 4 2 0 1 10 20 40 Label GeoServer MapServer CGI MapServer FCGI 1 7.6 5 8.2 10 15 11 17.1 20 13.1 11 16.8 40 15.1 11.5 17.6 Open Source Geospatial Foundation 26
  27. 27. T3: PostGIS GeoServer MapServer FCGI 18 16 14 12 10 8 6 4 2 0 1 10 20 40 Label GeoServer MapServer FCGI 1 7.2 6.6 10 16.3 15.8 20 14.8 15.1 40 16.6 16.4 Open Source Geospatial Foundation 27
  28. 28. T3: Oracle GeoServer MapServer FCGI 14 12 10 8 6 4 2 0 1 10 20 40 Label GeoServer MapServer FCGI 1 5.9 5.6 10 12.5 12.6 20 11.5 11.6 40 12.7 12.8 Open Source Geospatial Foundation 28
  29. 29. T3: SDE GeoServer MapServer FCGI 14 12 10 8 6 4 2 0 1 10 20 40 Label GeoServer MapServer FCGI 1 6.1 4.7 10 13.3 11.1 20 11.6 9.7 40 13 11 Open Source Geospatial Foundation 29
  30. 30. T4: labelled roads • Same as T3, but higher scale (between 1:10k and 1:3k) and with labels following lines Open Source Geospatial Foundation 30
  31. 31. T4: shapefiles GeoServer MapServer CGI MapServer FCGI 18 16 14 12 10 8 6 4 2 0 1 10 20 40 Label GeoServer MapServer CGI MapServer FCGI 1 6.3 5.4 6.6 10 13.4 11.9 14.5 20 13.9 12.7 15.5 40 13.2 12.6 15.3 Open Source Geospatial Foundation 31
  32. 32. T4: PostGIS GeoServer MapServer FCGI 25 20 15 10 5 0 1 10 20 40 Label GeoServer MapServer FCGI 1 6.6 8 10 14.3 18.6 20 14.9 19.4 40 14.7 19.1 Open Source Geospatial Foundation 32
  33. 33. T4: Oracle GeoServer MapServer FCGI 20 18 16 14 12 10 8 6 4 2 0 1 10 20 40 Label GeoServer MapServer FCGI 1 6 6.8 10 13.2 16.8 20 13.9 17.5 40 13.5 17.3 Open Source Geospatial Foundation 33
  34. 34. T4: SDE GeoServer MapServer FCGI 20 18 16 14 12 10 8 6 4 2 0 1 10 20 40 Label GeoServer MapServer FCGI 1 5.6 6.4 10 12.8 16.4 20 13.3 17.4 40 13.1 17.2 Open Source Geospatial Foundation 34
  35. 35. T5: raster • Bluemarble TNG, as single ECW file, 512 mosaic TIFF, and BigTIFF (MapServer) vs 8 tiles mosaic TIFF (GeoServer) • ECW file is 260MB • TIFFs are uncompressed, tiled, with overviews. Around 16GB for each mosaic Open Source Geospatial Foundation 35
  36. 36. T5: ECW GeoServer MapServer CGI MapServer FCGI 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer CGI MapServer FCGI 1 5 4 7.4 10 11.1 12.6 20 20 11.2 12.8 19.8 40 10.5 12.6 19.1 Open Source Geospatial Foundation 36
  37. 37. T5: 512 tiles mosaic GeoServer MapServer CGI MapServer FCGI 18 16 14 12 10 8 6 4 2 0 1 10 20 40 Threads GeoServer MapServer CGI MapServer FCGI 1 4.8 2.7 6.5 10 9.7 11 14.9 20 9.6 11.4 16.4 40 9.9 11.3 16.9 Open Source Geospatial Foundation 37
  38. 38. T5: BigTiff vs 8 tiles tiff GeoServer MapServer CGI MapServer FCGI 30 25 20 15 10 5 0 1 10 20 40 Threads GeoServer MapServer CGI MapServer FCGI GeoServer cannot do 1 6.4 4.8 11.1 10 13.7 13.3 26.9 BigTiff, so a 8 tiles 20 13.3 15 28.1 mosaic was used 40 13.6 15.2 27.3 (almost 2G B per tile) Open Source Geospatial Foundation 38
  39. 39. FOSS4G 2009 improvements • Several code enhancements were made during this year's exercise: – MapServer • Fixed problem with labels following features • Fixed problem with 5.6.0-beta for SDE layers • Found problem with 5.6.0-beta for Oracle fastcgi and memory leak – GeoServer • Fixed problem with Oracle connection pooling and index scanning • Fixed problem with ECW crashes under high load Open Source Geospatial Foundation 39
  40. 40. Hopeful Changes for Next Shootout • Involvement of more WMS servers • Put the servers under a faster internet connection for quicker data transfers and tests setups • Modify the number of parallel client requests performed so that the 2-4-8 parallel clients case gets exposed • Phases/cycles for benchmarking – to allow for teams to interpret results and make software enhancements • Reprojection in both vector and raster data • Audience suggestions? Open Source Geospatial Foundation 40
  41. 41. More Information • Scripts, configuration files, results stored in OSGeo SVN: http://svn.osgeo.org/osgeo/foss4g/benchmarking/ • Wiki home: http://wiki.osgeo.org/wiki/Benchmarking_2009 • Mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking Open Source Geospatial Foundation 41
  42. 42. Additional MapServer Results (not presented) Open Source Geospatial Foundation 42
  43. 43. MapServer Roads/Lines Test Results Line Drawing 20 18 16 14 12 Throughput - 1 Maps/sec 10 Throughput - 10 8 Throughput - 20 6 Throughput - 40 4 2 0 shape 54 fcgi pg oracle oracle ogr sde fcgi shape shape 56 fcgi pg fcgi oracle fcgi sde Open Source Geospatial Foundation 43
  44. 44. MapServer Labelled Roads/Lines Test Results Lines with Labels 25 20 15 Throughput - 1 Maps/sec Throughput - 10 10 Throughput - 20 Throughput - 40 5 0 shape 54 fcgi pg oracle oracle ogr sde fcgi shape shape 56 fcgi pg fcgi oracle fcgi sde Open Source Geospatial Foundation 44
  45. 45. MapServer Polygon Test Results Polygon Drawing 30 25 20 Throughput - 1 Maps/sec 15 Throughput - 10 Throughput - 20 10 Throughput - 40 5 0 Shape 54 fcgis pg oracle oracle ogr sde fcgi shape shape 56 fcgi pg fcgi oracle fcgi sde Open Source Geospatial Foundation 45
  46. 46. MapServer Point Test Results Point Drawing 30 25 20 Throughput - 1 Maps/sec 15 Throughput - 10 Throughput - 20 10 Throughput - 40 5 0 shape 54 fcgi pg oracle oracle ogr sde fcgi shape shape 56 fcgi pg fcgi oracle fcgi sde Open Source Geospatial Foundation 46
  47. 47. MapServer Raster Test Results Raster 30 25 20 Throughput - 1 Maps/sec 15 Throughput - 10 Throughput - 20 10 Throughput - 40 5 0 bigtiff fcgi 54 ecw cgi 56 ecw fcgi 56 tiled tiff fcgi 54 bigtiff cgi 56 bigtiff fcgi 56 ecw fcgi 54 tiled tiff cgi 56 tiled tiff fcgi 56 Open Source Geospatial Foundation 47
  48. 48. FastCGI Configuration Used for MapServer Tests • FastCgiServer /var/www/cgi-bin/mapserv.fcgi -processes 8 -initial_env LD_LIBRARY_PATH=/usr/local/lib:/opt/build/instant_client_11_1/li b:/opt/build/arcsde_c_sdk/lib Open Source Geospatial Foundation 48

×