Map Server comparison, OGC WMS - Random Extent

1,019 views
980 views

Published on

Michal Šeliga: Map Server comparison, OGC WMS - Random Extent

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
1,019
On SlideShare
0
From Embeds
0
Number of Embeds
54
Actions
Shares
0
Downloads
0
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Map Server comparison, OGC WMS - Random Extent

  1. 1. Map server comparison OGC WMS – Random ExtentAuthor: Ing. Michal Šeliga, Ph.D., E-mail: michal.seliga@tmapy.czDate: 2010-03-25The objective of this test is to check a server ability to handle WMS requests for random extentand constant output size without having to perform data transformation between differentSRS.Test description and settingTest client is increasing the number of active clients linearly in time, who simultaneously querythe server. Each of an active clients sends its request immediately after that what is previousrequest finished and each of the newly generated request has a different random extent.The proposed test should check the following qualities: • Dealing with physical data, • Scaling, • Transformation of raster data into the desired output format (PNG), • Ability to respond to the huge application server load.DataAs test data the aerial photographs of Prague were used, with resolution of 10 cm, which weremosaic by EIM into one large IMG file, than were built pyramids the IMG file. The final size ofthe test data was 290GB, including the pyramids.SettingNumber of users: 1 – 300Delay between two users tests: 0 sDelay to create an additional user: 15 sTotal test time: 5000 sOutput size: 1200x800Output format: PNGHWServer: Windows 2003 64bit, Sun Fire X2200 (2x QuadCore, 8GB RAM, 2x HDD 400GB, 3xEthernet 1Gb, 1x Ethernet 100Gb), System and data are stored on different HDD.Client: Linux Ubuntu 8.10 LTS, PC 1x DualCore, 2GB RAM, 1x Ethernet 1Gb.Test client machine is connected to the server directly via one STP CAT6 cable. Connection usetwo reserved network cards, which are configure to special subnet.
  2. 2. ArcGIS ServerInstallation: LIKUInstallation and configuration: − AGS 9.3.1 .NET, − IIS, − Instances: min. 7 - max. 7, − Only WMS, − No user restrictions defined.WMS service average response timeWith native technology that uses AGS is behavior of AGS surprisingly harmonized withrelatively small demands on system resources and hardware. Server behavior is characterizedby nearly 100% linear dependence, the calculated coefficient of linear regression dependenceis equal to 0.99 after rounding to two decimal places. GetMap AGS + IIS 60000 50000 f(x) = 307.15 x^0.89 40000 R² = 0.99 Response [ms] AvgTotal 30000 Power Regression for AvgTotal 8x Core 20000 1x Core 10000 0 0 50 100 150 200 250 300 Users Graph 5: The average response time of WMS service, including downloading a picture§Curves labeled as 1x Core and 8x Core illustrate the theoretical course of conduct in case of amap server load balancing on one and eight cores. The calculation is based on the averagevalue of "Response time" for a single user.Number of requests processed, based on the number of usersIn this case, we can leave the test results almost uncommented. Only an explanation of errorsthat occurred during the test might deserve our attention. These errors are replicable andclearly not related to the tests carried out. After analysis, recording errors, we concluded thatit is an internal error in the implementation of AGS. This error occurs always at therequirements, which require “non-standard extent”. Specifically, an extent, which has a ratioof height and width smaller than 1/3000. This problem probably does not occur in average useand therefore, we do not give too much importance to it.
  3. 3. GetMap: request count AGS + IIS 100 80 60Requests OK IOError 40 ServiceError 20 0 0 50 100 150 200 250 300 Users Graph 6: Number of requests processed, depending on the number of users GetMap: wrong response count detail AGS + IIS 5 4 3Requests OK 2 IOError ServiceError 1 0 0 50 100 150 200 250 300 Users Graph 7: Number of requests processed, depending on the number of users (detail) GetMap: requests per second AGS + IIS 5 4Requests / s 3 OK/s 2 IOError/s ServiceError/s 1 0 0 50 100 150 200 250 300 Users Graph 8: Number of requests processed, depending on the number of users per second GetMap: wrong responses per second AGS + IIS 0.5 0.4 0.3Requests / s OK/s IOError/s 0.2 ServiceError/s 0.1 0 0 50 100 150 200 250 300 Users Graph 9: Number of requests processed, depending on the number of users per second (detail)
  4. 4. As already mentioned AGS is responsible for some of the ServiceErrors, the amount of such responses is summarized in the following chart. Dependence of number of errors is linear inthe number of random requests. AGS Users Median Sum Requests/s Requests Response time Requests IOError ServiceError 1 1.03 19.00 836.00 19 0 0 10 3.31 66.50 1262.00 568 0 2 50 2.86 74.00 5207.00 3561 0 5 100 2.41 77.00 9941.50 7507 0 8 200 1.75 79.00 19102.00 15838 0 21 300 1.41 82.00 27427.00 24531 0 29 Table 1: Tests result conclusion
  5. 5. ERDAS Apollo ServerInstallation: MISEInstallation and configuration: − EAS 9.3.2, − JAVA 1.5.0_18-b02, − Tomcat6 (NIO adapter), − Instances: min. 7 - max. 7, − No user restrictions defined.WMS service average response timeBehavior EAS showed considerable fluctuations, which were probably result of the "Garbagecollector" work in the JVM and, then by the greater memory load was running the applicationserver itself. EAS was tested in combination with the application server Tomcat6, but it is alsopossible to use a "big application server”. Deployment, combined with a commercial applicationserver such as Oracle AS, Weblogic or Websphere results could still bring a little better results. GetMap EAS + Tomcat6 100000 80000 Response [ms] 60000 AvgTotal 40000 Power Regression for AvgTotal 8x Core 20000 f(x) = 82.79 x^1.05 1x Core R² = 0.91 0 0 50 100 150 200 250 300 Users Graph 10: The average response time of WMS service, including downloading a pictureCurves labeled as 1x Core and 8x Core illustrate the theoretical course of conduct in case of amap server load is balancing on one and eight cores. The calculation is based on the averagevalue of "Response time" for one single user.Number of requests processed, depending on the number of users GetMap: request count EAS + Tomcat6 280 240 200 160 Requests OK 120 IOError ServiceError 80 40 0 0 50 100 150 200 250 300 Users Graph 11: Number of requests processed, depending on the number of users
  6. 6. GetMap: requests per second EAS + Tomcat6 12 10 8 Requests / s 6 OK/s IOError/s 4 ServiceError/s 2 0 0 50 100 150 200 250 300 Users Graph 12: Number of requests processed, depending on the number of users per secondThe following table show that all EAS server responses were successful. EAS Users Median Sum Requests/s Requests Response time Requests IOError ServiceError 1 1.87 33.00 460.00 33 0 0 10 6.81 142.00 605.00 1263 0 0 50 7.41 186.00 2021.00 8566 0 0 100 5.63 178.00 4195.50 16489 0 0 200 4.54 176.00 9478.50 32313 0 0 300 3.19 167.00 14850.00 44816 0 0 Table 2: Tests result conclusion
  7. 7. ERDAS Image ManagerInstallation: MISEInstallation and configuration − EAIM 9.3.2, − JAVA 1.5.0_18-b02, − Jboss 4.2.2 GA (NIO adapter), − Instances: min. 7 - max. 7, − Authentication and authorization is turned on: If the authentication and authorization for access to WMS services has been turned off, in a state of load (more than 5 users at the same time) the server turned off and on from time to time returned.Note - The execution of each request requires an access to the database, which first search for required layer or layers information and then verifies the access user’s rights to work with the layer and its extent.WMS service average response timeEAIM behavior showed considerable fluctuations, which resulted from the "Garbage collector"work in the JVM and then from the greater memory load running the application server itself. GetMap EAIM + JBoss 100000 80000 Response [ms] 60000 AvgTotal 40000 Power Regression for AvgTotal 8x Core 20000 f(x) = 96.15 x^1.03 1x Core R² = 0.85 0 0 50 100 150 200 250 300 Users Graph 13: The average response time of WMS service, including downloading a pictureCurves labeled as 1x Core and 8x Core illustrate the theoretical course of conduct in case of amap server load balancing on one and eight cores. The calculation is based on the averagevalue of "Response time" for a single user.
  8. 8. Number of requests processed, depending on the number of users GetMap: request count EAIM + JBoss 280 240 200 160 Requests OK 120 80 40 0 0 50 100 150 200 250 300 Users Graph 14: Number of requests processed, depending on the number of users GetMap: requests per second EAIM + JBoss 12 10 8 Requests / s 6 OK/s IOError/s 4 ServiceError/s 2 0 0 50 100 150 200 250 300 Users Graph 15: Number of requests processed, depending on the number of users per secondThe following table show that all EAIM server responses were successful. EAIM Users Median Sum Requests/s Requests Response time Requests IOError ServiceError 1 1.55 27.00 557.00 27 0 0 10 4.31 99.00 825.50 909 0 0 50 6.08 168.50 2322.50 7725 0 0 100 6.76 190.00 4291.00 17297 0 0 200 4.19 171.00 8209.00 31538 0 0 300 2.77 150.00 16735.00 43430 0 0 Table 3: Tests results conclusion
  9. 9. ERDAS APOLLO ProfessionalInstallation: MISEInstallation and configuration − EAP built on 2009.11.12 − JAVA 1.5.0_20-b02 − JBoss AS 4.2.2.GA − Configuration and its optimalization are completely identical with the setting made for the installation EAIM 2009 (EAIM 9.3.2)WMS service average response timeEAP behavior showed considerable fluctuations, which resulted from the "Garbage collector"work in the JVM and then from the greater memory load running the application server itself.The most significant fluctuations always occurred in connection with the increased number ofrequests in the disk I/O queue. GetMap EAP + JBoss 100000 80000 Response [ms] 60000 AvgTotal 40000 Power Regression for AvgTotal f(x) = 79.11 x^1.1 8x Core 1x Core 20000 R² = 0.91 0 0 50 100 150 200 250 300 Users Graph 16: The average response time of WMS service, including downloading a pictureCurves labeled as 1x Core and 8x Core illustrate the theoretical course of conduct in case of amap server load balancing on one and eight cores. The calculation is based on the averagevalue of "Response time" for a single user.
  10. 10. Number of requests processed, depending on the number of users GetMap: request count EAP + JBoss 280 240 200 160 Requests OK 120 IOError ServiceError 80 40 0 0 50 100 150 200 250 300 Users Graph 17: Number of requests processed, depending on the number of users GetMap: requests per second EAP + JBoss 12 10 8 Requests / s OK/s 6 IOError/s ServiceError/s 4 2 0 0 50 100 150 200 250 300 Users Graph 18: Number of requests processed, depending on the number of users per second EAP Users Median Sum Requests/s Requests Response time Requests IOError ServiceError 1 1.28 23.00 676.00 23 0 0 10 6.51 117.50 736.50 1082 0 0 50 6.78 155.50 2371.00 7443 0 0 100 5.47 150.50 5420.00 14574 0 0 200 4.00 143.00 10514.50 26609 0 0 300 2.72 132.00 18218.00 36516 0 0 Table 4: Tests results conclusion
  11. 11. ERDAS APOLLO Professional (optimal)Instalace: MISEInstallation and configuration − EAP built on 2009.11.12 − JAVA 1.5.0_20-b02 − JBoss AS 4.2.2.GA − Configuration is identical to the previous settings EAP and EAIM with the only difference: on the basis of empirical tests, I came to the adjusting of the value for a minimum and maximum number of RDS value instances: minimum number of instance is 6 and maximum number of instance is 6.WMS service average response timeEAP behavior showed considerable fluctuations, which resulted from the "Garbage collector"work in the JVM and then from the greater memory load running the application server itself.The most significant fluctuations always occurred in connection with the increased number ofrequests in the disk I/O queue. GetMap EAP + JBoss 100000 80000 Response [ms] 60000 AvgTotal 40000 Power Regression for AvgTotal f(x) = 79.11 x^1.1 8x Core 1x Core 20000 R² = 0.91 0 0 50 100 150 200 250 300 Users Graph 19: The average response time of WMS service, including downloading a picture GetMap EAP + JBoss 100000 80000 Response [ms] 60000 AvgTotal 40000 Power Regression for AvgTotal 8x Core f(x) = 96.68 x^1.04 1x Core 20000 R² = 0.91 0 0 50 100 150 200 250 300 Users Graph 20: The average response time of WMS service, including downloading a pictureCurves labeled as 1x Core and 8x Core illustrate the theoretical course of conduct in case of amap server load balancing on one and eight cores. The calculation is based on the averagevalue of "Response time" for a single user.
  12. 12. Number of requests processed, depending on the number of users GetMap: request count EAP + JBoss 280 240 200 160 Requests OK 120 IOError ServiceError 80 40 0 0 50 100 150 200 250 300 Users Graph 21: Number of requests processed, depending on the number of users GetMap: requests per second EAP + JBoss 12 10 8 Requests / s OK/s 6 IOError/s ServiceError/s 4 2 0 0 50 100 150 200 250 300 Users Graph 22: Number of requests processed, depending on the number of users per second EAP (optimal) Users Median Sum Requests/s Requests Response time Requests IOError ServiceError 1 1.36 24.00 632.00 24 0 0 10 5.59 105.50 730.00 1011 0 0 50 6.43 148.00 2402.50 7156 0 0 100 5.43 148.00 5168.50 14449 0 0 200 4.58 148.00 10297.00 28721 0 0 300 3.78 146.00 16371.00 40041 0 0 Table 5: Tests results conclusion
  13. 13. ConclusionThe performed test is a compromise among the other proposed tests. This test is the bestapproaching the real state from all of them. The objective of this test was to examine themethodology of testing and technique of treatment results processing. During testing, we havespent a lot of time tweaking the optimal settings of the test servers. AGS configurationschange its performance of about 5%, but in the case of ERDAS APOLLO it was about 200%.ERDAS has better quality output at smaller output sizeResolution is the same and color depth for both was 24bit per pixel. Real number of colors:[ERDAS, 49,462], [ESRI, 71,897].File size of ESRI output is larger by about 60% (probably because of there is a greater numberof colors). Even if the ESRI output is larger, but according to technical parameters better itlooks "eyemetricaly" worse. ERDAS probably has better histogram functions.ERDAS GIF output is also better (significantly better). ERDAS GIF compare to ESRI issmaller and at the same color depth (8 bits as is custom for GIF) it gives a nice picture. Itmight be caused by the fact that ESRI perhaps insufficiently recasts the color depth.GIF output: Picture 1: EAIM (GIF)§
  14. 14. Picture 2: AGS (GIF)PNG output: Picture 3: EAIM (PNG)
  15. 15. Picture 4: AGS (PNG)AGS has a balanced predictable behavior depending on CPUs numberAGS has a very balanced behavior, which is almost linear. This allows sufficient accuracyin predicting the speed of responses and system resources in advance.AGS is primarily written by ArcObjects, which are native COM objects and these are wrappedwith .NET interface.The main reason for AGS balanced behavior with comparison to EA* is that AGS uses the allcapacity on the CPU levels soon and then no longer depends on the permeability of CPU (diskI/O queue is almost empty during the test, the memory is used by the two thirds only). EA*does the opposite, the CPU uses the capacity at 60% on average and processes are waiting forprocessing requests from the disk I/O queue, application server (AS) process is limited byJAVA 32bit, which does not allocate enough memory for large amount of clients => EA* ismore prone to speed of I/O operations and memory size.Another reason for the uneven EA* behavior is that JAVA works with larger amount of memoryand sometimes the operating system had problems with the allocation of large blocks ofmemory (384 megabytes).Unbalanced performance of ERDAS APOLLO is also due to JAVA technology in which is writtenand operated. JAVA unlike native code has a different approach to memory management, theJVM free memory via Garbage Collector (GC), which from time to time, or if the free memoryis low, suspends the running program and finds all instances of objects having no furtherreference and releases them.AGS requires less system resourcesDue to AGS has been installed in .NET version, than application server MS IIS was used, which
  16. 16. unlike the JAVA AS is written for native environment. This has the effect of minimum reductionof the required resources, especially memory of the server.AGS has an internal restriction causing a ServiceErrorImplementation of AGS has probably an internal restriction, which causes that some queriesresults are ServiceError. ERDAS APOLLO did not come back a single wrong answer during thetest.These errors can be replicated outside the test as well - after consultation with VLAMA, LIKUand TONO we agreed that AGS response ServiceError occurred in extreme cases whererequested extent has big difference between height and width. The ServiceError is caused byunknown internal error "Unknown exception - Internal Server Error". This problem wouldnot theoretically occur in common operations => It does not seem to be important.ERDAS shows less dependence on the number of usersThe following table shows coefficients of dependency based on response time (Response time)and the number of users, according to limit 300 users. Input values for calculations were the300 users and a maximum value of the "Response time", taken from the regression curvesbased on "Response time" and "the number of users" (bigger value indicates strongerdependency). Response time [s] Coefficient AGS 50 0.17 EAIM 38 0.13 EAS 37 0.12 EAP 39 0.13 Table 6: Coefficient of dependency between response time and number of usersERDAS is significantly fasterThe following table shows the percentage performance comparison ERDAS APOLLO Serversand ArcGIS Server for the given number of users (AGS means 100%). These results must betreated deliberately, because the input values for median are taken from continuousincremental test with a defined constant step.It must be taken into account, that the results are related to the above describeddataset and in case of individual datasets (data formats) the results may vary.To obtain more accurate results, it would be essential to prolong the period between additionof requested user to get server time to stabilize for the current number of users. The timebetween additions of users should be significantly longer than the response time "Responsetime".
  17. 17. EAIM EAS Users Requests/s Requests Response time Requests/s Requests Response time 1 149.83 142.11 66.63 181.08 173.68 55.02 10 158.16 148.87 65.41 224.31 213.53 47.94 50 241.72 227.70 44.60 263.83 251.35 38.81 100 261.87 246.75 43.16 242.58 231.17 42.20 200 228.23 216.46 42.97 232.27 222.78 49.62 300 192.87 182.93 61.02 212.33 203.66 54.14 Table 7: ERDAS APOLLO 2009 percentage compared with ArcGIS ServerAs shown above (Table 6) EAS is slightly more powerful than EAIM. The performance differencecan be explained by the fact that EAIM compared to EAS also tests included evaluation of theERDAS Catalog (user rights depend on layer and extent). The difference is also influenced bythe application server, for the EAS was chosen Tomcat6 servlet container and for EAIM it wasJBoss 4.2 with EJB3 support. EAP – optimal [%] EAP [%] Users Requests/s Requests Response time Requests/s Requests Response time 1 132.12 126.32 75.60 123.65 121.05 80.86 10 168.62 158.65 57.84 196.40 176.69 58.36 50 225.12 200.00 46.14 237.29 210.14 45.53 100 225.36 192.21 51.99 227.21 195.45 54.52 200 261.53 187.34 53.91 228.18 181.01 55.04 300 268.72 178.05 59.69 192.97 160.98 66.42 Table 8: ERDAS APOLLO 2010 percentage compared with ArcGIS Server Table 9:Table 8 compares the results of the tests of servers ArcGIS and ERDAS APOLLO, where AGS istaken as a reference and it represents 100% as well as in the previous chart. It says that asfor as a number of requests bigger is better, and regarding the response time smaller is betteron the other hand. The results show that EAP is not significantly different from those of hispredecessor EAIM and retains a significant performance advantage over AGS.It is worth mentioning the percentage difference between the values of “Requests/s” and“Requests”. Those differences mirror the load and direction of an application server. As thenumber of requests to a server rises, the time required for an initialization of TCP connectionbetween the test client and application server is increasing as well. This as a direct resultmeans that the client in a given period of time does not manage to send so many requests to asever, as in case of small load (small number of clients asking at the same time), thereforesuch a result increases with the number of users simultaneously.
  18. 18. GetMap: response time Comparation 80000 70000 60000 50000Response [ms] EAP + JBoss 40000 EAIM + Jboss AGS + IIS 30000 EAS + Tomcat6 20000 10000 0 0 50 100 150 200 250 300 Users Graph 23: Comparison of speed responses depending on the number of users GetMap: response time Comparation 20000 18000 16000 14000 12000Response [ms] EAP + JBoss 10000 EAIM + Jboss 8000 AGS + IIS EAS + Tomcat6 6000 4000 2000 0 0 10 20 30 40 50 60 70 80 90 100 Users Graph 24: Comparison of speed responses depending on the number of users (detail) GetMap: requests per second Comparation 14 12 10 8Requests / s EAP + JBoss 6 EAIM + Jboss AGS + IIS 4 EAS + Tomcat6 2 0 -2 0 50 100 150 200 250 300 UsersGraph 25: Comparing the average number of correct answers per second, depending on the number of users
  19. 19. GetMap: requests per second Comparation 14 12 10 8 Requests / s EAP + JBoss EAIM + Jboss 6 AGS + IIS EAS + Tomcat6 4 2 0 0 10 20 30 40 50 60 70 80 90 100 Users Graph 26: Comparing the average number of correct answers per second, depending on the number of users (detail)ERDAS 2010 versus ERDAS 2009Based on the results published earlier there was a comparison of two product lines EAIM 2009and EAP 2010. The results of that comparison are summarized in the following table, where asthe comparative criteria (100%) EAIM tests results were used. For values of “Request/s” and“Requests” exceeding 100% it means that ERDAS 2010 surpassed ERDAS 2009. The remainingcolumns “Response time”, IOError and ServiceError the opposite is true.The results show that the performance of EAP exceeds the EAIM in case that we compare themaccording the character “Requests/s” and especially if there is a large number of users (morethan 200). When comparing EAP and EAIM by the total number of processed requests for thenumber of users, the power of EAP is approximately 10% worse than in case of EAIM 2009.It is interesting to see the results minding the fact that the quicker are the responses persecond in case of EAP, the number of processed requests declined. Such a behavior is causedby testing EAP 2010, where more intensive latency was identified during establishing andterminating connections with this application server. It resulted in sending a smaller possiblemaximum number of client requests to a server and resulting in a burden on the server, andtherefore faster processing of requests received. This increased latency might be caused by thechanges in behavior at side of: − OS Windows 2003 as a result of installing security patches, servlets and their service in AS − AS JBoss and configuration changes beyond my adjustments compare to the original versions EAP 2010 / EAIM 2009 [%] Users Median Sum Requests/s Requests Response time Requests IOError ServiceError 1 88.18 88.89 113.46 88.89 100.00 100.00 10 129.58 106.57 88.43 111.22 100.00 100.00 50 105.86 87.83 103.44 92.63 100.00 100.00 100 80.29 77.89 120.45 83.53 100.00 100.00 200 109.33 86.55 125.44 91.07 100.00 100.00 300 136.59 97.33 97.82 92.20 100.00 100.00 Table 10: ERDAS APOLLO 2010 percentage compared with ERDAS APOLLO 2009
  20. 20. The tested version of EAP compared to EAIM balanced better between the use of disk and CPUtime, which is positively reflected in the concordant behavior of the server. Furtherimprovement was seen in the memory performance EAP, where Memory Footprint applicationsfell by about 15% in comparison with EAIM.Perspective in the future stepsConcerning ERDAS APOLLO it is expected to be converted to Java6 in the future. It will have apositive impact on improving performance in the areas of the memory management,supporting of multiprocessor machines, and optimizing of byte code processing. Transition toJava6 64bit will also break the limit for maximum amount of allocated memory for a singleJAVA process, which will in case you have enough memory, eliminate the effects of JAVA GCand it will further increase the performance and it will harmonized load curves in general.The power and concordant behavior of the server could be influenced positively byimplementation of rules to control the access to disk I/O operations that would preventoverloading the disk I/O queue.With the transition to Java6 64bit we can expect performance increases by about10% -20% and it will harmonize performance excessively compared to the existingset-up.

×