Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Axis2 client memory leak
Outline• Issue & diagnosis• Postmortem from technique point  – Axis2 1.4/1.4.1/1.5.6 client  – Implement & design issue
IssueCServer exhaust 8G memory -> memory leak
diagnosis      jmap –histo               pidnum     #instances         #bytes class name----------------------------------...
diagnosis jmap –dump:file=cdump pid• We got a 2.39G memory heap dump  – Very fast, less than 30 seconds (because 8G    mem...
diagnosisAxisConfiguration retain 540.9m heap
diagnosisHashtable allEndPoints holds more than 2k axis2 endpoint instances
diagnosisAll of the endpoint instances are about FavoriteService1) Confirm in the code, only FavoriteService related did n...
Axis2 1.4 memory leak• Reproducefor( int i=0;i<count; i++) {    VersionStub stub = new VersionStub(configContext,null);   ...
Axis2 1.4 memory leak • Informal Model (OO design)AxisConfigurationMap allServices = new Hashtable();Map allEndpoints = ne...
Axis2 1.4 memory leak• Cause : Stub Initiation, add into AxisConfiguration   //Init, Add   allServices.put(serviceName, ax...
Axis2 1.4 memory leak• Cause : Stub Finalize  //Stub  protected void finalize() throws Throwable {          super.finalize...
Axis2 1.4 memory leak• Cause : Client Finalize  //ServiceClient  protected void finalize() throws Throwable {          sup...
Axis2 1.4 memory leak• Cause  //Init, Add  allServices.put(serviceName, axisService);  ..  allEndpoints.put(serviceName + ...
Axis2 1.4.1 fix• Fix the bug AXIS2-3870//AsixConiguration.removeServiceGroup//removes the endpoints to this serviceString ...
Axis2 1.4.1 memory leak• Reproducefor( int i=0;i<count; i++) {    VersionStub stub = new VersionStub(configContext,null); ...
Axis2 1.4.1 memory leak• Cause : Stub Finalize (no change)  //Stub  protected void finalize() throws Throwable {          ...
Axis2 1.4.1 memory leak• Cause: Client Finalize (no change)  //ServiceClient  protected void finalize() throws Throwable {...
Axis2 1.4.1 memory leak• Cause: Client Finalize (no change)//AsixConigurationpublic AxisServiceGroup removeServiceGroup(St...
Axis2 1.4.1 memory leak• Cause  – Stub cleanup and ServiceClient cleanup have    dependency  – Can not call in below order...
Axis2 1.5(.6) fix• Fix the bug AXIS2-4007 AXIS2-4163  //Stub  protected void finalize() throws Throwable {          super....
Implement issue• Forget to cleanup  – Container object: add only, no remove  – Resource object: apply only, no return/clos...
Design Issue• AxisConfiguration   is a global shared object  – Usually only 1 instance even in client side.• Purpose for p...
Design Issue• Message Dispatch in server side// RequestURIBasedServiceDispatcherpublic AxisService findService(MessageCont...
Design Issue• Service is ‘singleton’ in server side   // AxisConfiguration   public AxisService getService(String name) th...
Design Issue• Client side  – (Usually) New instance every method invocation  – No need for message routing     • Why still...
Upcoming SlideShare
Loading in …5
×

Axis2 client memory leak

9,029 views

Published on

Published in: Technology, Education
  • Be the first to comment

Axis2 client memory leak

  1. 1. Axis2 client memory leak
  2. 2. Outline• Issue & diagnosis• Postmortem from technique point – Axis2 1.4/1.4.1/1.5.6 client – Implement & design issue
  3. 3. IssueCServer exhaust 8G memory -> memory leak
  4. 4. diagnosis jmap –histo pidnum #instances #bytes class name---------------------------------------------- 1: 1001744 246906600 [C //char array 2: 2855952 137085696 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment 3: 282338 115692192 [I // int array 4: 677362 102508648 [Ljava.util.HashMap$Entry; 5: 192117 99164392 [B // byte array 6: 2856122 91732120 [Ledu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry; 7: 2855952 91390464 edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync 8: 1337498 53499920 java.lang.String 9: 433634 43130120 [Ljava.lang.Object; 10: 597908 38266112 java.util.HashMap From the histogram, the memory leak is caused by Axis2. 1) The main change in this release is using web service client (axis2 1.4.1) 2) char[] / edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap 3) Google ‘axis2 1.4.1 concurrentHashMap memory leak‘
  5. 5. diagnosis jmap –dump:file=cdump pid• We got a 2.39G memory heap dump – Very fast, less than 30 seconds (because 8G memory in production)• Problem : too big to open in dev box – Need enough physical memory – (May) need 64 bit OS – Try Jhat/Jprofile/YJP/Jmat eclipse plugin (windows/2G) Open by Jmat standalone application
  6. 6. diagnosisAxisConfiguration retain 540.9m heap
  7. 7. diagnosisHashtable allEndPoints holds more than 2k axis2 endpoint instances
  8. 8. diagnosisAll of the endpoint instances are about FavoriteService1) Confirm in the code, only FavoriteService related did not call cleanup
  9. 9. Axis2 1.4 memory leak• Reproducefor( int i=0;i<count; i++) { VersionStub stub = new VersionStub(configContext,null); GetVersion request = new GetVersion(); stub.getVersion(request);}
  10. 10. Axis2 1.4 memory leak • Informal Model (OO design)AxisConfigurationMap allServices = new Hashtable();Map allEndpoints = new Hashtable();
  11. 11. Axis2 1.4 memory leak• Cause : Stub Initiation, add into AxisConfiguration //Init, Add allServices.put(serviceName, axisService); .. allEndpoints.put(serviceName + "." + endpointName, axisService);
  12. 12. Axis2 1.4 memory leak• Cause : Stub Finalize //Stub protected void finalize() throws Throwable { super.finalize(); cleanup(); } public void cleanup() throws AxisFault { _service.getAxisConfiguration().removeService(_service.getName()); } //AsixConiguration public synchronized void removeService(String name) throws AxisFault { AxisService service = (AxisService) allServices.remove(name); if (service != null) { AxisServiceGroup serviceGroup = service.getAxisServiceGroup(); serviceGroup.removeService(name); log.debug(Messages.getMessage("serviceremoved", name)); } }
  13. 13. Axis2 1.4 memory leak• Cause : Client Finalize //ServiceClient protected void finalize() throws Throwable { super.finalize(); cleanup(); } public void cleanup() throws AxisFault { … axisConfiguration.removeServiceGroup(serviceGroupName); … } //AsixConiguration public AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault { … Iterator services = axisServiceGroup.getServices(); while (services.hasNext()) { AxisService axisService = (AxisService) services.next(); allServices.remove(axisService.getName()); … } …
  14. 14. Axis2 1.4 memory leak• Cause //Init, Add allServices.put(serviceName, axisService); .. allEndpoints.put(serviceName + "." + endpointName, axisService); //Cleanup(Finalize), Remove allServices.put(serviceName, axisService);• Memory Leak : allEndpoints
  15. 15. Axis2 1.4.1 fix• Fix the bug AXIS2-3870//AsixConiguration.removeServiceGroup//removes the endpoints to this serviceString serviceName = axisService.getName();String key = null;for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){ key = serviceName + "." + (String)iter.next(); this.allEndpoints.remove(key);}
  16. 16. Axis2 1.4.1 memory leak• Reproducefor( int i=0;i<count; i++) { VersionStub stub = new VersionStub(configContext,null); GetVersion request = new GetVersion(); stub.getVersion(request); stub.cleanup();} Programmer: 1.4 has memory leak issue, so call cleanup
  17. 17. Axis2 1.4.1 memory leak• Cause : Stub Finalize (no change) //Stub protected void finalize() throws Throwable { super.finalize(); cleanup(); } public void cleanup() throws AxisFault { _service.getAxisConfiguration().removeService(_service.getName()); } //AsixConiguration public synchronized void removeService(String name) throws AxisFault { AxisService service = (AxisService) allServices.remove(name); if (service != null) { AxisServiceGroup serviceGroup = service.getAxisServiceGroup(); serviceGroup.removeService(name); log.debug(Messages.getMessage("serviceremoved", name)); } }
  18. 18. Axis2 1.4.1 memory leak• Cause: Client Finalize (no change) //ServiceClient protected void finalize() throws Throwable { super.finalize(); cleanup(); } public void cleanup() throws AxisFault { … axisConfiguration.removeServiceGroup(serviceGroupName); … }
  19. 19. Axis2 1.4.1 memory leak• Cause: Client Finalize (no change)//AsixConigurationpublic AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault { … Iterator services = axisServiceGroup.getServices(); while (services.hasNext()) { AxisService axisService = (AxisService) services.next(); allServices.remove(axisService.getName()); … for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter. key = serviceName + "." + (String)iter.next(); this.allEndpoints.remove(key); } … } Servce are already removed from ServiceGroup in Stub cleanup …} The while loop would never enter
  20. 20. Axis2 1.4.1 memory leak• Cause – Stub cleanup and ServiceClient cleanup have dependency – Can not call in below order Stub.cleanup ServiceClient.cleanup• Two are two memory leak bugs in 1.4, 1.4.1 only fix 1 bug
  21. 21. Axis2 1.5(.6) fix• Fix the bug AXIS2-4007 AXIS2-4163 //Stub protected void finalize() throws Throwable { super.finalize(); cleanup(); } public void cleanup() throws AxisFault { // _service.getAxisConfiguration().removeService(_service.getName()); _serviceClient.cleanup(); }
  22. 22. Implement issue• Forget to cleanup – Container object: add only, no remove – Resource object: apply only, no return/close• Cleanup dependency – One object cleanup depend on other objects – Two object cleanup/two method has order dependency
  23. 23. Design Issue• AxisConfiguration is a global shared object – Usually only 1 instance even in client side.• Purpose for put service/endpoint map in this global object AxisConfiguration ?
  24. 24. Design Issue• Message Dispatch in server side// RequestURIBasedServiceDispatcherpublic AxisService findService(MessageContext messageContext) throws AxisFault { AxisConfiguration registry = configurationContext.getAxisConfiguration(); AxisService axisService = registry.getService(values[0]); // If the axisService is not null we get the binding that the request came to add // add it as a property to the messageContext if (axisService != null) { Map endpoints = axisService.getEndpoints(); if (endpoints != null) { if (endpoints.size() == 1) { messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME, endpoints.get(axisService.getEndpointName())); } else { String endpointName = values[0].substring(values[0].indexOf(".") + 1); messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME, endpoints.get(endpointName)); } } } return axisService;}
  25. 25. Design Issue• Service is ‘singleton’ in server side // AxisConfiguration public AxisService getService(String name) throws AxisFault { AxisService axisService = (AxisService) allServices.get(name); if (axisService != null) { if (axisService.isActive()) { return axisService; } else { throw new AxisFault(Messages .getMessage("serviceinactive", name)); } } else { axisService = (AxisService) allEndpoints.get(name); if (axisService != null) { if (axisService.isActive()) { return axisService; } else { throw new AxisFault(Messages .getMessage("serviceinactive", name)); } } } return null; }
  26. 26. Design Issue• Client side – (Usually) New instance every method invocation – No need for message routing • Why still register in Axi2Configuration ? axisConfig = configContext.getAxisConfiguration(); if (axisService == null) { axisService = createAnonymousService(); } this.axisService = axisService; if (axisConfig.getService(axisService.getName()) == null) { axisService.setClientSide(true); axisConfig.addService(axisService); } else { … } …

×