Tuning and development with SIP Servlets on Mobicents


Published on

Tuning of JVM options for running Mobicents and Rapid Application Development with CDI Telco Framework and Spring Signaling Framework

Published in: Technology, Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • At first I’m not very fluent in English. So I will look at my notes during the presentation. Please bear with me and feel free to interrupt me and ask any questions. All right, first of all, thanks for your coming this session. Today, I’d like to talk about tuning and development with sip servlet on mobicents. This is my first speech in English. And I participate in JUDCon for the first time too. Please be easy on me Let’s get started.
  • This is agenda. First introduction Second I describe about SIP Application behavior and mobicents sip servlet container. Third How to tune the JavaVM for SIP Servlet Last How to develop SIP Servlet Application with frameworks
  • Introduction I’m mobicents fellow. All you know, mobicents is the most popular OSS community for Telecom. I’ve joined to core team of mobicents since this March. And I’m contributing SSF to mobicents. SSF is for developing SIP Servlet Application with Spring Framework. My company, OKI has developed SIP Servlet and container since 2003 (two thousands and three) (After this, Introduce Jean)
  • Next, The brief description of my company All you would not know OKI, but you may know OKI data. OKI is parent company of OKI data.:-0 Founded, January 1881 (eighteen, eighty one) Major operation is as you can see it.
  • My current works are: Support Japan carriers They are evaluating the oss platform. They are usually very conservative and will be very sensitive and careful for managing and handling error and abnormal cases. But they are very important for platforms. We are analyzing and evaluating their requirements, and how can I feedback them to mobicents. Customize mobicents sip servlet for proprietary. Our customer requires Do not include internal IP address in SIP Header fields. Do not over 256 characters per line in SIP Header. Etceteras Developing customized sip load balancer for Japanese carrier.
  • Let’s get down to business. #Return to the subject. Start to talk about Sip Servlet behavior
  • 1 st , As you know. Some SIP messages will be sent for creating one session. INVITE, 180./INVITE, 200/INVITE, ACK, BYE, 200/BYE and etc…
  • This is most basic b2bua sequence. 細かい質問があったら、 Jean に任せる。 At least, sip specification creates 2 dialogs, 6 transactions, 12 messages for one session. If UDP retransmissions occurred, more messages will be sent and create more objects and discard them
  • In addition, many objects are created.
  • For example, on mobicents sip servlets, This table is instance counts after 10,000 calls are established. These objects are top counts of remained objects related to jain-sip. Almost these objects are SIP header. There are short life time objects, or related to SipSession or SipApplicationSession Related objects will be left on JVM until SipSessions and SipApplicationSession are terminated. Heap will be filled with like these objects.
  • As MSS own behavior, many threads run on MSS. For transport, timer, cluster, http, j2ee container… and so on For Jain-sip and sip-servlets stack, about 100 threads run by default settings. But you can configure and reduce them.
  • SIP with UDP transport has a common problem due to retransmission. In normal case, UDP transport will be used. when sip response sent late over 500ms, retransmission of sip message will coccur. And If jvm was stopped long time by gc, Call failure will be occurred and error handling may not work properly. Or SIP has many timers, and they could not work properly. Retry with http is relatively easier than sip You would have to start over from scratch to initiate SIP session.
  • From these behaviors, the results can be derived as follows. MSS will discard many objects in one session. When retransmissions of UDP messages occurred, Many threads runs at the same time CPU usage go up New GC will be missed Full GC will run If many retransmissions occurred, more objects will discard. GC will be missed more… And fall in a vicious cycle.
  • For these reasons, I will introduce How to tune the JVM to avoid these problems.
  • My plan is Reduce retransmissions of UDP Response time less than 200ms Measure the performance 400call/sec(1440,000BHCA Busy Hour Call Attempts) I want to use multiple cpu effectively Most of server-machine have multiple CPU, and I want to use multiple CPU well with the concurrent GC.
  • This is test sequence that is basic b2bua sequence I described before. Wait 10 seconds after ringing response sent Wait 30 seconds after ACK request sent
  • These are recommended Java VM Options. These options are set, you will reduce the “Stop the World” by GC. UsetTLAB is to use thread-local object allocation These options enable the concurrent mark sweep GC
  • These are other recommended options for mss. When These options is enabled, mobicents sip servlet will work more stable.
  • These are other recommended options These options are for optimization. SIP message parser will create many string objects. I thought these options will be useful for mss.
  • I tried four patterns. And I measured GC, CPU usage, failed call, succeeded calls, retransmission, and response time.
  • First These are almost default gc options for mss Heap memory is set to 6 giga bytes Permanent space size is 256 mega bytes
  • Let see charts. CPU usage has many peaks, but average is 30% to 40% There are over thousands Retransmissions and failed calls “ Stop the world” is occurred. Many responses took more than 200ms.
  • Next, Add the recommended VM options. They enable concurrent gc.
  • These charts are CPU usage is increased, seems to be used more efficiently . Failed call is 0. CPU pause time less than 100ms. Time-consuming response is decreased! Almost happy to me. But many retransmission is still remained.
  • Next I added the CMS tuning options. These options is enabled customized the duty cycle. 100% of time is used for concurrent collector. You will be able to use multiple CPU power more efficiently for GC.
  • These are charts of result. Retransmission is decreased to about 10%. GC pause time is less than 100ms and heap utilization is decreased too. In addition, time-consuming response is decreased. These results is very satisfactory.
  • Next I tried to add more options . These options are for optimization. For more information about these options, please see the Java VM documentation.
  • The result are not any better the previous chart. CPU usage and memory usage is slightly decreased. But retransmissions and time-consuming response is increased.
  • Conclusion Recommended options are useful. In addition, Options to tune the duty cycle are very useful. As mentioned before, multiple CPU can be used effectively. In addition, If you set optimization options instead of duty cycle options, You can see a similar effect. These results may also vary depending on machine spec However, it should be able to get the same effect by tuning the recently server machine Moreover, it is also effective to change the parameter of mss-sip-stack.properties to decrease retransmissions. These retransmissions are caused by udp packet errors. MSS is executing with one thread and one port for udp package transmission. Mobicents will improve this issu in the near future with NIO. まず、最初の VM オプションはかなり有効であることがおわかりいただけたと思います。 さらに duty cycle のチューニングをすることで、大きく性能があがったことがおわかりいただけたでしょう。 また、追加のオプションは少しは役に立ちましたが、 duty cycle のチューニングをしていなかった場合、同程度の性能向上を 行えたこともお伝えしておきます。 今回は VM のオプションを設定することでチューニングを施しました。 しあし、 MSS のオプションによっても性能を向上させることはできます。 再送は UDP の SC_RCVBUF の設定によっても減らすことが可能です。 再送が発生している時、 netstat で取得した情報から、 UDP のパケットエラーが発生していることに気づきました。 SC_RCVBUF の値を倍の値を 131072 にすることで、パケットエラーを減らすことができました。 しかし、0にすることはできませんでした。 この問題は、ある程度のマシンスペックがないと発生しないことが我々の試験でわかっています。 低スペックのマシンでは、それ以前に処理が間に合わずにエラーが発生するためです。 このため、この問題を解決するための方法について考察したことがあります。 MSS は UDP のパケット通信をひとつのポートおよびひとつのスレッドで処理しています。 Rport に対応させ、 UAC として動作するときにデフォルトの受信ポートと違うポートを使用してメッセージを送受信することで、 さらなる性能向上が見込めます。 実際に、 NIO を使用し、 UAC として動作する際にポートを開くようにトランスポートを改造してみたところ、パケットエラーを0にすることに成功しました。 ただ、試験用のためコミュニティに寄贈できるレベルのものを作成することができませんでした。 この提案はすでに行っており、近い将来に MSS で実装されるものと考えています。
  • Let me show you developing sip servlet with frameworks.
  • Mobicents has 2 sip servlet frameworks. First, My project, ssf This framework based on spring framework. Second CTF This framework based on cdi that is using standard api. These frameworks have same concept Simplify SipServlets development. The concept of sip servlet, web developer is also being able to facilitate development of sip. But all you know, sip servlet like the early days http servlet development, you have low-level api for sip development, And development is not easy. We want to accelerate application development.
  • First, I talk about ssf.
  • I selected to develop the framework based on spring, about 5 years ago. This because. Standard DI Container at the time Familiar to many Java Developers Spring has customizable context And SSF provides customized Context for SIP Servlet Many functionality modules are working on spring framework So you can create the converged application with Spring Framework modules
  • This is a sample sip servlet with POJO. @Component annotation Spring will scan this pojo by this annotation. @Autowired annottion Spring will inject objects by this annotation @SipServletRequestMapping annotation SSF call this method by this annotation, when Sip request is received. This is very simple.
  • Of course, you can use AOP function. Let’s add Call-Blocking function to this service.
  • First create a new pojo for call-blocking. This logic is very simple. Determine whether you want to block requests And send response with specified error status.
  • Second Create aspect class for advice This is a simple aspect class with annotations. If proxyhandler.handlerequest is called, this aspect class is called by spring.
  • And setup the configrations for AOP. These configurations are basic settings for spring. Enable the aspectj fucntions And scan classes annotated with @AspectJ annotation.
  • Next Introduce CTF by Jean
  • Contact information Naoki, it’s me. SSF lead Jean MSS Lead Georges CDI Telco framework lead Vladimir Mobicents core developere and high availability support
  • On March 11, 2011, the largest earthquake occurred and tremendous tsunami hit the eastern part of Japan. Immediately after this happened , many countries announced their support. We are in a difficult situation, many disaster victims have no place to live. But we will try to stand up and go on. Many many thanks from Japan Thanks for listening.
  • Tuning and development with SIP Servlets on Mobicents

    1. 2. Tuning and development with SIP Servlet on Mobicents Naoki Nishihara OKI Electric Industry Co.,Ltd Jean Deruelle Mobicents Sip Servlets Lead
    2. 3. Agenda <ul><li>Introduction </li></ul><ul><li>SIP Application behavior </li></ul><ul><li>How to tune the JavaVM for SIP Servlet </li></ul><ul><li>How to develop SIP Servlet Application with Frameworks </li></ul>
    3. 4. Introduction <ul><ul><li>Mobicents fellow </li></ul></ul><ul><ul><ul><li>Member of mobicents core team from March 2011 </li></ul></ul></ul><ul><ul><li>Leading mobicents SSF project </li></ul></ul><ul><ul><ul><li>For developing SIP Servlet Application with Spring Framework </li></ul></ul></ul><ul><ul><li>OKI has been involved SIP Servlet development since 2003 </li></ul></ul>
    4. 5. About OKI <ul><li>Founded </li></ul><ul><ul><li>January 1881 </li></ul></ul><ul><li>Major Operation </li></ul><ul><ul><li>Manufacturing and sales of products, technologies, software and solutions for telecommunications systems and information systems </li></ul></ul>
    5. 6. Current works <ul><li>Support Japanese carrier (They are evaluating the OSS platform) </li></ul><ul><li>Customize MSS for proprietary </li></ul><ul><ul><li>ex) NOT include internal IP Address in SIP header fields </li></ul></ul><ul><li>Developing customized SIP Load balancer </li></ul>
    6. 7. SIP Servlet Behavior
    7. 8. SIP Servlet Behavior (1) <ul><li>To establish one session, some messages are sent and received. </li></ul>
    8. 9. Basic B2BUA sequence UAC UAS MSS INVITE INVITE 180/INVITE 180/INVITE 200/INVITE 200/INVITE ACK ACK BYE BYE Established Session 200/BYE 200/BYE <ul><li>2 dialogs </li></ul><ul><li>6 transactions </li></ul><ul><li>12 messages </li></ul>
    9. 10. SIP Servlet Behavior (2) <ul><li>Many object will be generated in one sequence. </li></ul>
    10. 11. Created instances After 10,000 calls (NOT TERMINATED SESSION) NameValueList NameValue HostPort Host DuplicateNameValueList MultiValueMapImpl SipURI Authority AddressImpl gov.nist.core. gov.nist.javax.sip. :810,041 :330,015 :310,036 :310,036 :290,049 :290,049 :240,006 :240,006 :210,003 Instance counts
    11. 12. Mobicents SIP Servlet own behavior <ul><li>JAIN-SIP, SIP Servlet, JBoss(J2EE)… </li></ul>About 200 Threads run on MSS (for transport, timer, cluster…)
    12. 13. Other problems related to SIP <ul><li>Retransmissions will occur, if response was sent late over 500ms with UDP transport </li></ul><ul><ul><li>In normal case, UDP transport will be used. </li></ul></ul><ul><li>If JVM was stopped long time by GC </li></ul><ul><ul><li>Call failure will be occurred and error handling may not work properly. </li></ul></ul><ul><ul><li>SIP has many timers, they could not work properly. </li></ul></ul><ul><li>It’s relatively easy to retry with HTTP, but you would have to start over from scratch to initiate session with SIP </li></ul>
    13. 14. Conclusion <ul><li>MSS will discard many objects in one session. </li></ul><ul><li>When retransmissions of UDP messages occurred, </li></ul><ul><ul><li>Many threads runs at the same time </li></ul></ul><ul><ul><li>Many objects are discarded </li></ul></ul><ul><ul><li>CPU usage go up </li></ul></ul><ul><ul><li>GC will be missed </li></ul></ul><ul><ul><li>Full GC will run </li></ul></ul>Fall in a vicious cycle
    14. 15. How To Tune the Java VM
    15. 16. Plan <ul><li>Reduce retransmissions of UDP </li></ul><ul><ul><li>Response time less than 200ms </li></ul></ul><ul><ul><li>Measure the performance on tuned JVM </li></ul></ul><ul><ul><ul><li>400call/sec(1440,000BHCA) </li></ul></ul></ul><ul><li>Effective use of multiple CPU </li></ul><ul><ul><li>Most of server-machine have multiple CPU </li></ul></ul><ul><li>Reduce pause times by “Stop-The-World” </li></ul>
    16. 17. Test Sequence UAC UAS Mobicnets INVITE INVITE 180/INVITE 180/INVITE 200/INVITE 200/INVITE ACK ACK BYE BYE Established Session 200/BYE 200/BYE 10sec 30sec About 16,000 sessions will remain on JVM
    17. 18. Recommended Sun Java VM Options <ul><li>-server </li></ul><ul><ul><li>-XX:+UseTLAB is enabled. </li></ul></ul><ul><li>-XX:+UseConcMarkSweepGC </li></ul><ul><ul><li>CMS GC reduce the “Stop the World”. </li></ul></ul><ul><li>-XX:+CMSIncrementalMode </li></ul><ul><ul><li>Enable the incremental mode. You should tune the below options </li></ul></ul><ul><ul><ul><li>-XX:CMSIncrementalDutyCycle=<N> </li></ul></ul></ul><ul><ul><ul><li>-XX:CMSIncrementalDutyCycleMin=<N> </li></ul></ul></ul>
    18. 19. Other performance options(1) <ul><li>-XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=128 </li></ul><ul><ul><li>Object that related to SipSessions will NOT be collected in NewGC. This option will make the full NewSize available to every NewGC cycle. </li></ul></ul><ul><li>-XX:+UseParNewGC </li></ul><ul><ul><li>Enable a parallel young generation GC. (for multiple cpu machine) </li></ul></ul><ul><li>-XX:CMSInitiatingOccupancyFraction=75 </li></ul><ul><ul><li>Set the level at which the collection is started </li></ul></ul><ul><li>-XX:+CMSParallelRemarkEnabled </li></ul><ul><ul><li>Reduce remark pause with multi threads </li></ul></ul>
    19. 20. Other performance options (2) <ul><li>-XX:+UseStringCache </li></ul><ul><ul><li>Enables caching of commonly allocated strings </li></ul></ul><ul><li>-XX:+OptimizeStringConcat </li></ul><ul><ul><li>Optimize String concatenation operations where possible </li></ul></ul><ul><li>-XX:+UseCompressedStrings </li></ul><ul><ul><li>Use a byte[] for Strings which can be represented as pure ascii </li></ul></ul><ul><li>-XX:+UseCompressedOops </li></ul><ul><ul><li>Enables the use of compressed pointers for optimized 64-bit performance </li></ul></ul>
    20. 21. Data and results <ul><li>Default GC </li></ul><ul><li>CMS GC without tuning the duty cycle </li></ul><ul><li>CMS GC with tuning the duty cycle </li></ul><ul><li>Add parallel NewGC and performance options </li></ul>Analyze: GC, CPU usage, failed call, succeeded call, Retransmissions, Average response time,…
    21. 22. GC options (1) -Xms6g -Xmx6g -XX:PermSize=256m -XX:MaxPermSize=256m
    22. 23. Default GC Over thousands failed calls And retransmissions “ Stop The World” about 8 seconds Thousands responses took more than 200ms
    23. 24. GC options (2) -Xms6g -Xmx6g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:UseConcMarkSweepGC -XX:+CMSIncrementalMode
    24. 25. CMS GC without tuning the duty cycle 0 failed call GC pause time less than 100ms CPU usage increased Decreased time-consuming response
    25. 26. GC options (3) -XX:UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:-CMSIncrementalPacing -XX:CMSIncrementalDutyCycle=100 -XX:CMSIncrementalDutyCycleMin=100
    26. 27. CMS GC with tuning the duty cycle CPU usage decreased Retransmissions decreased GC pause time less than 100ms Decreased time-consuming response
    27. 28. GC options (4) -XX:-CMSIncrementalPacing -XX:CMSIncrementalDutyCycle=100 -XX:CMSIncrementalDutyCycleMin=100 -XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseCompressedStrings -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=128 -XX:+UseParNewGC -XX:+UseCompressedOops -XX:CMSInitiatingOccupancyFraction=75 -XX:+CMSParallelRemarkEnabled
    28. 29. Add parallel NewGC and performance options CPU usage decreased Memory usage decreased Retransmissions increased Increased time-consuming response
    29. 30. Results <ul><li>Recommended VM options are useful. </li></ul><ul><ul><li>Reduce pause times </li></ul></ul><ul><ul><li>No failed calls </li></ul></ul><ul><ul><li>Reduce retransmissions </li></ul></ul><ul><li>Tuning options of the duty cycle are very useful </li></ul><ul><li>Other performance options are slightly useful </li></ul>
    30. 31. Developing SIP Servlet with frameworks
    31. 32. SIP Servlet Frameworks <ul><li>Mobicents has 2 SIP Servlet Frameworks </li></ul><ul><li>SSF (Spring Signaling Framework) </li></ul><ul><ul><li>Based on Spring Framework </li></ul></ul><ul><li>CTF(CDI-Telco-Framework) </li></ul><ul><ul><li>CDI Based framework </li></ul></ul><ul><li>These frameworks have same concept… </li></ul><ul><li>Simplify SipServlets development </li></ul><ul><li>We want to merge these framework’s functions and support many developers! </li></ul>
    32. 33. Spring Signaling Framework SSF
    33. 34. How to develop SIP Servlet application with SSF
    34. 35. Why using Spring Framework? <ul><li>Standard DI Container </li></ul><ul><ul><li>Familiar to many Java Developers </li></ul></ul><ul><li>Customizable Context </li></ul><ul><ul><li>SSF provide customized Context for SIP Servlet </li></ul></ul><ul><li>Many functionality modules are working on Spring Framework. </li></ul><ul><ul><li>You can create the converged application with Spring Framework modules. </li></ul></ul>
    35. 36. Create SIP Servlet with POJO @Component public class ProxyHandler { @Autowired ProxyBean proxyBean; @Autowired CheckRequireBean checkRequire; @SipServletRequestMapping(methods = { &quot;INVITE&quot;, &quot;UPDATE&quot;, &quot;MESSAGE&quot;, &quot;PUBLISH&quot; }) public void handleRequest(SipServletRequest req) throws Exception { // start checkRequire.handleRequest(req); if (req.isCommitted()) { return ; } proxyBean.startProxy(req); // end } … . Component Scan Autowired Original annotation Call bean’s method
    36. 37. Easy to add functions <ul><li>Of course, you can use AOP function </li></ul><ul><li>Try to add the Call-Blocking function to the proxy service.  </li></ul>
    37. 38. Add Call-Blocking function (1) <ul><li>Create new POJO for Call-Blocking </li></ul>@Component public class CallBlockingBean { public void handleRequest(SipServletRequest req) throws IOException { if (isBlocked(req)) { SipServletResponse res = req.createResponse(SipServletResponse. SC_SERVICE_UNAVAILABLE ); res.send(); return ; } } … .
    38. 39. Add Call-Blocking function(2) <ul><li>Create Aspect class for Advice </li></ul>@Aspect public class ProxyAround { @Autowired CallBlockingBean callBlocking; @Around(“execution(ProxyHandler.handleRequest(..)) && args(req)”) public void handleRequest(ProceedingJointPoint pjp, SipServletRequest req) throws IOException { callBlocking.handleRequest(req); pjp.proceed(new Object[]{req}); } … .
    39. 40. Add Call-Blocking function (3) <ul><li>Add configurations for AOP </li></ul><aop:aspectj-autoproxy proxy-target-class= &quot;true&quot; /> <context:component-scan base-package= &quot;org.mobicents.ssf.examples.sip.beans&quot; > <context:include-filter type= &quot;aspectj&quot; expression= &quot;org..ProxyAround*&quot; /> </context:component-scan> <context:component-scan base-package= &quot;org.mobicents.ssf.examples.sip.sipserver“/ > Just Do It! 
    40. 41. CDI-Telco-Framework CTF
    41. 42. CDI JSR-299 <ul><li>CDI is the Java standard for dependency injection and contextual lifecycle management, led by Gavin King for Red Hat , Inc. and is a Java Community Process (JCP) specification that integrates cleanly with the Java EE platform. Any Java EE 6-compliant application server provides support for JSR-299 (even the web profile). </li></ul><ul><li>Loose coupling with strong typing </li></ul><ul><li>Well-defined lifecycle for stateful objects bound to lifecycle contexts </li></ul><ul><li>Support for Java EE modularity and the Java EE component architecture </li></ul>
    42. 43. CDI-Telco-Framework <ul><li>Mobicents brings the power and productivity benefits of CDI into the Mobicents Sip Servlets platform providing dependency injection and contextual lifecycle management for converged HTTP/SIP applications. </li></ul>
    43. 44. CDI-Telco-Framework <ul><li>Mission Statement </li></ul><ul><ul><li>simplify SipServlets development by introducing a clean programming model </li></ul></ul><ul><ul><li>ease of development by making available SIP utilities out of the box </li></ul></ul><ul><ul><li>providing dependency injection and contextual lifecycle management to the SipServlets. </li></ul></ul>
    44. 45. CDI-Telco-Framework <ul><li>public class SipRegistarClient { </li></ul><ul><li>.... </li></ul><ul><li>@Inject </li></ul><ul><li>SipFactory sipFactory; </li></ul><ul><li>protected void doRegister(@Observes @Register SipServletRequest req) throws ServletException, IOException { </li></ul><ul><li>         .... register user here ... </li></ul><ul><li>} </li></ul>
    45. 46. Contact <ul><li>Naoki Nishihara </li></ul><ul><ul><li>[email_address] </li></ul></ul><ul><li>Jean Deruelle </li></ul><ul><ul><li>[email_address] </li></ul></ul><ul><li>Georges </li></ul><ul><ul><li>[email_address] </li></ul></ul><ul><li>Vladimir Ralev </li></ul><ul><ul><li>[email_address] </li></ul></ul>
    46. 47. Gratitude
    47. 48. Appendix
    48. 49. Configuration <ul><li>mss-1.6.0-snapshot-jboss-jdk6-5.1.0GA-1103310643 </li></ul><ul><ul><li>With Simple B2BUA application </li></ul></ul><ul><li>SIPp as UAC & UAS </li></ul><ul><li>Jdk1.6.0_24 </li></ul><ul><li>RHEL5 2.6.18-164.el5 </li></ul>
    49. 50. Target server machine <ul><li>HP PROLIANT DL360 G6(504634-291) </li></ul><ul><li>Intel(R) Xeon(R) CPU E5540 @ 2.53GHz stepping 05 </li></ul><ul><ul><li>4 core </li></ul></ul><ul><li>Memory: 32G </li></ul>
    50. 51. Default GC
    51. 52. Long Load
    52. 53. Over Load