Building a high
performance directory
server in Java
Lessons learned and tips
from the OpenDS project.
Ludovic Poitou
Matt...
AGENDA

> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclus...
AGENDA

> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclus...
The OpenDS Project




> Released in Open Source in July 2006
  – CDDL
   –   Source code at https://opends.dev.java.net/
...
What is it ?
> OpenDS is effectively a Java based Server supporting the LDAPv3 protocol and
  services
  – Objet Oriented,...
What is it for ?
> Generic object oriented data store
> White pages and Email Address Book
> Mostly the data store for Ide...
Who for ?
> Telecom service providers, financial institutions use LDAP directories for
  customer related services (Portal...
OpenDS 2.2
> Released in December 2009
> LDAPv3 directory server fully standard compliant
   –   Supports many LDAP standa...
Performance characteristics
> As for most servers, scalability is extremely important
  – Up to hundreds million of entrie...
Searchrate on Sun x4170 box
Modrate on a Sun x4170 box
AGENDA

> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclus...
How to reach those results ?
> 2 main aspects
  – Architecture and code
   –   Run-time : JVM and Garbage Collector optimi...
Architecture Overview
Patterns
> Use of Asynchronous I/O
  – Exception for write disk transactions
> Use of Immutable Objects
  – Intrinsic thre...
Patterns
> Producers / Consumers
  – Queues
  –   Thread Pool
  –   Monitors
> Strategies
  – Queue Strategies : Concurren...
Anti-Patterns
> String concatenation
  – Make sure to use a StringBuilder
   –   Compiler now optimize simple “Aaa” + “Bbb...
Java Collections
> Vector and Hashtable are synchronized for all methods
  – Pay the price even if not necessary
> Some Ja...
Critical Sections
> Try to minimise the code and time spent in the critical sections
> But the throughput is limited by th...
Caching data
> Using Caches reduce the disk access thus should provide better performances
> But cache eviction add pressu...
Server monitoring
> Getting statistics for a server is mandatory
> Beware of contention
   –   Stats are updated frequentl...
AGENDA

> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with tuning Sun JVM
> ...
Performance Tuning

> When dealing with performances, you should consider the whole system
  – Java VM
   –   OS
   –   Ha...
JVM Tuning for OpenDS
> Super Size The Heap !
  – We use 32GB Heaps, sometime
     up to 96GB
  – 2GB for the New Generati...
Some interesting JVM Options
> -XX:CMSInitiatingOccupancyFraction=70
  – Define the amount of occupancy in Old Gen before ...
The Garbage First (G1) GC
> Introduced in the Java HotSpot VM in JDK 7.
> An experimental version of G1 has also been rele...
G1 Characteristics
> Future CMS Replacement
  – Server “Style” Garbage Collector
   –   Parallel, Concurrent
   –   Genera...
JVM Options With G1
> -XX:+UnlockExperimentalVMOptions
  -XX:+UseG1GC
> PauseTime (Hints, Goal with no promise, otherwise ...
OpenDS et G1
> Goal: Avoid any Full GC, best control of pauses
> Collaboration between the HotSpot and the OpenDS teams
  ...
OpenDS G1 and Searches




                         30
AGENDA

> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclus...
Summary
> OpenDS
  – A open source LDAP directory server, 100% pure Java
   –   Easy to install and use
   –   Designed fo...
The Art of GC Tuning




           JavaOne Presentation: GC Tuning in HotSpot JVM
      http://developers.sun.com/learnin...
Now...
> Give OpenDS a try
  – http://www.opends.org
> Join our community:
  – Join/Login sur Java.net
   –   http://opend...
Ludovic Poitou     blogs.sun.com/Ludo
Sun Microsystems   Ludovic.Poitou@sun.com


Matthew Swift
Sun Microsystems   Matthew...
Upcoming SlideShare
Loading in...5
×

OpenDS_Jazoon2010

907

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
907
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

OpenDS_Jazoon2010

  1. 1. Building a high performance directory server in Java Lessons learned and tips from the OpenDS project. Ludovic Poitou Matthew Smith Sun Microsystems #118
  2. 2. AGENDA > Introduction to the OpenDS project > Architecture, Design Patterns and Tips > Experiences with Sun JVM > Conclusion 2
  3. 3. AGENDA > Introduction to the OpenDS project > Architecture, Design Patterns and Tips > Experiences with Sun JVM > Conclusion 3
  4. 4. The OpenDS Project > Released in Open Source in July 2006 – CDDL – Source code at https://opends.dev.java.net/ > Sponsored by Sun Microsystems > Written in Java by LDAP experts
  5. 5. What is it ? > OpenDS is effectively a Java based Server supporting the LDAPv3 protocol and services – Objet Oriented, Hierarchical Data Model – CRUD operations > It comes with its own embedded database – Based on Berkeley DB Java Edition – Not accessible from outside > It has all security, access controls, password management features to safely store the information about Users
  6. 6. What is it for ? > Generic object oriented data store > White pages and Email Address Book > Mostly the data store for Identities – For Authentication and Authorization – For profiles and personalization > The underlying infrastructure in all Enterprises – Leveraged by Web and Mail infrastructure products – Cornerstone of Identity Management products:  Access Management and Federation  Provisioning and De-provisioning tools
  7. 7. Who for ? > Telecom service providers, financial institutions use LDAP directories for customer related services (Portal) – Storing customers identities, phones, services associated – Building highly available services for 10 Millions, up to 200 Millions users > But OpenDS can be used as OS naming service, or for SMB – OpenSolaris, Solaris, Linux... – Coupled with SAMBA, as a Domain controler – Integrated with Kerberos – White Pages... > And being 100% pure Java, OpenDS can be embedded in other Java applications or Web applications – OpenSSO 7
  8. 8. OpenDS 2.2 > Released in December 2009 > LDAPv3 directory server fully standard compliant – Supports many LDAP standard and experimental extensions – Supports Multi Master Replication with 3 different levels of data consistency – Extensive security features > Improved performances, reliability over OpenDS 1.0 > Installs in 6 clicks and less than 3 minutes > Several GUI and CLI to manage, monitor the OpenDS server > Extensive documentation > Localized in 6 different languages
  9. 9. Performance characteristics > As for most servers, scalability is extremely important – Up to hundreds million of entries – Up to thousands connections – Maximize use of CPUs > What is the operations throughput ? > What is the average response time ? The maximum response time ? > Our basic test – 10 M entries, with an average size of 2.6K – 2 servers, with Multi-Master replication between them
  10. 10. Searchrate on Sun x4170 box
  11. 11. Modrate on a Sun x4170 box
  12. 12. AGENDA > Introduction to the OpenDS project > Architecture, Design Patterns and Tips > Experiences with Sun JVM > Conclusion 12
  13. 13. How to reach those results ? > 2 main aspects – Architecture and code – Run-time : JVM and Garbage Collector optimization > There is a strong relationship between code design and memory optimization
  14. 14. Architecture Overview
  15. 15. Patterns > Use of Asynchronous I/O – Exception for write disk transactions > Use of Immutable Objects – Intrinsic thread safety – Avoid need for defensive copies > Use of “Factories” over Constructors – Avoid creating an object – Ease optimization for common cases  Example : Most AttributeDescriptions have 0 options  Example : Attributes generally have 1 value – For immutable objects. 15
  16. 16. Patterns > Producers / Consumers – Queues – Thread Pool – Monitors > Strategies – Queue Strategies : ConcurrentLinkedQueue vs LinkedBlockingQueue 16
  17. 17. Anti-Patterns > String concatenation – Make sure to use a StringBuilder – Compiler now optimize simple “Aaa” + “Bbb” concatenation > Avoid very long methods (thousands of lines of code) > Avoid exposing the concrete representation of an object – Set vs LinkedHashSet. – Not a performance issue, but will require more work when optimizing code for performance later > Try to define only the methods you need. 17
  18. 18. Java Collections > Vector and Hashtable are synchronized for all methods – Pay the price even if not necessary > Some Java collection classes are not synchronized by default – ArrayList, LinkedList replace Vector – HashSet, HashMap replace Hashtable > To synchronize, wrap in a class > With a “static factory” – Collections.synchronizedList(new ArrayList()) > ConcurrentHashMap, for concurrency – But watch when using the iterator 18
  19. 19. Critical Sections > Try to minimise the code and time spent in the critical sections > But the throughput is limited by the time spent in the largest critical section – Example : LinkedBlockingQueue  200 000 operations on x64 processor  20 000 operations on the T2000 processor – We use it for the WorkQueue and Access Logs
  20. 20. Caching data > Using Caches reduce the disk access thus should provide better performances > But cache eviction add pressure to the GC – When modifying entries – When the cache is too small to hold all data > A cache is also a contention point > If you want to cache objects, make sure you cache those that will be reused. > Alternate possibility, use “thread local” cache. – But watch out for the cost (with 1000 threads?)
  21. 21. Server monitoring > Getting statistics for a server is mandatory > Beware of contention – Stats are updated frequently – But seldom read > A strategy could be to keep per thread statistics and collect them on demand – Not yet implemented in OpenDS !
  22. 22. AGENDA > Introduction to the OpenDS project > Architecture, Design Patterns and Tips > Experiences with tuning Sun JVM > Conclusion 22
  23. 23. Performance Tuning > When dealing with performances, you should consider the whole system – Java VM – OS – Hardware : CPU, Memory, Disks, Network... > In our case, we try to avoid disk I/Os – And try to cache as much of the database > We also want deterministic response times – Avoid any Full GC (Stop The World) – Make sure minor GC pauses are as small as possible 23
  24. 24. JVM Tuning for OpenDS > Super Size The Heap ! – We use 32GB Heaps, sometime up to 96GB – 2GB for the New Generation (or ¼ of heap if < 8GB) – -Xms32768M -Xmx32768M -Xmn2048M > Use CMS – -XX:+UseConcMarkSweepGC – -XX:+UseParNewGC > -XX:MaxTenuringThreshold=1 – Avoid copy of objects in New Gen 24
  25. 25. Some interesting JVM Options > -XX:CMSInitiatingOccupancyFraction=70 – Define the amount of occupancy in Old Gen before starting to collect – Larger = better throughput but higher full GC risk > -XX:+UseCompressedOops – For 64bits JVM, less than 32BG of heap – Will be the default in coming Java 6 updates > If running on processor with NUMA architecture – -XX:+UseNUMA > -XX:+AggressiveOpts – Enables aggressive JIT optimizations, not related to GC 25
  26. 26. The Garbage First (G1) GC > Introduced in the Java HotSpot VM in JDK 7. > An experimental version of G1 has also been released since Java SE 6 Update 14. > G1 is the long-term replacement for HotSpot's low-latency Concurrent Mark- Sweep (CMS) GC > Should be officially supported with Java SE 6 Update 21 26
  27. 27. G1 Characteristics > Future CMS Replacement – Server “Style” Garbage Collector – Parallel, Concurrent – Generational – Good Throughput – Compacting – Improved ease-of-use – Predictable (though not hard real-time) > The main stages consist of remembered set (RS) maintenance, concurrent marking, and evacuation pauses. 27
  28. 28. JVM Options With G1 > -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC > PauseTime (Hints, Goal with no promise, otherwise use Java Real Time ) – -XX:MaxGCPauseMillis=50 (target of 50 milliseconds) – -XX:GCPauseIntervalMillis=1000 (target of 1000 msecs) > Generation Size – -XX:+G1YoungGenSize=512m (for a 512 MB young gen) > Parallelism – -XX:+G1ParallelRSetUpdatingEnabled – -XX:+G1ParallelRSetScanningEnabled 28
  29. 29. OpenDS et G1 > Goal: Avoid any Full GC, best control of pauses > Collaboration between the HotSpot and the OpenDS teams – OpenDS is used as a “Large” reference application – Between 10 and 20 enhancements integrated in G1 following the tests – Performance with large heaps improved by a factor of 10 > We're still discovering it – When doing read operations, we see pauses between 10 and 20 ms with 32GB JVM – But we're still seeing Full GC when doing Write operations (More garbage, stresses more the Old Gen) – Hopefully this will be resolved in next builds 29
  30. 30. OpenDS G1 and Searches 30
  31. 31. AGENDA > Introduction to the OpenDS project > Architecture, Design Patterns and Tips > Experiences with Sun JVM > Conclusion 31
  32. 32. Summary > OpenDS – A open source LDAP directory server, 100% pure Java – Easy to install and use – Designed for high performance and high scalability > We saw some patterns and tips used in the OpenDS project > Knowing and understanding the JVM and GC is required to build high performance server – Tuning JVM and GC is an art – Performance engineering is a profession > Who said that Java is slow ?! 32
  33. 33. The Art of GC Tuning JavaOne Presentation: GC Tuning in HotSpot JVM http://developers.sun.com/learning/javaoneonline/j1sessn.jsp?sessn=TS-4887&yr=2009&track=javase 33
  34. 34. Now... > Give OpenDS a try – http://www.opends.org > Join our community: – Join/Login sur Java.net – http://opends.dev.java.net – Request a Role – Subscribe to the mailing lists – IRC: #opends on freenode.net > OpenDS is localized in several languages. It's community based, through online tools. An easy way to participate. 34
  35. 35. Ludovic Poitou blogs.sun.com/Ludo Sun Microsystems Ludovic.Poitou@sun.com Matthew Swift Sun Microsystems Matthew.Swift@sun.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×