This document summarizes a presentation about building a high performance directory server in Java using the OpenDS project. It discusses the OpenDS architecture, design patterns used like asynchronous I/O and immutable objects, experiences tuning the Sun JVM like using large heap sizes and the CMS garbage collector. Performance tests showed search and modification rates of over 200,000 and 20,000 operations per second respectively on a Sun x4170 server. The presentation concludes by encouraging people to try OpenDS and get involved in the open source community around it.

1. Building a high
performance directory
server in Java
Lessons learned and tips
from the OpenDS project.
Ludovic Poitou
Matthew Smith
Sun Microsystems
#118

2. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
2

3. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
3

4. The OpenDS Project
> Released in Open Source in July 2006
– CDDL
– Source code at https://opends.dev.java.net/
> Sponsored by Sun Microsystems
> Written in Java by LDAP experts

5. What is it ?
> OpenDS is effectively a Java based Server supporting the LDAPv3 protocol and
services
– Objet Oriented, Hierarchical Data Model
– CRUD operations
> It comes with its own embedded database
– Based on Berkeley DB Java Edition
– Not accessible from outside
> It has all security, access controls, password management features to safely
store the information about Users

6. What is it for ?
> Generic object oriented data store
> White pages and Email Address Book
> Mostly the data store for Identities
– For Authentication and Authorization
– For profiles and personalization
> The underlying infrastructure in all Enterprises
– Leveraged by Web and Mail infrastructure products
– Cornerstone of Identity Management products:
 Access Management and Federation
 Provisioning and De-provisioning tools

7. Who for ?
> Telecom service providers, financial institutions use LDAP directories for
customer related services (Portal)
– Storing customers identities, phones, services associated
– Building highly available services for 10 Millions, up to 200 Millions users
> But OpenDS can be used as OS naming service, or for SMB
– OpenSolaris, Solaris, Linux...
– Coupled with SAMBA, as a Domain controler
– Integrated with Kerberos
– White Pages...
> And being 100% pure Java, OpenDS can be embedded in other Java
applications or Web applications
– OpenSSO
7

8. OpenDS 2.2
> Released in December 2009
> LDAPv3 directory server fully standard compliant
– Supports many LDAP standard and experimental extensions
– Supports Multi Master Replication with 3 different levels of data consistency
– Extensive security features
> Improved performances, reliability over OpenDS 1.0
> Installs in 6 clicks and less than 3 minutes
> Several GUI and CLI to manage, monitor the OpenDS server
> Extensive documentation
> Localized in 6 different languages

9. Performance characteristics
> As for most servers, scalability is extremely important
– Up to hundreds million of entries
– Up to thousands connections
– Maximize use of CPUs
> What is the operations throughput ?
> What is the average response time ? The maximum response time ?
> Our basic test
– 10 M entries, with an average size of 2.6K
– 2 servers, with Multi-Master replication between them

10. Searchrate on Sun x4170 box

11. Modrate on a Sun x4170 box

12. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
12

13. How to reach those results ?
> 2 main aspects
– Architecture and code
– Run-time : JVM and Garbage Collector optimization
> There is a strong relationship between code design and memory optimization

14. Architecture Overview

15. Patterns
> Use of Asynchronous I/O
– Exception for write disk transactions
> Use of Immutable Objects
– Intrinsic thread safety
– Avoid need for defensive copies
> Use of “Factories” over Constructors
– Avoid creating an object
– Ease optimization for common cases
 Example : Most AttributeDescriptions have 0 options
 Example : Attributes generally have 1 value
– For immutable objects.
15

16. Patterns
> Producers / Consumers
– Queues
– Thread Pool
– Monitors
> Strategies
– Queue Strategies : ConcurrentLinkedQueue vs LinkedBlockingQueue
16

17. Anti-Patterns
> String concatenation
– Make sure to use a StringBuilder
– Compiler now optimize simple “Aaa” + “Bbb” concatenation
> Avoid very long methods (thousands of lines of code)
> Avoid exposing the concrete representation of an object
– Set vs LinkedHashSet.
– Not a performance issue, but will require more work when optimizing code
for performance later
> Try to define only the methods you need.
17

18. Java Collections
> Vector and Hashtable are synchronized for all methods
– Pay the price even if not necessary
> Some Java collection classes are not synchronized by default
– ArrayList, LinkedList replace Vector
– HashSet, HashMap replace Hashtable
> To synchronize, wrap in a class
> With a “static factory”
– Collections.synchronizedList(new ArrayList())
> ConcurrentHashMap, for concurrency
– But watch when using the iterator
18

19. Critical Sections
> Try to minimise the code and time spent in the critical sections
> But the throughput is limited by the time spent in the largest critical section
– Example : LinkedBlockingQueue
 200 000 operations on x64 processor
 20 000 operations on the T2000 processor
– We use it for the WorkQueue and Access Logs

20. Caching data
> Using Caches reduce the disk access thus should provide better performances
> But cache eviction add pressure to the GC
– When modifying entries
– When the cache is too small to hold all data
> A cache is also a contention point
> If you want to cache objects, make sure you cache those that will be reused.
> Alternate possibility, use “thread local” cache.
– But watch out for the cost (with 1000 threads?)

21. Server monitoring
> Getting statistics for a server is mandatory
> Beware of contention
– Stats are updated frequently
– But seldom read
> A strategy could be to keep per thread statistics and collect them on demand
– Not yet implemented in OpenDS !

22. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with tuning Sun JVM
> Conclusion
22

23. Performance Tuning
> When dealing with performances, you should consider the whole system
– Java VM
– OS
– Hardware : CPU, Memory, Disks, Network...
> In our case, we try to avoid disk I/Os
– And try to cache as much of the database
> We also want deterministic response times
– Avoid any Full GC (Stop The World)
– Make sure minor GC pauses are as small as possible
23

24. JVM Tuning for OpenDS
> Super Size The Heap !
– We use 32GB Heaps, sometime
up to 96GB
– 2GB for the New Generation (or
¼ of heap if < 8GB)
– -Xms32768M -Xmx32768M
-Xmn2048M
> Use CMS
– -XX:+UseConcMarkSweepGC
– -XX:+UseParNewGC
> -XX:MaxTenuringThreshold=1
– Avoid copy of objects in New Gen
24

25. Some interesting JVM Options
> -XX:CMSInitiatingOccupancyFraction=70
– Define the amount of occupancy in Old Gen before starting to collect
– Larger = better throughput but higher full GC risk
> -XX:+UseCompressedOops
– For 64bits JVM, less than 32BG of heap
– Will be the default in coming Java 6 updates
> If running on processor with NUMA architecture
– -XX:+UseNUMA
> -XX:+AggressiveOpts
– Enables aggressive JIT optimizations, not related to GC
25

26. The Garbage First (G1) GC
> Introduced in the Java HotSpot VM in JDK 7.
> An experimental version of G1 has also been released since Java SE 6 Update
14.
> G1 is the long-term replacement for HotSpot's low-latency Concurrent Mark-
Sweep (CMS) GC
> Should be officially supported with Java SE 6 Update 21
26

27. G1 Characteristics
> Future CMS Replacement
– Server “Style” Garbage Collector
– Parallel, Concurrent
– Generational
– Good Throughput
– Compacting
– Improved ease-of-use
– Predictable (though not hard real-time)
> The main stages consist of remembered set (RS) maintenance, concurrent
marking, and evacuation pauses.
27

28. JVM Options With G1
> -XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
> PauseTime (Hints, Goal with no promise, otherwise use Java Real Time )
– -XX:MaxGCPauseMillis=50 (target of 50 milliseconds)
– -XX:GCPauseIntervalMillis=1000 (target of 1000 msecs)
> Generation Size
– -XX:+G1YoungGenSize=512m (for a 512 MB young gen)
> Parallelism
– -XX:+G1ParallelRSetUpdatingEnabled
– -XX:+G1ParallelRSetScanningEnabled
28

29. OpenDS et G1
> Goal: Avoid any Full GC, best control of pauses
> Collaboration between the HotSpot and the OpenDS teams
– OpenDS is used as a “Large” reference application
– Between 10 and 20 enhancements integrated in G1 following the tests
– Performance with large heaps improved by a factor of 10
> We're still discovering it
– When doing read operations, we see pauses between 10 and 20 ms with
32GB JVM
– But we're still seeing Full GC when doing Write operations (More garbage,
stresses more the Old Gen)
– Hopefully this will be resolved in next builds
29

30. OpenDS G1 and Searches
30

31. AGENDA
> Introduction to the OpenDS project
> Architecture, Design Patterns and Tips
> Experiences with Sun JVM
> Conclusion
31

32. Summary
> OpenDS
– A open source LDAP directory server, 100% pure Java
– Easy to install and use
– Designed for high performance and high scalability
> We saw some patterns and tips used in the OpenDS project
> Knowing and understanding the JVM and GC is required to build high
performance server
– Tuning JVM and GC is an art
– Performance engineering is a profession
> Who said that Java is slow ?!
32

33. The Art of GC Tuning
JavaOne Presentation: GC Tuning in HotSpot JVM
http://developers.sun.com/learning/javaoneonline/j1sessn.jsp?sessn=TS-4887&yr=2009&track=javase
33

34. Now...
> Give OpenDS a try
– http://www.opends.org
> Join our community:
– Join/Login sur Java.net
– http://opends.dev.java.net
– Request a Role
– Subscribe to the mailing lists
– IRC: #opends on freenode.net
> OpenDS is localized in several
languages. It's community based,
through online tools. An easy way to
participate.
34

35. Ludovic Poitou blogs.sun.com/Ludo
Sun Microsystems Ludovic.Poitou@sun.com
Matthew Swift
Sun Microsystems Matthew.Swift@sun.com