Slide from presentation titles Concurrency in Eclipse: Best Practices and Gotchas by Andrew McCulloch and Carlin Rogers at EclipseCon 2011 in Santa Clara, CA.

1. Concurrency in Eclipse:Best Practices and Gotchas Andrew McCulloch andrew.mcculloch@oracle.com Carlin Rogers carlin.rogers@oracle.com

2. About Us Andrew McCulloch Andrew.McCulloch@oracle.com Principle Member of Technical Staff at Oracle working on the Oracle Enterprise Pack for Eclipse (OEPE) Lead on AppXray feature which manages dependencies in a user's workspace Area of interest include concurrency and performance tuning Carlin Rogers Carlin.Rogers@oracle.com Principle Member of Technical Staff at Oracle working on the Oracle Enterprise Pack for Eclipse (OEPE) Committer on the JavaServer Faces Tools Project at Eclipse Areas of interest include web frameworks, concurrency, and performance tuning

4. Reactions to Concurrency "...we always get deadlock messages, and have no idea what they mean (in context) and we just ignore them.“ - Anonymous SQL Server Developer

5. Agenda Overview concurrency Describe our project and experience Problems and gotchas we encountered Practical debugging tips Best practices Questions

7. java.lang.Thread, java.lang.Runnable

8. synchronized keyword

9. java.util.concurrent package (Java SE 5)

10. Eclipse Building Blocks

11. org.eclipse.core.runtime.jobs.Job

12. org.eclipse.core.runtime.jobs.ISchedulingRule

13. org.eclipse.core.runtime.jobs.ILock

15. Interleaving of threads results in undefined or incorrect state

16. Deadlock / Livelock

17. Progress can not be made due to competing locks (deadlock), broken lock avoidance (livelock)

18. Starvation / Fairness

20. Problems We Encountered Race Conditions Lazy initialization of, and access to, models that were not thread safe - uncontrolled concurrent access to mutable data.

21. Race Conditions Exceptions from EMF models Exceptions during access/initialization of EMF Models [Bugzilla 228748] java.lang.NullPointerException at org.eclipse.emf.ecore.util.EContentAdapter.addAdapter(EContentAdapter.java:352) at org.eclipse.emf.ecore.util.EContentAdapter.setTarget(EContentAdapter.java:225) at org.eclipse.emf.ecore.util.EContentAdapter.setTarget(EContentAdapter.java:188) at org.eclipse.emf.common.notify.impl.BasicNotifierImpl$EAdapterList.didAdd(BasicNotifierImpl.java:77) at org.eclipse.emf.common.util.BasicEList.addUnique(BasicEList.java:646) at org.eclipse.emf.common.util.BasicEList.add(BasicEList.java:626) at org.eclipse.emf.common.notify.impl.BasicNotifierImpl$EAdapterList.add(BasicNotifierImpl.java:129) at org.eclipse.jst.jsf.core.jsfappconfig.JSFAppConfigManager.addFacesConfigChangeAdapter( JSFAppConfigManager.java:769) … at org.eclipse.jst.jsf.core.jsfappconfig.JSFAppConfigManager.getFacesConfigModels( JSFAppConfigManager.java:409)

22. Debugging Tip Intermittent bugs can be very difficult to reproduce in a system test. Difficult to reproduce race condition in EMF code using debugger. Narrow the scope - create simple controlled test of the specific code path in the stack trace from the exception. Example: launch multiple threads, each calling the JSFAppConfigManager.getFacesConfigModels() method

23. Best Practices “Synchronize access to shared mutable data” – Joshua Bloch

24. Problems We Encountered Race Conditions Lazy initialization of, and access to, models that were not thread safe - uncontrolled concurrent access to mutable data. Deadlocks Know your third party code - it may take a scheduling rule.

25. Lock Order and Third Party Code Deadlocks from calls in other libraries that require a scheduling rule. For example… Thread A - AppXRay Job Thread B – Main - Calls beginRule with the IProject as the scheduling rule. - UI component calls to access information from shared model, enters synchronized block. - Tries to access information from shared model, waits to enter synchronized block locked by Thread B. - Third party code requires JDT to resolve classpaths for the projects, and tries to take the IProjectscheduling rule.

27. Eclipse Debugger - turn on "Show Monitors" to see locks for synchronized blocks

28. Mission Control with Oracle JRockit JVM

29. jstack with a process ID for a thread dump

31. Problems We Encountered Race Conditions Lazy initialization of, and access to, models that were not thread safe - uncontrolled concurrent access to mutable data. Deadlocks Know your third party code - it may take a scheduling rule. Thread Starvation JobManager.waitForRun prevented jobs with complex relationships from running in a timely manner.

32. Scheduler issues [Bugzilla 320329] Not a typical Thread Starvation case, but illustrates a point. Job dependencies can be complex when multiple jobs launch in response to a single event. In the UI progress dialog the running job appeared to be deadlocked when it was not at fault. Dynamic Analysis tools can pinpoint the true problem Our profiling tool pointed out that a loop in the scheduler was executed millions of times. The issue was in the Job scheduler in open source code After several iterations at a patch the issue was resolved

33. Debugging Tip Be familiar with dynamic analysis tools. We use several including Oracle JRockit Mission Control and third-party tools. Don’t ignore org.eclipse code in your analysis. Eclipse is a quality product but bugs do get through and the community can help resolve them.

34. Best Practices “Synchronize access to shared mutable data” Enforce a consistent locking order. “Avoid excessive synchronization” Don’t be hesitant to debug into open source code, submit questions, bugs, and patches. This make your product and Eclipse better

35. Gotchas Job.yieldRule() Third party code may move your job to waiting state and yield the scheduling rule to other jobs to avoid potential deadlocks. (bug 283449)

36. Job.yieldRule() Thread dump from deadlock java.lang.Object.wait(Native Method) at org.eclipse.core.internal.jobs.ThreadJob.waitForRun(ThreadJob.java:270) at org.eclipse.core.internal.jobs.ThreadJob.joinRun(ThreadJob.java:199) at org.eclipse.core.internal.jobs.JobManager.yieldRule(JobManager.java:1398) at org.eclipse.core.internal.jobs.InternalJob.yieldRule(InternalJob.java:600) at org.eclipse.core.runtime.jobs.Job.yieldRule(Job.java:709) at org.eclipse.wst.sse.core.internal.model.ModelManagerImpl$ SharedObject.waitForLoadAttempt(ModelManagerImpl.java:139) at org.eclipse.wst.sse.core.internal.model.ModelManagerImpl.getExistingModel( ModelManagerImpl.java:1137)

38. Look for yieldRule() in the stack.

39. Determine if you can move the call to third party code outside of the synchronization.

41. Gotchas Job.yieldRule() Third party code may move your job to waiting state and yield the scheduling rule to other jobs to avoid potential deadlocks. (bug 283449) EventLoopProgressMonitor Starting a rule on the UI thread will have unintended consequences regardless of how long the rule is held.

42. Event loop will be interleaved Acquiring a scheduling rule from the UI thread may appear as a UI freeze. Alternatively in some instances acquiring a scheduling rule on the UI thread may cause calls to the progress monitor to execute other UI events which will eliminate the appearance of a freeze but can cause errors with non-reentrant code. RunnableWithProgress also replaces null progress monitors.

43. Debugging Tip Analyze stack trace for EventLoopProgressMonitor Look for entries on the stack that you would not expect to be executing from within your code. Look for tell-tale exception “Attempted reentrant call …” Determine where the scheduling rule is acquired Can that code be moved off of the UI thread? Can the UI widget be redesigned to update asynchronously?

44. Best Practices “Synchronize access to shared mutable data” Enforce a consistent locking order. “Avoid excessive synchronization” Don’t be hesitant to work with open source code. Use ILock for synchronization Run as little as possible on the UI thread. Move code requiring ISchedulingRules to a Job.

45. In Conclusion… “Synchronize access to shared mutable data” Enforce a consistent locking order. “Avoid excessive synchronization” Don’t be hesitant to work with open source code. Use ILock for synchronization Run as little as possible on the UI thread. Move code requiring ISchedulingRules to a Job.

46. Suggested References Java Concurrency in Practice Brian Goetz (2006) Effective Java (2nd Edition) Joshua Bloch (2008) Concurrent Programming in Java Doug Lea (1999)

47. Any Questions?