This document discusses techniques for testing multithreaded Java applications for synchronization problems. It begins by explaining why multithreading is used and the new types of defects that can arise, such as deadlocks, livelocks, and race conditions. It then covers various testing techniques like load/stress testing, testing specific thread interleavings, testing all possible interleavings, and using instrumentation. It provides examples of tools that can be used for each technique and recommends books on the topic.

1. Testing Multithreaded Java
Applications for
Synchronization Problems
Vassil Popovski
vpopovski@vmware.com
www.vmware.com

2. Why multithreading?
core 1 core N
…
cache cache
shared memory
A modern CPU

3. Why multithreading?
Thread 1
thread … Thread N
thread
synchronization
Java heap
A modern Java
Application

4. Chuck Norris can write multi-threaded Java applications with a
single thread
Image source: http://4.bp.blogspot.com/-RhqCAgEyhxs/T7CqtsC0RUI/AAAAAAAAJuw/LnkKl4xBcFA/s1600/ChuckNorris.png

5. New concept -> new problems
Multithreading leads to:
• Non-determinism
• New types of defects exist such as deadlocks,
livelocks, thread contentions and race conditions

6. Threads interleaving
Thread 1 Thread 2
T1: Block A
T2: Block 1
T2: Block 2
T1: Block B

7. Threads interleaving
Thread 1 Thread 2
T2: Block 1
T1: Block A
T2: Block 2
T1: Block B

8. T1: Block A T1: Block A T1: Block A
T1: Block B T2: Block 1 T2: Block 1
T2: Block 1 T1: Block B T2: Block 2
T2: Block 2 T2: Block 2 T1: Block B
T2: Block 1 T2: Block 1 T2: Block 1
T2: Block 2 T1: Block A T1: Block A
T1: Block A T2: Block 2 T1: Block B
T1: Block B T1: Block B T2: Block 2

9. Number of different thread interleavings

10. QUIZ
Q: What is the number of all interleavings for 3
threads with 3 blocks each?

11. QUIZ#2
Q: What is the number of all interleavings for 100
threads with 5 blocks each?
A: 65425664332523564432322
424953034168268979855850496989515584307752298932760805211068470651079402188475376110921674489441757
100735903402821009297053297794211809385509051615356527155445783425009164096533350469391547252484286
352170768347268799054153532042868164413927778850486516865333071658697094887692175873537666079346686
842361469250698011080482809500911511130491996847437245286443821625843666165114941515090783620437429
415942216292191680146737140994107899459749143795942309665194029514713124663377783172923345131361940
229192120804703412493451507233690788801188111838709132087076627314402580265097945582570927195669521
598427700259644897641650396932627145510414432349054625767654125966166480949639718802313608310300009
569524993138782412735779520574624492099944487052157575950286282799011493381356627940388478077680384
306582854732555208087863968812536837179492786875593176275633769413252546329077858090833823115446889
102306231699826529926420707529253684105656295906933999334649790703077726680609672178258148765366567
390695607298478918976308327339329621077231101219394256694201487015134200730008353085104356877584247
630194547240102504944587886157078430944582189784261271608483330872723017090959069769287306115617186
352177030505501663380710718097318392846079061230267394203670163609062726137904455998424717036312138
754515697607350323258954538452071707071282635719375409459015826288484347553786552736926086009976292
360046325376069078312201822678281992756614855571919537160977811910388279229176987762264097262002703
425073318544671121015763363797279655888433399855420190892643324103098493700405493206195043772398231
236266443176076877706895863803891037350530509973799077521555018278731597362632766739808771841037553
742905870882213799310132964516518399560481797842812543528613844661728630365645349106177833375046753
439848225204235068908502565168578778645210878189710308017886705257632553314025472000000000000000000
0000000000000000000000000000000

12. What to test for?
• safety: nothing bad happens
• liveness: something good eventually happens

13. Example 1: Single Lane Bridge
Image source: http://www.doc.ic.ac.uk/~jnm/book/ppt/ch7.ppt
• safety:
– no car crash
• liveness:
– every car eventually get an opportunity to cross the bridge

14. Example 2: BoundedBuffer or
Producer/Consumer
Producer 1
Consumer 1
Producer 2
Consumer 2
Producer 3
…
…
Consumer M
Producer N
• safety:
– head and tail of the queue must not over run each other
• liveness:
– when the queue contains an item, the consumer process must be able
to access the queue and when the queue contains space for another
item, the producer process must be able to access the queue

15. How to test for synchronization issues?
• Load/Stress testing (blackbox, whitebox)
• Specific interleavings testing (whitebox)
• All interleavings testing (whitebox)
• Instrumentation (blackbox, whitebox)

16. Load/Stress testing
• A lot of threads and operations to exercise
different interleavings

17. Demo!

18. Deadlock example
Thread 1 Thread 2
synchronized ( A ) {
synchronized ( B ) {
synchronized ( B ) {
synchronized ( A ) {
Deadlock !!!

19. Load/Stress testing
• Tools
– ExecutorService -
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Execu
– TestNG -
http://testng.org/doc/documentation-main.html#annotations
– JUnit -
http://www.junit.org/apidocs/junit/extensions/ActiveTestSuite.html
• JUnitPerf -
http://testng.org/doc/documentation-main.html#annotations
– GroboUtils -
http://groboutils.sourceforge.net/testing-junit/using_mtt.html
– Custom Threading

20. Specific interleavings testing
• Deterministic and repeatable tests

21. MultithreadedTC
• Based on internal clock
– Tick increases when all threads are blocked
– waitForTick(tick) – blocks the thread until tick reaches
certain value
– assertTick(tick) – asserts the current tick
• Methods that start with “thread” are executed in
separate threads

22. MultithreadedTC
BoundedBuffer with size = 1
Thread 1 Thread 2
put 42
put 17
(blocks)
get 42
get 17

23. MultithreadedTC
BoundedBuffer with size = 1
Thread 1 Thread 2
Tick 0
put 42 waitForTick(1)
put 17 Tick 1
(blocks)
get 42
assertForTick(1)
get 17

24. Demo!

25. IMUnit
• Events based
– IMUnit.fireEvent(“event1”)
– @Schedule(“event1 -> event2”)
– @Schedule(“[event1] -> event2”)

26. Demo!

27. ThreadWeaver
• Two threads only – Main and Secondary
• By default: For each line of Main thread (T1) ->
block and execute fully Secondary thread (T2)
T2: Block 1 T1: Block A T1: Block A
T2: Block 2 T2: Block 1 T1: Block B
T1: Block A T2: Block 2 T2: Block 1
T1: Block B T1: Block B T2: Block 2

28. ThreadWeaver
• Two threads only – Main and Secondary
• By default: For each line of Main thread (T1) ->
block and execute fully Secondary thread (T2)
• Powerful Breakpoints

29. Demo!

30. Specific interleavings testing
• Tools
– MultithreadedTC -
http://code.google.com/p/multithreadedtc/
• Enhanced version for Junit 4 -
http://code.google.com/p/multithreadedtc-junit4/
– IMUnit - http://mir.cs.illinois.edu/imunit/
– ThreadWeaver -
http://code.google.com/p/thread-weaver/
– Awaitility - http://code.google.com/p/awaitility/

31. All interleavings testing
• Exercise all possible interleavings

32. JavaPathFinder
Traditional testing
OK
Code
Testing
error
Image source: http://javapathfinder.sourceforge.net/events/JPF-workshop-050108/tutorial.ppt

33. JavaPathFinder
Model Checking with JavaPathFinder
Code
OK
Model Checking
error trace
properties Line 5: …
Line 12: …
…
Line 41:…
Line 47:…
Image source: http://javapathfinder.sourceforge.net/events/JPF-workshop-050108/tutorial.ppt

34. JavaPathFinder
Program testing can be used to show the presence
of bugs, but never to show their absence!
--Edsger Dijkstra

35. Demo!

36. All interleavings testing
• Tools
– JavaPathFinder (JPF) -
http://babelfish.arc.nasa.gov/trac/jpf
• JavaRaceFinder -
http://www.cise.ufl.edu/research/JavaRacefinder/Java_Rac
eFinder/JRF_Home.html

37. Instrumentation
• Instrument the code to catch problems easier

38. AspectJ
AspectJ Compiler
Original code Instrumented .
.class or .jar class or .jar
Aspect Definition
.aj

39. Demo!

40. Instrumentation
• Tools
– AspectJ - www.eclipse.org/aspectj/
• RacerAJ - http://www.bodden.de/tools/raceraj/
– CalFuzzer - http://srl.cs.berkeley.edu/~ksen/calfuzzer/
– (commercial) Flashlight / Jsure -
http://www.surelogic.com/concurrency-tools.html

41. Other tools
• CHESS (native DLLs & managed executables) -
http://research.microsoft.com/en-
us/projects/chess/
• MoonWalker (.NET)-
http://code.google.com/p/moonwalker/

42. Recommended books

43. Thank you!

44. Q&A

45. Source code packages (1)
• org.java2days2012.multithreaded.common
– Several implementations of AccountManager, Counter, BounderBuffer and
MultipleReadersSingleWriter
• org.java2days2012.multithreaded.executorservice
– Load/stress tests using ExecutorService
• org.java2days2012.multithreaded.testng
– Load/stress tests using TestNG
• org.java2days2012.multithreaded.mtc
– MultithreadedTC tests
• org.java2days2012.multithreaded.imunit
– IMUnit tests
• org.java2days2012.multithreaded.threadweaver
– ThreadWeaver tests

46. Source code packages (2)
• org.java2days2012.multithreaded.javapathfinder
– JavaPathFinder examples
• ca.mcgill.sable.racer
– RacerAJ source code
• org.java2days2012.multithreaded.aspectj
– Custom aspect that increase the chance of hitting multithreading problem
during testing
• org.java2days2012.multithreaded.blockbox.*
– Rest based AccountManager client/test and server (in .server package)