The document discusses practical testing tips related to concurrency and unit testing using JUnit, highlighting issues like race conditions and deadlocks that arise in multithreaded applications. It introduces various tools and techniques for effective testing, including the importance of writing tests, using custom runners, and managing shared resources. The document also emphasizes the significance of understanding concurrency challenges and provides resources for further learning.

1. JUnit PowerUP
Practical Testing Tips
James McGivern

2. About James
Likes cats
Talks fast
Technical Evangelist
Hates Marmite
Mathematician turned
Computer Scientist
Lives in London

3. JUnit
vs
TestNG

4. One Busy Day On
Concurrent Island...
Intro

6. Concurrency?
concurrent - adjective
1. occurring or existing simultaneously or side by
side: concurrent attacks by land, sea, and air.
2. acting in conjunction; co-operating.
3. having equal authority or jurisdiction.
4. accordant or agreeing.
5. tending to or intersecting at the same point: four
concurrent lines.

7. The behaviour of a single threaded application is
deterministic
A multithreaded application may appear
stochastic
Concurrency introduces new problems:
deadlocks
resource starvation (e.g livelocks)
race conditions
contention
•
•
•
•

8. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

9. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

10. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

11. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

12. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

13. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

14. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

15. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

16. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

17. But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
INSTANCE = new Singleton();
}
return INSTANCE;
}
and two threads: Thread A, Thread B

18. Each statement is an atomic block
Each thread executes a non-negative number of
atomic blocks
Threads take turns but order is not guaranteed
Given a number of threads t, and a group of
statements s, the number of execution order
permutations is given by:
interleavings(t, s) =
(ts)!
t
(s!)

19. The JUnit Cup
Challenge
Level 0

21. public class FibonacciSequenceTest {
FibonacciSequence fib = new FibonacciSequence();
@Test public void shouldOutputZeroWhenInputIsZero() {
assertThat(fib.get(0), is(0))
}
@Test public void shouldOutputOneWhenInputIsOne() {
assertThat(fib.get(1), is(1));
}
@Test public void shouldCalculateNthNumber(){
for(int i = 2; i <= 10; i++) {
int x = fib.get(i-1);
int y = fib.get(i-2);
int sum = x + y;
assertThat(fib.calculate(i), is(sum);
}
}
}

22. Cost/Benefit Ratio
(TDD) Unit tests are executable specifications of
the code
Unit (+ integration tests) will never find all the bugs
Writing tests takes time
Time is limited
Which tests should I write and which should I
forgo?

23. JUnit Runners
Bundled
Suite, Parameterized
•
• Theories, Categories, Enclosed
Popular 3rd party runners
Mockito, PowerMock(ito)
•
Custom
JBehave, Spock
•
• dynamic tests?

24. Custom Runner
public abstract class Runner
implements Describable {
public Runner(Class<?> testClass){...}
public abstract Description getDescription();
public abstract void run(RunNotifier n);
public int testCount() {
return getDescription().testCount();
}
}

25. Description
Description.createSuiteDescription(...)
Description.createTestDescription(...)
Description#addChild(Description d)

26. RunNotifier
fireTestStarted(Description d)
fireTestFinished(Description d)
fireTestFailure(Failure f)
fireTestIgnored(Description d)
fireTestAssumptionFailed(Failure f)

27. Warning
• A very coarse way of modifying test execution
• Even when extending from BlockJUnit4ClassRunner
there is a degree of complex code
• Runners can not be composed e.g.
@RunWith(MockitoJUnitRunner.class}
@RunWith(SpringJUnit4ClassRunner.class)
public class SomeTest {...}
@RunWith({MockitoRunner.class,
SpringJUnit4ClassRunner.class})
public class SomeTest {...}

28. JUnit Rules
PowerUP

29. Rules can:
Read/Write test metadata
Modify the test before execution
Modify the result after execution
Suite vs Class == @ClassRule vs @MethodRule

30. public interface TestRule {
Statement apply(Statement base,
Description description);
}
public class MockRule implements TestRule {
private final Object target;
public MockRule(Object target) {
this.target = target;
}
public Statement apply(final Statement base,
Description description) {
return new Statement() {
public void evaluate() throws Throwable {
MockitoAnnotations.initMocks(target);
base.evaluate();
}
};
}
}

31. The Thread
Challenge
Level 1

33. static class VolatileInt { volatile int num = 0; }
public void shouldIncrementCounter() {
final int count = 32 * 1000;
final int nThreads = 64;
ExecutorService es = Executors.newFixedThreadPool(nThreads);
final VolatileInt vi = new VolatileInt();
for (int i = 0; i < nThreads; i++) {
es.submit(new Runnable() {
public void run() {
for (int j = 0; j < count; j += nThreads)
vi.num++;
}
});
es.shutdown();
es.awaitTermination(10, TimeUnit.SECONDS);
assertEquals(count, vi.num);
}
http://vanillajava.blogspot.co.uk/2011/08/why-testing-code-for-thread-safety-is.html

34. Weapons Guide
• Java Concurrency in Practice - Brian Goetz
• Java Performance - Charlie Hunt
• Effective Unit Testing: A guide for Java developers Lasse Koskela
• Programming Concurrency on the JVM -Venkat
Subramaniam

35. Choreographer’s
Workshop
Level 2

37. http://www.recessframework.org/page/map-reduce-anonymous-functions-lambdas-php
Text

38. Awaitility
PowerUP

39. A Simple DSL
No more Thread.sleep(), or while loops
await().until(
new Callable<Boolean>() {
public Boolean call() throws Exception {
return userRepository.size() == 1;
}
};
)

40. The previous example was not particularly re-usable.
Let’s fix that!
await().until(sizeOf(repository), equalTo(1));
where
private Callable<Integer> sizeOf(Collection c){
return new Callable<Integer>() {
public Boolean call() throws Exception {
return c.size();
}
};
}

41. Advanced Waiting
• Callable is still a lot of boiler plate...
await().untilCall(to(repository).size(), equalTo(3));
• Using reflection
await().until(
fieldIn(repository).ofType(int.class)
.andWithName(“size”), equalTo(3)
);
• Polling
with()
.pollInterval(ONE_HUNDERED_MILLISECONDS)
.and().with().pollDelay(20, MILLISECONDS)
.await("user registration").until(
userStatus(), equalTo(REGISTERED));

42. The Shared
Resource Contest
Level 3

44. Race Conditions
A race condition is a situation in which two or more
threads or processes are reading or writing some
shared data, and the final result depends on the timing
of how the threads are scheduled.
public class Counter {
protected long count = 0;
public void add(long value){
this.count = this.count + value;
}
}

45. ThreadWeaver
PowerUP

46. public class ConcurrentHashMapTest {
ConcurrentMap<String, Integer> map;
@ThreadedBefore
public void before() {
map = new ConcurrentHashMap();
}
@ThreadedMain
public void mainThread() {
map.putIfAbsent("A", 1);
}
@ThreadedSecondary
public void secondThread() {
map.putIfAbsent("A", 2);
}
}
@ThreadedAfter
public void after() {
assertEquals(map.get(“A”), 1);
}

47. Works by instrumenting bytecode at runtime
Does not integrate with JUnit but can be embedded
@Test
public void testThreading() {
new AnnotatedTestRunner()
.runTests(
this.getClass(),
ConcurrentHashMapTest.class
);
}
Has fine-grained control over breakpoints/code position
Documentation is ok but not a lot on in-depth material
PROJECT STATUS UNKNOWN, MAYBE INACTIVE

48. Princess
Protection HQ
Level 4

50. A recurrent
producer-consumer
information
processing network
for anomaly
detection

52. Java Path Finder
http://babelfish.arc.nasa.gov/trac/jpf
JPF created by NASA
Open-sourced in 2005
Is a JVM written in Java that runs on the JVM
Primarily used as a model checker for concurrent
programs, e.g deadlocks, race conditions, NPEs
Very powerful
Not very usable (e.g. limited CI support)

53. Byteman
PowerUP

54. Java Bytecode Manipulating Agent - Byteman
Uses java.lang.intrument as an agent
Can be used for:
fault injection testing, tracing during test and in
production, an alternative AOP (e.g. cross-cutting
logging)
Using instrumentation in tests means:
fewer mocks
less boilerplate to generate abnormal senarios

55. Why Byteman?
AOP is powerful but complex:
• code is targeted indirectly rather than at the class/
method declaration site
• AOP definition languages can be a bit odd/hard to
work with
• Impact at runtime is sometimes significant to
application performance due to pointcut definitions
• not always trivial to turn-off, alter, or remove advice

56. EBCA Rules
A Byteman rule consists of:
• Event - the class/interface/method target
• Binding - initialise rule values
• Condition - a boolean expression (optional)
• Action - some Java code that may return or
throw (can not break contracts)
Rules may have state, i.e 1st invocation do A, 2nd
invocation do B, etc.

57. Fault Injection
RULE simulate exception from Executor
INTERFACE ^java.util.Executor
METHOD execute
AT ENTRY
IF callerEquals("ServiceInstanceImpl.execute", true)
DO traceln(“Throwing exception in execute”);
THROW new java.util.concurrent.RejectedExecutionException();
ENDRULE

58. In-built Rules
Tracing
traceOpen, traceClose, traceln, traceStack
Shared Rule State
flag, clear, flagged, countDown, incrementCounter
Synchronization
waitFor, signalWake, rendezvous, delay
Timing
createTimer, getElapsedTime, resetTimer
Call Stack Checking
callerEquals, callerMatches
Recursive Triggering
disable/enableTriggering

59. BMUnit
@RunWith(BMUnitRunner.class)
@BMScript(value="traceRules", dir="scripts")
class DBTests1 {
@Test
@BMRule(
className="UserRepository”,
methodName="findByName",
condition=
"$1.getName().contains("James")",
action="THROW new UserNotFoundException")
public void shouldErrorIfNoUserFound()
{
...
}
}

60. Machine Lab
Bouns LeveL

62. ConcurrentUnit
Website: https://github.com/jhalterman/concurrentunit
JUnit addon library
Comprised of:
• A thread co-ordinator Waiter
• Allows assertions to be made in worker threads
• Waiter collects and returns outcomes of
assertions

63. FindBugs
Website: http://findbugs.sourceforge.net
Integrates with Maven, Gradle, Sonar, etc
Multithreaded correctness group:
• Inconsistent synchronization
• Field not guarded against concurrent access
• Method calls Thread.sleep() with a lock held
• Creation of ScheduledThreadPoolExecutor with
zero core threads

64. Freud
Website: https://github.com/LMAX-Exchange/freud
A framework for writing static analysis tests
Lacks good documentation except code examples
Has rules for:
✦
✦
✦
✦
Java sources
Java class objects. (i.e analysing the java.lang.Class
object)
Java class files (i.e analysing the ".class" file)
Spring framework XML configuration files

65. Summary
Scores

67. Lv.0 - Testing Tutorial
• TDD is your friend
• Prefer rules over runners
• Don’t be afraid to write custom rules and
runners
• Ensure you spend your time on the
important tests

68. Lv.1 - The Thread
Challenge
• Modelling real world behaviour of your application
is hard
• Don’t assume your tests are exhaustively testing
the scenarios
• Behaviour depends heavily on the system (e.g. load)
• Know your stuff - read read read

69. Lv2. Choreographer’s
Workshop
• It is essential to test the co-ordinated
behaviour of threads
• If threads are not sharing resources we
don’t need to care about synchronisation
between thread
• Awaitility allows us to check that some end
state if finally reached (within the given
time)

70. Lv.3 - The Shared
Resource Contest
• When a resource is shared between
threads new problem cases can arise, e.g.
races
• Often the problem can be simplified to the
case of 2 threads
• To test the orders of execution of 2
threads we can use ThreadWeaver

71. Lv.4 - Princess
Protection HQ
• Many problems can’t be reduced to a 2 thread
model
• JPF can provide some automated analysis of
concurrency issues
• Byteman allows us to take control of the whole
application
• Both are very heavy weight and should not be

73. JUnit PowerUP
GAME OVER

74. Credits

75. JUnit
http://www.junit.org
Awaitility
http://code.google.com/p/awaitility

76. ByteMan
http://www.jboss.org/byteman
ThreadWeaver
http://code.google.com/p/thread-weaver

77. ConcurrentUnit
https://github.com/jhalterman/concurrentunit
FindBugs
http://findbugs.sourceforge.net

78. Freud
https://github.com/LMAX-Exchange/freud
Java PathFinder
http://babelfish.arc.nasa.gov/trac/jpf

79. JUnit PowerUP
Practical Testing Tips
James McGivern