Your SlideShare is downloading. ×
JUnit PowerUP
Practical Testing Tips
James McGivern
About James
Likes cats

Talks fast
Technical Evangelist
Hates Marmite

Mathematician turned
Computer Scientist

Lives in L...
JUnit
vs
TestNG
One Busy Day On
Concurrent Island...

Intro
Concurrency?
concurrent - adjective
1. occurring or existing simultaneously or side by
side: concurrent attacks by land, s...
The behaviour of a single threaded application is
deterministic
A multithreaded application may appear
stochastic
Concurre...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
But Why?
Consider the old singleton pattern
getInstance() method:
public Singleton getInstance() {
if(INSTANCE == null) {
...
Each statement is an atomic block
Each thread executes a non-negative number of
atomic blocks
Threads take turns but order...
The JUnit Cup
Challenge

Level 0
public class FibonacciSequenceTest {
FibonacciSequence fib = new FibonacciSequence();
@Test public void shouldOutputZeroWh...
Cost/Benefit Ratio
(TDD) Unit tests are executable specifications of
the code
Unit (+ integration tests) will never find all ...
JUnit Runners
Bundled
Suite, Parameterized

•
• Theories, Categories, Enclosed

Popular 3rd party runners
Mockito, PowerMo...
Custom Runner
public abstract class Runner
implements Describable {
public Runner(Class<?> testClass){...}
public abstract...
Description
Description.createSuiteDescription(...)
Description.createTestDescription(...)
Description#addChild(Descriptio...
RunNotifier
fireTestStarted(Description d)
fireTestFinished(Description d)
fireTestFailure(Failure f)
fireTestIgnored(Descript...
Warning
• A very coarse way of modifying test execution
• Even when extending from BlockJUnit4ClassRunner
there is a degre...
JUnit Rules

PowerUP
Rules can:
Read/Write test metadata
Modify the test before execution
Modify the result after execution
Suite vs Class == @...
public interface TestRule {
Statement apply(Statement base,
Description description);
}
public class MockRule implements T...
The Thread
Challenge

Level 1
static class VolatileInt { volatile int num = 0; }
public void shouldIncrementCounter() {
final int count = 32 * 1000;
fin...
Weapons Guide
• Java Concurrency in Practice - Brian Goetz
• Java Performance - Charlie Hunt
• Effective Unit Testing: A g...
Choreographer’s
Workshop

Level 2
http://www.recessframework.org/page/map-reduce-anonymous-functions-lambdas-php

Text
Awaitility

PowerUP
A Simple DSL
No more Thread.sleep(), or while loops
await().until(
new Callable<Boolean>() {
public Boolean call() throws ...
The previous example was not particularly re-usable.
Let’s fix that!
await().until(sizeOf(repository), equalTo(1));

where
...
Advanced Waiting
• Callable is still a lot of boiler plate...
await().untilCall(to(repository).size(), equalTo(3));

• Usi...
The Shared
Resource Contest

Level 3
Race Conditions
A race condition is a situation in which two or more
threads or processes are reading or writing some
shar...
ThreadWeaver

PowerUP
public class ConcurrentHashMapTest {
ConcurrentMap<String, Integer> map;
@ThreadedBefore
public void before() {
map = new ...
Works by instrumenting bytecode at runtime
Does not integrate with JUnit but can be embedded
@Test
public void testThreadi...
Princess
Protection HQ

Level 4
A recurrent
producer-consumer
information
processing network
for anomaly
detection
Java Path Finder
http://babelfish.arc.nasa.gov/trac/jpf
JPF created by NASA
Open-sourced in 2005
Is a JVM written in Java t...
Byteman

PowerUP
Java Bytecode Manipulating Agent - Byteman
Uses java.lang.intrument as an agent
Can be used for:
fault injection testing, ...
Why Byteman?
AOP is powerful but complex:

• code is targeted indirectly rather than at the class/
method declaration site...
EBCA Rules
A Byteman rule consists of:

• Event - the class/interface/method target
• Binding - initialise rule values
• C...
Fault Injection
RULE simulate exception from Executor
INTERFACE ^java.util.Executor
METHOD execute
AT ENTRY
IF callerEqual...
In-built Rules
Tracing
traceOpen, traceClose, traceln, traceStack
Shared Rule State
flag, clear, flagged, countDown, increme...
BMUnit
@RunWith(BMUnitRunner.class)
@BMScript(value="traceRules", dir="scripts")
class DBTests1 {
@Test
@BMRule(
className...
Machine Lab

Bouns LeveL
ConcurrentUnit
Website: https://github.com/jhalterman/concurrentunit
JUnit addon library
Comprised of:

• A thread co-ordi...
FindBugs
Website: http://findbugs.sourceforge.net
Integrates with Maven, Gradle, Sonar, etc
Multithreaded correctness group...
Freud
Website: https://github.com/LMAX-Exchange/freud
A framework for writing static analysis tests
Lacks good documentati...
Summary

Scores
Lv.0 - Testing Tutorial

• TDD is your friend
• Prefer rules over runners
• Don’t be afraid to write custom rules and
runn...
Lv.1 - The Thread
Challenge

• Modelling real world behaviour of your application
is hard

• Don’t assume your tests are e...
Lv2. Choreographer’s
Workshop

• It is essential to test the co-ordinated
behaviour of threads

• If threads are not shari...
Lv.3 - The Shared
Resource Contest

• When a resource is shared between

threads new problem cases can arise, e.g.
races

...
Lv.4 - Princess
Protection HQ

• Many problems can’t be reduced to a 2 thread
model

• JPF can provide some automated anal...
JUnit PowerUP

GAME OVER
Credits
JUnit
http://www.junit.org
Awaitility
http://code.google.com/p/awaitility
ByteMan
http://www.jboss.org/byteman

ThreadWeaver
http://code.google.com/p/thread-weaver
ConcurrentUnit
https://github.com/jhalterman/concurrentunit

FindBugs
http://findbugs.sourceforge.net
Freud
https://github.com/LMAX-Exchange/freud

Java PathFinder
http://babelfish.arc.nasa.gov/trac/jpf
JUnit PowerUP
Practical Testing Tips
James McGivern
JUnit PowerUp
JUnit PowerUp
JUnit PowerUp
JUnit PowerUp
JUnit PowerUp
JUnit PowerUp
JUnit PowerUp
JUnit PowerUp
JUnit PowerUp
JUnit PowerUp
Upcoming SlideShare
Loading in...5
×

JUnit PowerUp

670

Published on

This is a rapid fire talk that outline some of the issues around multithreaded code, some of the tools you can use to test it and how best to spend your resources doing so

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
670
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "JUnit PowerUp"

  1. 1. JUnit PowerUP Practical Testing Tips James McGivern
  2. 2. About James Likes cats Talks fast Technical Evangelist Hates Marmite Mathematician turned Computer Scientist Lives in London
  3. 3. JUnit vs TestNG
  4. 4. One Busy Day On Concurrent Island... Intro
  5. 5. Concurrency? concurrent - adjective 1. occurring or existing simultaneously or side by side: concurrent attacks by land, sea, and air. 2. acting in conjunction; co-operating. 3. having equal authority or jurisdiction. 4. accordant or agreeing. 5. tending to or intersecting at the same point: four concurrent lines.
  6. 6. The behaviour of a single threaded application is deterministic A multithreaded application may appear stochastic Concurrency introduces new problems: deadlocks resource starvation (e.g livelocks) race conditions contention • • • •
  7. 7. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  8. 8. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  9. 9. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  10. 10. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  11. 11. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  12. 12. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  13. 13. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  14. 14. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  15. 15. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  16. 16. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
  17. 17. Each statement is an atomic block Each thread executes a non-negative number of atomic blocks Threads take turns but order is not guaranteed Given a number of threads t, and a group of statements s, the number of execution order permutations is given by: interleavings(t, s) = (ts)! t (s!)
  18. 18. The JUnit Cup Challenge Level 0
  19. 19. public class FibonacciSequenceTest { FibonacciSequence fib = new FibonacciSequence(); @Test public void shouldOutputZeroWhenInputIsZero() { assertThat(fib.get(0), is(0)) } @Test public void shouldOutputOneWhenInputIsOne() { assertThat(fib.get(1), is(1)); } @Test public void shouldCalculateNthNumber(){ for(int i = 2; i <= 10; i++) { int x = fib.get(i-1); int y = fib.get(i-2); int sum = x + y; assertThat(fib.calculate(i), is(sum); } } }
  20. 20. Cost/Benefit Ratio (TDD) Unit tests are executable specifications of the code Unit (+ integration tests) will never find all the bugs Writing tests takes time Time is limited Which tests should I write and which should I forgo?
  21. 21. JUnit Runners Bundled Suite, Parameterized • • Theories, Categories, Enclosed Popular 3rd party runners Mockito, PowerMock(ito) • Custom JBehave, Spock • • dynamic tests?
  22. 22. Custom Runner public abstract class Runner implements Describable { public Runner(Class<?> testClass){...} public abstract Description getDescription(); public abstract void run(RunNotifier n); public int testCount() { return getDescription().testCount(); } }
  23. 23. Description Description.createSuiteDescription(...) Description.createTestDescription(...) Description#addChild(Description d)
  24. 24. RunNotifier fireTestStarted(Description d) fireTestFinished(Description d) fireTestFailure(Failure f) fireTestIgnored(Description d) fireTestAssumptionFailed(Failure f)
  25. 25. Warning • A very coarse way of modifying test execution • Even when extending from BlockJUnit4ClassRunner there is a degree of complex code • Runners can not be composed e.g. @RunWith(MockitoJUnitRunner.class} @RunWith(SpringJUnit4ClassRunner.class) public class SomeTest {...} @RunWith({MockitoRunner.class, SpringJUnit4ClassRunner.class}) public class SomeTest {...}
  26. 26. JUnit Rules PowerUP
  27. 27. Rules can: Read/Write test metadata Modify the test before execution Modify the result after execution Suite vs Class == @ClassRule vs @MethodRule
  28. 28. public interface TestRule { Statement apply(Statement base, Description description); } public class MockRule implements TestRule { private final Object target; public MockRule(Object target) { this.target = target; } public Statement apply(final Statement base, Description description) { return new Statement() { public void evaluate() throws Throwable { MockitoAnnotations.initMocks(target); base.evaluate(); } }; } }
  29. 29. The Thread Challenge Level 1
  30. 30. static class VolatileInt { volatile int num = 0; } public void shouldIncrementCounter() { final int count = 32 * 1000; final int nThreads = 64; ExecutorService es = Executors.newFixedThreadPool(nThreads); final VolatileInt vi = new VolatileInt(); for (int i = 0; i < nThreads; i++) { es.submit(new Runnable() { public void run() { for (int j = 0; j < count; j += nThreads) vi.num++; } }); es.shutdown(); es.awaitTermination(10, TimeUnit.SECONDS); assertEquals(count, vi.num); } http://vanillajava.blogspot.co.uk/2011/08/why-testing-code-for-thread-safety-is.html
  31. 31. Weapons Guide • Java Concurrency in Practice - Brian Goetz • Java Performance - Charlie Hunt • Effective Unit Testing: A guide for Java developers Lasse Koskela • Programming Concurrency on the JVM -Venkat Subramaniam
  32. 32. Choreographer’s Workshop Level 2
  33. 33. http://www.recessframework.org/page/map-reduce-anonymous-functions-lambdas-php Text
  34. 34. Awaitility PowerUP
  35. 35. A Simple DSL No more Thread.sleep(), or while loops await().until( new Callable<Boolean>() { public Boolean call() throws Exception { return userRepository.size() == 1; } }; )
  36. 36. The previous example was not particularly re-usable. Let’s fix that! await().until(sizeOf(repository), equalTo(1)); where private Callable<Integer> sizeOf(Collection c){ return new Callable<Integer>() { public Boolean call() throws Exception { return c.size(); } }; }
  37. 37. Advanced Waiting • Callable is still a lot of boiler plate... await().untilCall(to(repository).size(), equalTo(3)); • Using reflection await().until( fieldIn(repository).ofType(int.class) .andWithName(“size”), equalTo(3) ); • Polling with() .pollInterval(ONE_HUNDERED_MILLISECONDS) .and().with().pollDelay(20, MILLISECONDS) .await("user registration").until( userStatus(), equalTo(REGISTERED));
  38. 38. The Shared Resource Contest Level 3
  39. 39. Race Conditions A race condition is a situation in which two or more threads or processes are reading or writing some shared data, and the final result depends on the timing of how the threads are scheduled. public class Counter { protected long count = 0; public void add(long value){ this.count = this.count + value; } }
  40. 40. ThreadWeaver PowerUP
  41. 41. public class ConcurrentHashMapTest { ConcurrentMap<String, Integer> map; @ThreadedBefore public void before() { map = new ConcurrentHashMap(); } @ThreadedMain public void mainThread() { map.putIfAbsent("A", 1); } @ThreadedSecondary public void secondThread() { map.putIfAbsent("A", 2); } } @ThreadedAfter public void after() { assertEquals(map.get(“A”), 1); }
  42. 42. Works by instrumenting bytecode at runtime Does not integrate with JUnit but can be embedded @Test public void testThreading() { new AnnotatedTestRunner() .runTests( this.getClass(), ConcurrentHashMapTest.class ); } Has fine-grained control over breakpoints/code position Documentation is ok but not a lot on in-depth material PROJECT STATUS UNKNOWN, MAYBE INACTIVE
  43. 43. Princess Protection HQ Level 4
  44. 44. A recurrent producer-consumer information processing network for anomaly detection
  45. 45. Java Path Finder http://babelfish.arc.nasa.gov/trac/jpf JPF created by NASA Open-sourced in 2005 Is a JVM written in Java that runs on the JVM Primarily used as a model checker for concurrent programs, e.g deadlocks, race conditions, NPEs Very powerful Not very usable (e.g. limited CI support)
  46. 46. Byteman PowerUP
  47. 47. Java Bytecode Manipulating Agent - Byteman Uses java.lang.intrument as an agent Can be used for: fault injection testing, tracing during test and in production, an alternative AOP (e.g. cross-cutting logging) Using instrumentation in tests means: fewer mocks less boilerplate to generate abnormal senarios
  48. 48. Why Byteman? AOP is powerful but complex: • code is targeted indirectly rather than at the class/ method declaration site • AOP definition languages can be a bit odd/hard to work with • Impact at runtime is sometimes significant to application performance due to pointcut definitions • not always trivial to turn-off, alter, or remove advice
  49. 49. EBCA Rules A Byteman rule consists of: • Event - the class/interface/method target • Binding - initialise rule values • Condition - a boolean expression (optional) • Action - some Java code that may return or throw (can not break contracts) Rules may have state, i.e 1st invocation do A, 2nd invocation do B, etc.
  50. 50. Fault Injection RULE simulate exception from Executor INTERFACE ^java.util.Executor METHOD execute AT ENTRY IF callerEquals("ServiceInstanceImpl.execute", true) DO traceln(“Throwing exception in execute”); THROW new java.util.concurrent.RejectedExecutionException(); ENDRULE
  51. 51. In-built Rules Tracing traceOpen, traceClose, traceln, traceStack Shared Rule State flag, clear, flagged, countDown, incrementCounter Synchronization waitFor, signalWake, rendezvous, delay Timing createTimer, getElapsedTime, resetTimer Call Stack Checking callerEquals, callerMatches Recursive Triggering disable/enableTriggering
  52. 52. BMUnit @RunWith(BMUnitRunner.class) @BMScript(value="traceRules", dir="scripts") class DBTests1 { @Test @BMRule( className="UserRepository”, methodName="findByName", condition= "$1.getName().contains("James")", action="THROW new UserNotFoundException") public void shouldErrorIfNoUserFound() { ... } }
  53. 53. Machine Lab Bouns LeveL
  54. 54. ConcurrentUnit Website: https://github.com/jhalterman/concurrentunit JUnit addon library Comprised of: • A thread co-ordinator Waiter • Allows assertions to be made in worker threads • Waiter collects and returns outcomes of assertions
  55. 55. FindBugs Website: http://findbugs.sourceforge.net Integrates with Maven, Gradle, Sonar, etc Multithreaded correctness group: • Inconsistent synchronization • Field not guarded against concurrent access • Method calls Thread.sleep() with a lock held • Creation of ScheduledThreadPoolExecutor with zero core threads
  56. 56. Freud Website: https://github.com/LMAX-Exchange/freud A framework for writing static analysis tests Lacks good documentation except code examples Has rules for: ✦ ✦ ✦ ✦ Java sources Java class objects. (i.e analysing the java.lang.Class object) Java class files (i.e analysing the ".class" file) Spring framework XML configuration files
  57. 57. Summary Scores
  58. 58. Lv.0 - Testing Tutorial • TDD is your friend • Prefer rules over runners • Don’t be afraid to write custom rules and runners • Ensure you spend your time on the important tests
  59. 59. Lv.1 - The Thread Challenge • Modelling real world behaviour of your application is hard • Don’t assume your tests are exhaustively testing the scenarios • Behaviour depends heavily on the system (e.g. load) • Know your stuff - read read read
  60. 60. Lv2. Choreographer’s Workshop • It is essential to test the co-ordinated behaviour of threads • If threads are not sharing resources we don’t need to care about synchronisation between thread • Awaitility allows us to check that some end state if finally reached (within the given time)
  61. 61. Lv.3 - The Shared Resource Contest • When a resource is shared between threads new problem cases can arise, e.g. races • Often the problem can be simplified to the case of 2 threads • To test the orders of execution of 2 threads we can use ThreadWeaver
  62. 62. Lv.4 - Princess Protection HQ • Many problems can’t be reduced to a 2 thread model • JPF can provide some automated analysis of concurrency issues • Byteman allows us to take control of the whole application • Both are very heavy weight and should not be
  63. 63. JUnit PowerUP GAME OVER
  64. 64. Credits
  65. 65. JUnit http://www.junit.org Awaitility http://code.google.com/p/awaitility
  66. 66. ByteMan http://www.jboss.org/byteman ThreadWeaver http://code.google.com/p/thread-weaver
  67. 67. ConcurrentUnit https://github.com/jhalterman/concurrentunit FindBugs http://findbugs.sourceforge.net
  68. 68. Freud https://github.com/LMAX-Exchange/freud Java PathFinder http://babelfish.arc.nasa.gov/trac/jpf
  69. 69. JUnit PowerUP Practical Testing Tips James McGivern

×